GPR 57641 GLOBAL DEVELOPMENT NETWORK GLOBAL PROGRAM REVIEW Volume 3 Issue 2 THE WORLD BANK GROUP WORKING FOR A WORLD FREE OF POVERTY The World Bank Group consists of five institutions—the International Bank for Reconstruction and Development (IBRD), the International Finance Corporation (IFC), the International Development Association (IDA), the Multilateral Investment Guarantee Agency (MIGA), and the International Centre for the Settlement of Investment Disputes (ICSID). Its mission is to fight poverty for lasting results and to help people help themselves and their envi- ronment by providing resources, sharing knowledge, building capacity, and forging partnerships in the public and private sectors. THE INDEPENDENT EVALUATION GROUP IMPROVING DEVELOPMENT RESULTS THROUGH EXCELLENCE IN EVALUATION The Independent Evaluation Group (IEG) is an independent, three-part unit within the World Bank Group. IEG-World Bank is charged with evaluating the activities of the IBRD (The World Bank) and IDA, IEG-IFC focuses on assessment of IFC’s work toward private sector development, and IEG-MIGA evaluates the contributions of MIGA guarantee projects and services. IEG reports directly to the Bank’s Board of Directors through the Director-General, Evaluation. The goals of evaluation are to learn from experience, to provide an objective basis for assessing the results of the Bank Group’s work, and to provide accountability in the achievement of its objectives. It also improves Bank Group work by identifying and disseminating the lessons learned from experience and by framing recommendations drawn from evaluation findings. Global Development Network May 28, 2009 Corporate and Global Evaluations and Methods http://www.globalevaluations.org ©2009 Independent Evaluation Group, The World Bank Group 1818 H Street NW Washington DC 20433 Telephone: 202-458-4497 Internet: http://www.globalevaluations.org E-mail: grpp@worldbank.org All rights reserved This volume is a product of the staff of the Independent Evaluation Group (IEG) of the World Bank Group. It is part of an ongoing series that reviews global and regional partnership programs in which the World Bank is engaged as one of the partners. The findings, interpretations, and conclusions expressed in this volume do not necessarily reflect the views of the Executive Directors of The World Bank or the governments they represent. IEG does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of IEG concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this publication is copyrighted. IEG encourages the dissemination of its work and permits these reviews to be copied or otherwise transmitted, with appropriate credit given to IEG as the authoring body. Cover image: World map showing GDN’s Regional Network Partners. ISBN-13: 978-1-60244-118-7 ISBN-10: 1-60244-118-9 Printed on Recycled Paper Independent Evaluation Group Knowledge Programs and Evaluation Capacity Development (IEGKE) E-mail: grpp@worldbank.org Telephone: 202-458-4497 iii IEG Mission: Improving Development Results Through Excellence in Evaluation The Independent Evaluation Group (IEG) of the World Bank reviews global and regional partnership programs (GRPPs) in which the Bank is engaged as one partner among many for two main purposes: (a) to provide accountability in the achievement of the program’s objectives by providing an independent opinion of the program’s effectiveness, and (b) to identify and disseminate lessons learned from the experience of individual GRPPs. The preparation of a global or regional program review (GPR) is contingent on a recently completed evaluation of the program, typically commissioned by the governing body of the program. The first purpose includes validating the findings of the GRPP evaluation with respect to the effectiveness of the program, and assessing the Bank’s performance as a partner in the program. The second purpose includes assessing the independence and quality of the GRPP evaluation itself and drawing implications for the Bank’s continued involvement in the program. Assessing the quality of GRPP evaluations is an important aspect of GPRs, since encouraging more consistent evaluation methodology and practice across Bank-supported GRPPs is one of the reasons why IEG embarked on this new product in 2005. IEG annually reviews a number of GRPPs in which the Bank is a partner. In selecting programs for review, preference is given to those that are innovative, large, or complex; those that are relevant to upcoming sector studies; those for which the Executive Directors or Bank management have requested reviews; and those that are likely to generate important lessons. IEG also aims for a representative distribution of GPRs across sectors in each fiscal year. A GPR is a “review” and not a full-fledged “evaluation.” It assesses the independence and quality of the relevant evaluation; provides a second opinion on the effectiveness of the program; assesses the performance of the Bank as a partner in the program; and draws lessons for the Bank’s engagement in global and regional programs. The GPR does not formally rate the various attributes of the program. A GPR involves a desk review of key documents, consultations with key stakeholders, and a mission to the program management unit (secretariat) of the program if this is located outside of the World Bank or Washington, DC. Key stakeholders include the Bank’s representative on the governing body of the program, the Bank’s task team leader (if separate from the Bank’s representative), the program chair, the head of the secretariat, other program partners (at the governance and implementing levels), and other Bank operational staff involved with the program. The writer of a GPR may also consult with the person(s) who conducted the evaluation of the GRPP. Each GPR is subject to internal IEG peer review, Panel review, and management approval. Once cleared internally, the GPR is reviewed by the responsible Bank department and the secretariat of the program. Comments received are taken into account in finalizing the document, and the formal management response from the program is attached as an annex to the final report. After the document has been distributed to the Bank’s Board of Executive Directors, it is disclosed to the public on IEG’s external Web site. iv Abbreviations and Acronyms 3ie International Initiative for Impact Evaluation AERC African Economic Research Consortium (GDN regional network partner) CERGE-EI Center for Economic Research and Graduate Education – Economics Institute (GDN regional network partner for Eastern and Central Europe) CODE Committee on Development Effectiveness (World Bank) DEC Development Economics Vice Presidency (World Bank) DFID Department for International Development (UK) DGF Development Grant Facility (World Bank) EADN East Asian Development Network (GDN regional network partner) EERC Economics Education and Research Consortium (GDN regional network partner for Russia and the Commonwealth of Independent States) ERF Economic Research Forum (GDN regional partner in the Middle East and North Africa) GDAMC Global Development Awards and Medal Competition (GDN) GDN Global Development Network GPR Global Program Review GRP Global Research Project (GDN) IDS Institute for Development Studies (University of Sussex, UK) IEG Independent Evaluation Group (World Bank) IRP Inter-regional Research Project (GDN) LACEA Latin American and Caribbean Economic Association (GDN regional network partner) MOU Memorandum of understanding ODN Oceania Development Network (GDN regional network partner) PREM Poverty Reduction and Economic Management Vice Presidency and Network (World Bank) RNP Regional Network Partner (GDN) RRC Regional Research Competition (GDN) SANEI South Asia Network of Economic Research Institutes (GDN regional network partner) TTL Task team leader (World Bank) UNDP United Nations Development Program VPU Vice Presidential Unit (World Bank) Fiscal Year of Program July 1 to June 30 Director-General, Independent Evaluation Mr. Vinod Thomas Director, Independent Evaluation Group (World Bank) Ms. Cheryl Gray Manager, Corporate and Global Evaluations and Methods Mr. Mark Sundberg Global Programs Coordinator Mr. Chris Gerrard Consultant Ms. Catherine Gwin v Contents ABBREVIATIONS AND ACRONYMS ..................................................................................................IV  PROGRAM AT A GLANCE: GLOBAL DEVELOPMENT NETWORK ...............................................VII  KEY BANK STAFF RESPONSIBLE DURING PERIOD UNDER REVIEW .......................................VIII  GLOSSARY...........................................................................................................................................IX  PREFACE..............................................................................................................................................XI  SUMMARY ..........................................................................................................................................XIII  1.  PROGRAM OBJECTIVES, ACTIVITIES, AND COSTS ........................................................... 1  Program Origin and Objectives.............................................................................................. 1  Activities ................................................................................................................................... 2  The Regional Research Competitions (RRCs) .................................................................................. 2  The Global Research Projects (GRPs) .............................................................................................. 4  Global Development Award and Medal Competitions ....................................................................... 4  Annual Conferences .......................................................................................................................... 4  GDNet ............................................................................................................................................... 5  Governance and Organization ............................................................................................... 5  2.  THE 2007 INDEPENDENT EVALUATION PROCESS ............................................................ 8  Design and Methodology ........................................................................................................ 9  Findings, Recommendations, and GDN’s Response ........................................................ 10  3.  THE EFFECTIVENESS OF GDN ............................................................................................ 11  Relevance ............................................................................................................................... 11  Efficacy ................................................................................................................................... 15  Achievements of GDN’s Five Main Activities ...................................................................................17  Overall Achievement of Core Objectives ......................................................................................... 20  Efficiency ................................................................................................................................ 25  4.  GOVERNANCE, MANAGEMENT, AND FINANCIAL SUSTAINABILITY ........................... 28  5.  THE WORLD BANK’S PERFORMANCE AS A PARTNER .................................................. 33  Initiation and Incubation ....................................................................................................... 33  Strategic Guidance ................................................................................................................ 35  Funding................................................................................................................................... 35  Oversight ................................................................................................................................ 37  Exit Strategy ........................................................................................................................... 38  6.  LESSONS LEARNED ............................................................................................................. 39  vi REFERENCES ..................................................................................................................................... 43  ANNEX A. EVALUATION FRAMEWORK FOR GLOBAL PROGRAM REVIEWS............................ 45  ANNEX B. GLOBAL RESEARCH PROJECTS (COMPLETED AND ONGOING) ............................ 53  ANNEX C. 2007 EXTERNAL EVALUATION: RECOMMENDATIONS AND GDN MANAGEMENT RESPONSE ......................................................................................................................................... 55  ANNEX D. MEMBERS OF THE GOVERNING BODIES .................................................................... 61  ANNEX E. SOURCES AND USES OF FUNDS .................................................................................. 62  ANNEX F. PERSONS CONSULTED .................................................................................................. 65  ANNEX G. RESPONSE OF THE PROGRAM TO IEG’S GLOBAL PROGRAM REVIEW ................ 67  Boxes Box 1. GDN’s Plans for Expanding Its Activities ............................................................................ 21 Box 2. Risk and Risk Mitigation Measures ...................................................................................... 33 Tables Table 1. GDN Timeline ......................................................................................................................... 1 Table 2. GDN Grant Support to RNPs as Percentage of Their Total Annual Revenues, FY07..... 7 Table 3. Level of Activity ................................................................................................................... 16 Table 4. Researchers’ Overall Satisfaction with GDN Activities .................................................. 17 Table 5. Beneficiaries’ Ratings of GDN Contributions to Various Dimensions of Capacity Building .................................................................................................................................. 23 Table 6. Cost-Effectiveness Comparison with Other Research Organizations ........................... 27 Table 7. GDN Revenue from Donors (FY05–FY08) ......................................................................... 32 Table 8. World Bank Support to GDN from Its Inception as an Independent Organization Through FY08 ........................................................................................................................ 36 Figures Figure 1. GDN Expenditures on Five Core Program Activities, FY02–08....................................... 3 Figure 2. Research Output (as percentage of the grantee survey respondents) ........................ 22 Figure 3. Assessment of Publishability (as percentage of selected studies assessed by independent referees)........................................................................................................................ 23 vii Program at a Glance: Global Development Network Start date GDN was formally launched in December 1999 as a global program with its Secretariat located in the World Bank. It was spun-off as an independent not- for-profit organization based in Washington, DC, in March 2001, and then relocated to New Delhi, India, as an international organization in 2005. Goal GDN’s overarching goal has been to promote the generation, sharing, and application to policy of multidisciplinary knowledge for the purpose of development. Program objectives GDN has had three core program (outcome) objectives: (1) to generate high-quality, policy-relevant development research from within developing and transitional countries (2) to build research and policy outreach capacity of researchers from those countries (3) to promote linkages between research and the policy-making process to foster better policies. Activities GDN has sponsored five core activities: regional research competitions, global research projects, an Annual Conference, a Global Development Awards and Medals Competition, and GDNet (a Web-based source of knowledge, information, and services). In FY09, GDN became the host for the International Initiative for Impact Evaluation (3ie). WBG contributions From fiscal year 1998 through fiscal year 2008, the Bank provided a total of $43.6 million to GDN, of which $41.6 million was in grants from the Development Grant Facility (DGF) and the other $2.0 million from Bank’s administrative budget. Since leaving the World Bank, the DGF contribution to total GDN expenditures has declined from 75 percent in FY02 to 38 percent in FY08. Other donor Currently, 12 bilateral aid agencies, one multilateral agency, and two contributions foundations provide support for GDN’s five main activities. Of those, two bilateral donors contribute core funding and the rest provide mainly project financing, or limited in-kind support. Location GDN is now an international organization headquartered in New Delhi, with offices in Cairo and Washington, DC. Established as a unit of the World Bank, GDN moved to independent quarters in Washington, DC, in 2001 and then to New Delhi in 2004. Governance and GDN is governed by an Assembly — comprised of states and public management organizations that are signatories to the GDN charter — and a Board of Directors — currently 18 individuals, 16 of whom are researchers from universities or research institutions in different regions and two of whom are senior representatives of GDN donor agencies, the World Bank and UNDP. There are three board subcommittees: executive, audit, and program. Day- to-day operations are managed by a GDN Secretariat, comprising a President and some 30 staff. Latest program- Evaluation of the Global Development Network, December 2007, conducted level evaluation by Marc D. Shapiro, principal investigator and managing partner of the private firm MDS Associates viii Key Bank Staff Responsible during Period under Review Position Person Period Program Manager Lyn Squire, Director 1999–2001 Global Program Task Team Ishac Diwan, Manager, WBI 2000–2001 Leader Jo Ritzen 2001–2002 Ines Garcia-Thoumi 2002–2004 Alan Winters, Director, DECRG 2004–2008 Ivar Cederholm, Sr. RM Officer, 2008–present DECRM World Bank Representative on Guillermo Perry, Chief Economist, 2002–2006 the Board of Directors LCR Alan Winters, Director, DECRG 2006–2007 Alan Gelb, Director, Development 2007–present Policy, DECVP Vice President Kemal Dervis, PRM 1999–2001 Gobind Nankini, PRM 2001–2004 Francois Bourguignon, DEC 2004–2008 Yifu Lin, DEC June 2008–present Trust Fund Operations Arif Zulfiqar, Director June 1999–September 2008 Global Programs and Margret Thalwitz, Director May 2004–November 2008 Partnerships Program Manager Position Person Period Director Lyn Squire 1999–2001 Executive Director Lyn Squire 2001–2005 President Lyn Squire 2005–2007 President Gobind Nankini 2007–March 2009 ix Glossary Devolution or exit A proactive strategy to change the design of a program, to devolve some of its strategy implementation responsibilities, to reduce dependency on external funding, or to phase out the program on the grounds that it has achieved its objectives or that its current design is no longer the best way to sustain the results which the program has achieved. Efficacy The extent to which the program has achieved, or is expected to achieve, its objectives, taking into account their relative importance. The term is also used as a broader, aggregate measure — encompassing relevance and efficiency as well — of the overall outcome of a development intervention such as a GRPP. Efficiency The extent to which the program has converted or is expected to convert its resources/inputs (such as funds, expertise, time, etc.) economically into results in order to achieve the maximum possible outputs, outcomes, and impacts with the minimum possible inputs. Evaluation The systematic and objective assessment of an ongoing to completed policy, program, or project, its design, implementation, and results. The aim is to determine the relevance and achievement of its objectives, and its developmental effectiveness, efficiency, impact, and sustainability. Governance The structures, functions, processes, and organizational traditions that have been put in place within the context of a program’s authorizing environment to ensure that the program is run in such a way that it achieves its objectives in an effective and transparent manner. It is the framework of accountability and responsibility to users, stakeholders and the wider community, within which organizations take decisions, and lead and control their functions, to achieve their objectives. Impacts Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended. Independent An evaluation that is carried out by entities and persons free from the control of evaluation those involved in policy-making, management, or implementation of program activities. This entails organizational and behavioral independence, protection from interference, and avoidance of conflicts of interest. Indicator A quantitative or qualitative factor or variable that provides a simple and reliable means to measure achievement, to reflect the changes connected to an intervention, or to help assess the performance of a development actor. Legitimacy As a criterion for assessing governance and management, the way in which governmental and managerial authority is exercised in relation to those with a legitimate interest in the program — including shareholders, other stakeholders, implementers, beneficiaries, and the community at large. Logical A management technique that is used to develop the overall design of a framework or program or project, to improve implementation monitoring, and to strengthen logframe evaluation, by presenting the essential elements of the program or project clearly and succinctly throughout its cycle. It is a “cause and effect” model which aims to establish clear objectives and strategies based on a results chain, to build commitment and ownership among the stakeholders during the preparation of the program or project, and to relate the program’s or project’s interventions to their intended outcomes and impacts for beneficiaries. Management The day-to-day operation of a program within the context of the strategies, policies, processes, and procedures that have been established by the governing body. x Monitoring The continuous assessment of progress achieved during program implementation in order to track compliance with a plan, to identify reasons for noncompliance, and to take necessary actions to improve performance. Monitoring is usually the responsibility of program management and operational staff. Outcomes The achieved or likely short-term and medium-term effects of the outputs of a development intervention. Oversight One of the core functions of the governing body of a program: Monitoring the performance of the program management unit, appointing key personnel, approving annual budgets and business plans, and overseeing major capital expenditures. Partners In most IEG Global Program Reviews, partners are understood as stakeholders who are involved in the governance or financing of the program (including the members of the governing, executive, or advisory bodies). Public goods Goods which produce benefits that are non-rival (many people can consume, use, or enjoy the good at the same time) and non-excludable (it is difficult to prevent people who do not pay for the good from consuming it). If the benefits of a particular public good accrue across all or many countries, then the good is deemed a global or international public good. Relevance The extent to which the objectives and design of the program are consistent with (a) the current global/regional challenges and concerns in a particular development sector and (b) the needs and priorities of beneficiary countries and groups. Shareholders The subset of donors that are involved in the governance of the program. Therefore, this does not include individual (particularly anonymous) donors who choose not to be so involved, or who are not entitled to be involved if their contribution does not meet the minimum requirement, say, for membership on the governing body. Stakeholders The parties who are interested in or affected, either positively or negatively, by the program. Stakeholders are often referred to as “principal” and “other”, or “direct” and “indirect”. While other or indirect stakeholders — such as taxpayers in both donor and beneficiary countries, visitors to a beneficiary country, and other indirect beneficiaries — may have interests as well, these are not ordinarily considered in evaluations unless a principal stakeholder acts as their proxy. Sustainability When the term is applied to the activities of a program, the extent to which the benefits arising from these activities are likely to continue after the activities have been completed. When the term is applied to organizations or programs themselves, the extent to which the organization or program is likely to continue its operational activities over time. Transparency As a criterion for assessing governance and management, the extent to which a program’s decision-making, reporting, and evaluation processes are open and freely available to the general public. This is a metaphorical extension of the meaning used in physical sciences — a “transparent” objective being one that can be seen through. Source: For evaluation terms, the Sourcebook for Evaluating Global and Regional Partnership Programs: Indicative Principles and Standards, Independent Evaluation Group – World Bank, 2007. xi Preface This is the Global Program Review (GPR) of the Global Development Network (GDN). Established in 1999 as a global partnership program with a secretariat located in the World Bank, it was spun off as an independent not-for-profit organization based in Washington, DC, in 2001 and then relocated to New Delhi, India, as an international organization in 2004. GDN’s overarching goal has been to promote the generation, sharing, and application to policy of multidisciplinary knowledge for the purpose of development. To accomplish this, GDN has focused on three core program objectives: (1) generating high-quality, policy-relevant research in developing and transition countries; (2) building research and policy outreach capacity among researchers in those countries to improve the quality and expand the policy influence of their work on a national and international level; and (3) promoting greater linkages between researchers and the policy process to foster effective, evidence-based policy-making. GDN has a notable record of external and internal evaluations for a relatively young organization. This GPR is based on the most recent external evaluation, which was commissioned by the GDN Board of Directors and completed by a small team of external consultants in 2007. The GPR assesses the independence and quality of the 2007 evaluation, provides a second opinion on the effectiveness of GDN’s work, assesses the performance of the World Bank as a partner in GDN, and draws some lessons for GDN and for the Bank’s engagement in global programs more generally. GDN was chosen for a GPR because the World Bank has been heavily involved in the program as founder, major donor, and Board member. The Bank has provided financial support through the long-term financing window of the Development Grant Facility (Window 1), provided the leadership of the Secretariat, and participated in the organization’s Annual Conferences and other activities. GDN is the major global partnership program in which the Bank’s Development Economics Vice Presidency (DEC) is involved. This review follows IEG’s Evaluation Framework for Global Program Reviews (Annex A). It is based on a desk review of the 2007 evaluation report and other GDN documents (annual reports, internal documents, strategy papers, etc.), a mission to the GDN Secretariat in New Delhi, interviews with key stakeholders (Board members, Regional Network Partners, donors, and World Bank staff), and a workshop with development policy analysts and practitioners. IEG gratefully acknowledges all those who made their time available for interviews and provided useful information and insights into the program. It wishes to especially acknowledge the availability of GDN staff and their cooperation in providing all necessary information and documents. The complete list of people consulted can be found in Annex F. Following IEG’s normal procedures, copies of the draft GPR were sent to GDN, to the Bank’s global program task team leader, to DECRG (which is responsible for the Bank’s engagement with GDN), and to other World Bank units that have responsibility for the Bank’s involvement with global programs more generally. Their comments have been taken into account in finalizing the GPR. The formal response received from GDN management is attached in Annex G. xiii Summary 1. The World Bank initiated the Global Development Network (GDN) in the late 1990s to increase the output of high-quality, policy-relevant research from developing and transition countries that could feed into local policy-making. Since then, the Bank has remained heavily involved as a major donor and a member of GDN’s governing board. 2. This IEG review assesses the relevance and effectiveness of GDN over its 10-year history and the Bank’s performance as a major partner. In making this assessment, the review draws on two independent evaluations of GDN (in 2004 and 2007) and other internal and external assessments of specific aspects of the program. It also incorporates views on developments since 2007 obtained through interviews conducted in late 2008 with some 30 key GDN stakeholders (notably representatives of regional network partners, donor agencies, GDN Board members, and GDN Secretariat staff) as well as views provided in a workshop of development policy analysts and practitioners. GDN’s Status, Objectives, and Activities 3. GDN has undergone a rapid and successful institutional transformation. Established in 1999 as an internal unit of the Bank, GDN is now an independent international organization located in New Delhi, India. 4. Throughout this institutional transformation, GDN’s main objectives and activities have remained constant. As stated in its charter, GDN’s overarching goal is to promote the generation, sharing, and application to policy of multidisciplinary knowledge for the purpose of development. To accomplish this overarching goal, GDN focuses on three core program objectives (outcomes): (1) to generate high-quality, policy-relevant research in developing and transition countries; (2) to build research and policy outreach capacity among researchers in those countries to improve the quality and expand the policy influence of their work on a national and international level; and (3) to promote greater linkages between researchers and the policy process to foster effective, evidence-based policy-making. 5. In pursuit of these program objectives, GDN sponsors five main activities (outputs):  Regional research competitions (RRCs) are funded with annual grants from GDN to eight Regional Network Partners (RNPs) and support the work of development researchers in developing and transition countries.  Global research projects (GRPs) — which GDN organizes and funds centrally — examine aspects of development and typically involve case studies in multiple countries by teams of 30–80 researchers from across two or more regions.  Global Development Awards and Medal Competitions (GDAMCs) annually award a total of about $300,000 for outstanding research and innovative development projects by researchers and development practitioners from developing and transition countries.  Annual Conferences, which involve about 500 participants yearly, provide a global forum for the exchange of ideas on a broad topic among developing country xiv researchers, policymakers, and donors as well as an opportunity for researchers to showcase their work.  GDNet, a Web-based activity, showcases the work of developing and transition country researchers, provides them with a package of online services, and helps build the communications capacity of research institutions to enhance the policy influence of their research. GDN’s Program Relevance and Achievements 6. For a relatively young organization, GDN has a notable record of external and internal evaluations. Also notable is GDN’s transparency and responsiveness in regard to the evaluations’ findings and recommendations. Overall, the evidence on GDN’s effectiveness from these various sources is rather thin — reflecting in part the inherent difficulty of assessing activities aimed at building research capacity and influencing policy and in part GDN weaknesses in setting and tracking clear and monitorable outcome objectives. Using the available evidence, this IEG review finds, overall, that GDN’s relevance is moderately strong, its record of outputs is strong, and its achievement of program outcome objectives is moderate, with notable shortcomings. 7. Relevance. The relevance of GDN’s overarching objective was high at the outset of the program and remains high today. Its objective of generating research from within developing and transitional countries is widely viewed as responsive to a need for such work to inform national policies and strengthen the voice of these countries in global policy- making. Historically, most development research has come from outside developing countries. Funding for developing country research is in short supply: little support comes from governments or other domestic sources and international research support is most often for inputs into international policies and practices. 8. Conceptually, GDN is also well designed for pursuing its objective. Its reach is global in scope, involving researchers and partner institutions from all regions of the world. Its program is wholly focused on promoting developing and transition country research. It is structured as a “network of networks” (as distinct from a global think tank) to maximize the benefits of sharing knowledge and research experience across regions. Also, it conducts a menu of activities as a way to achieve its three program objectives. 9. Yet, this review of GDN finds substantial weaknesses in the translation of its design into practice. Notably, GDN has never stated clearly the extent to which its goal is to address inadequacies that are due to a shortage of development-related research, poor quality research, or inadequate policy relevance of existing work; nor has it identified where its specific comparative advantage lies in relation to these inadequacies. As a result, there is a continuing lack of consensus among key stakeholders on the relative importance of its three objectives of increased high-quality research output, research capacity building, and policy outreach, and lack of consensus on the relative emphasis that each of its activities should give to these objectives. Also, as discussed further below, although GDN has put time and resources into conducting a menu of activities, it has not developed explicit, state-of-the art strategies that incorporate these activities into systematic approaches for advancing each of its three program objectives, nor has it developed a logical (results) framework for guiding xv and tracking the effectiveness of its work. Moreover, GDN has not adequately defined the mutual responsibilities and working relationships of its Secretariat and RNPs in the design and implementation of GDN-supported activities. 10. Program achievements. GDN’s output record is strong in each of its five areas of activity. Notably:  Through FY08, GDN provided funding for over 800 research grants through the competitive RRCs, with varying effectiveness across regions and across GDN’s three program objectives.  So far, GDN has completed four GRPs, and, at the end of 2008, had four other GRPs ongoing. The success of the GRPs has varied across projects and has been stronger in generating knowledge than in building the capacity of researchers.  As of 2008, a total of over 4,000 participants from some 100 countries had attended the eight Annual Conferences convened since 1999, of which close to 70 percent were from developing and transition countries. The conferences have enabled more networking among researchers from around the world than contact between researchers and policymakers  Nearly 4,300 individuals from over 100 countries have participated in GDN’s awards and medals competitions to date. In 2007, the competition attracted more than 600 applicants. But there is little evidence of the impact of these awards on individuals’ careers or their policy influence.  GDN’s Web-based activity, GDNet, now provides online access to some 14,000 research papers from developing and transition countries. It also provides a package of online services for developing country researchers and research institutes, but has not yet realized its aim of enhancing the policy outreach of GDN-supported researchers and their work. 11. Based on this output record, GDN is now planning to expand its range of activities to include training programs, strengthen efforts to link research to policy-making, and build the capacity of research institutions. While it is too soon to evaluate any of these new activities, it is apparent that they represent a substantial increase in the level and scope of GDN’s program. Thus, two key questions for GDN going forward are (a) how to ensure that its new activities reinforce rather than distract from the contribution of its existing core activities to achieving GDN’s intended outcomes and (b) how to monitor and evaluate program outcomes. 12. So far, the progress on advancing GDN’s outcome objectives appears to be much more modest than the record of outputs. Overall, evaluations show some “moderate evidence” of achievements in advancing two of GDN’s three objectives — increased high- quality, policy-relevant research and enhanced research capacity — and very limited progress in advancing its third aim of informing policy-making.  Generation of high quality, policy-relevant research. GDN’s strongest record to date is supporting increased amounts of development research from within developing and transitional countries. GDN-funded research has also led to an increase in the dissemination of that work through papers, journal articles, and books. But, there is not sufficient cumulative evidence to be able to assess the overall xvi quality, newness, or policy relevance of the research output, nor to know if the research has improved in quality and policy relevance over GDN’s 10 years of operation. The 2007 evaluation of GDN did not attempt to examine even a sample of the reports and publications that had been produced to assess the quality and newness of GDN-funded research or the policy relevance of the work.  Research capacity building. Combined findings on the effectiveness of GDN’s five activities indicate that GDN has been moderately effective in building research capacity at the individual level. In addition, survey data from 2005 and 2007 suggest a modest increase in GDN’s contribution to building research capacity over time. For example, grantees report stronger publication records emanating from their grants and prizes in 2007 as compared to 2005. But the 2007 evaluation found no corroborating evidence of positive capacity-building effects from RRCs or GRPs using other evaluative measures. It also noted that the mentoring process during the course of the RRCs and GRPs — seen by GDN as a major feature of its capacity-building effort — had declined since 2005 and was of widely varying effectiveness across regions. The dominant view of interviewees for this IEG review is similar: while all interviewees agreed that capacity building is a key objective of GDN, most stated that the capacity- building effort is not systematically designed. This suggests that if GDN is now going to expand its capacity-building effort (through training and support for research institutions), it ought to do so not as stand-alone additions to current activities, but as part of the development of a comprehensive capacity-building strategy.  Informing policy. While the 2007 evaluation concluded that GDN’s aim of “influencing policy” was unrealistic for a research capacity-building program, it stated that GDN could claim to be building a cadre of professionals capable of informing the policy process in support of better policy-making. But it also reported that stakeholders rate GDN’s contribution to “facilitating contacts with policy- makers” lower on average than its other contributions to individual capacity building and lower than reported in 2005. A similar conclusion emerged from a separate GDNet evaluation that found that while GDNet is a key means for enhancing the policy outreach of GDN-supported research, it needs to narrow the gap between this purpose and its current outputs. In sum, GDN’s efforts to strengthen the policy relevance of research training and output remains weak. 13. On the whole, this IEG review finds that to make greater progress, GDN needs to shift its strategic planning from outputs to outcomes in the design and implementation of its activities. It also needs to develop more strategic approaches to advancing its core program objectives. Organizational Efficiency, Governance, and Sustainability 14. Overall, GDN has handled resource allocations, administrative costs, and financial management efficiently. For example:  It has dealt efficiently with considerable volatility in the availability of annual revenues. Activities were scaled up or down to match available funds year by year, xvii and primary activities were funded to a level deemed satisfactory to key stakeholders even if not at levels planned.  Its own cost-effectiveness assessment indicates that it operates at relatively low cost compared to several other research institutions of similar or larger size.  An independent assessment of GDN’s financial management gave GDN’s practices an overall satisfactory rating, although it found that GDN did not have adequate processes in place to ensure proper use of sub-grants awarded by its RNPs. GDN has since implemented corrective actions recommended by the financial management assessment. 15. Still, three institutional features have hampered program effectiveness — GDN’s governance structure, working relations with its RNPs, and uncertain financial sustainability. Though GDN made some improvements in these areas after its first independent evaluation in 2004, further improvements are still needed as acknowledged in GDN’s 2008 Strategic Review and proposed actions approved by its Board. 16. Governance. GDN’s Board, which is the organization’s main governing body, has become increasingly effective over time. This development is a result of GDN’s shift to independent status and to the establishment of three Board committees (executive, program, and audit) in 2005. Also, in response to the 2007 evaluation, which suggested further strengthening of Board oversight, GDN is moving to build greater transparency into the process of choosing board members and greater board experience in managing organizations. Even so, this IEG review sees two unresolved governance problems. First, there is a potential for a conflict of interest posed by the Board members who are nominated by regional network partners and are in an ambiguous position of representing the interests of their specific regions as well as exercising rigorous oversight of the performance of GDN overall. Second, the regional network partners are not members of the Board and therefore are not principally and directly engaged in GDN’s research agenda-setting function currently carried out by the Board. One possible alternative approach might be to focus the business of the board on oversight and strategic direction and establish a separate research advisory body (including the heads of the RNPs) with responsibility for setting GDN’s research and capacity-building agenda. 17. Working relations with RNPs. Conceived as “a network or networks,” GDN works with 11 RNPs, 9 of which are in developing and transition regions and 2 in developed regions. The prevailing view across GDN stakeholders is that the RNPs are GDN’s strength and the cornerstone of the program. Yet, the RNPs, other stakeholders, and the 2004 and 2007 evaluations share the view that the working relationship between GDN and the RNPs is not well developed. In particular, GDN needs to do better in dealing with the variations in the capacity and performance of the different RNPs in building research excellence and informing policy — especially in the case of the newer RNPs for whom GDN is a relatively significant source of support. It needs to strengthen its global services to the RNPs where there is potential for effectiveness and efficiency gains through such undertakings as cross- regional capacity building, standard setting, and fund-raising. It also needs to adjust how it operates to increase the RNP’s voice in the design and implementation of GDN activities — thus making them true partners. xviii 18. In the past year, GDN’s management and Board have acted to make the deepening of relations with RNPs a major focus of GDN’s plans for the next several years. Two new steps have been the initiation of inter-regional research projects and GDN/RNP jointly funded activities. In interviews for this IEG review, RNP coordinators commended these and other recent efforts. Given stakeholder views and evaluation findings about the relevance, comparative advantage, and effectiveness of GDN in promoting and disseminating research for better policy-making, there would seem to be no higher priority for the organization in the coming years than to move effectively in this area. 19. Financial sustainability. GDN has improved its financial position in recent years by mobilizing larger and more diversified sources of funding. In 2008, GDN received a total of $8.7 million from some 16 donors compared to $7.0 million from some 11 donors in 2005. This increase included first-time support in 2007 from seven new donors to GDN. This is a notable improvement over the earlier period 2005–07, when it experienced a three-year decline in funds (due primarily to reduced support from the World Bank not compensated for by increased or new funding from others). Still, GDN faces a funding situation characterized by five difficult features:  Continuing dependence on some 2–3 donors (including the World Bank) for over 65 percent of total resources. The Bank continues to be the largest donor, even after 10 years.  Many small donors, entailing high transaction costs for GDN management  Low levels of support from private and developing country sources  High risks of losing some donors in any year (for example, the government of Italy in 2006)  Very high rigidity in the uses to which funding can be put, with the bulk linked to specific projects and only limited core (fungible) support for such things as strengthening its interactions with RNPs and integrating its activities in effective ways. 20. While this IEG review finds that on the whole GDN has been moving in the right direction to increase its effectiveness since the 2007 evaluation, this movement may now be at risk. The resignation of the previous president in March 2009 (after less than two years in that position) and his replacement effective mid-August 2009 leaves open questions about how far and how fast any changes will now be made. The World Bank’s Performance as a Major GDN Partner 21. The World Bank has provided important financial and technical support to GDN since its founding. Over fiscal years 1998–2008, the Bank has contributed a total of some $45.6 million to GDN, the bulk in the form of successive DGF grants. Active Bank staff members have served on GDN’s board, and active and retired staff have so far provided the leadership of the Secretariat and regularly participated in the organization’s Annual Conferences and research activities, thereby providing valued strategic guidance and technical assistance. Yet, Bank performance in the strategic direction and oversight of GDN has been inadequate. The inadequacy in Bank oversight is evidenced, for example, by the difficulty encountered by this IEG review in obtaining internal records regarding Bank xix participation and GDN activities. Moreover, by assuming a position on the Board, the Bank has a responsibility to push for strategic clarity and direction, and it is implicated in the GDN’s weakness in this regard. 22. Also, the Bank has lacked a sound funding strategy. GDN remains heavily dependent on Bank financing not only in dollar amounts, but also because the Bank is one of only three sources of core support (the other two at much lower levels) and the main source of GDN support to local researchers through the RRCs. This continued dependence suggests a lack of realism on the part of the Bank about its ability to leverage other donor resources, or a lack of sufficient effort by the Bank’s senior management and representative on the Board to help GDN mobilize resources. Notably, while GDN’s budget shows a reduction in the percentage of Bank support of from 75 percent in 2002 to 38 percent in 2008, GDN has not been successful in mobilizing other (non-Bank) funding equal to 85 percent of the program’s budget — which is the DGF target guideline for ongoing program financing. Rather this target has repeatedly been set and then ignored as a condition for successive DGF funding of GDN. Moreover, the Bank does not have an explicit strategy for exiting from its partnership with GDN, nor an appropriate approach to the design of such a strategy. Those responsible for overseeing the partnership contend that it is too soon to set out such a strategy. They argue that GDN is still young and it typically takes 15–20 years for an organization of its kind to become fully established. Yet, the Bank’s thinking about an exit strategy seems to focus mainly on the timing of Bank withdrawal, rather than on how the Bank should assist GDN and its RNPs in developing and implementing a resource mobilization strategy for achieving long-term sustainability. While the Bank may not choose to play this role in all partnerships in which it participates, certainly for a global partnership of its “invention” it needs to commit itself to a major engagement in securing the initiative’s financial sustainability. Setting and then ignoring guidelines for continued funding is not an adequate approach. Lessons Learned 23. The review of GDN’s relevance and effectiveness highlights six main lessons, of which four are relevant to design and implementation of global programs and two are relevant to the Bank’s support of such programs. 24. Effectiveness depends on having a theory of change and explicit strategies for achieving outcome objectives. GDN’s moderate performance to date highlights that having well-articulated theory of change and related operational strategies for achieving intended program results (especially when the intended results — such as capacity building and policy influence — are not easy to quantify or attribute directly to program activities). 25. Global networks require well-defined roles and responsibilities for their constituent parts. GDN rests on the assumption that there is knowledge to be gained and cost- efficiencies to be achieved by promoting research collaboration on a global scale across regions. But its experience shows the shortcomings of operating with limited interaction across regional network partners and limited support from the center for the differing needs of the regions. GDN's maximum value added would come from operating as a true network of networks that addresses these shortcomings. xx 26. It is difficult to design and implement a coherent global program without a substantial level of core funding. Increasing revenues have enabled GDN to expand its support of developing world research and voice. But the dominance of earmarked funding impedes GDN’s ability to support its regional partners and to design and link activities in ways that could best advance its threefold objectives of research generation, capacity building, and policy outreach. 27. There are weaknesses in a governance arrangement in which the governing board is responsible for setting the program’s research agenda as well as for exercising overall strategic direction and oversight of program performance. By both setting and overseeing GDN’s research agenda, its Board risks shorting its broader strategic stewardship role and obscuring the voice of the program’s target beneficiaries in defining what is needed to achieve core objectives. Although GDN’s governing board has become more efficient in recent years, issues of broad strategic direction and financial sustainability would seem to require more Board attention, while choices of GRP and other research activities might be better left to a non-Board research advisory body (including the management heads of the RNPs among others). 28. A realistic commitment by the Bank is required in establishing a donor-dependent program that seeks to advance a long-term objective. GDN’s financing history suggests that timetables for the reduction of World Bank support need to be better calibrated to the nature of specific programs than is currently the case with DGF guidelines. It also suggests that exit strategies should be devised from the outset that not only set out a schedule for termination of Bank support, but also specify the extent of the Bank’s commitment to helping the organization plan and achieve its long-term financial sustainability commensurate with the Bank’s interest in the program. 29. The Bank needs to develop strengthened guidelines for its participation, and the accountability of its performance, in global partnerships. The Bank’s accountability of its partnership role has been weak throughout the life of GDN. This weakness has been evidenced by repeated and exaggerated expectations in seeking DGF funding; poor record keeping; and understatements in progress reports of GDN’s lack of clarity and strategic direction in achieving its intended outcomes. This problem has been particularly severe in the case of a global program such as GDN, which has been founded and funded by the Bank and on which the Bank has served as a member of the governing board — thereby creating a tension for the Bank’s representative between responsibility to the organization and to the Bank as donor. The Bank’s engagement with GDN demonstrates the need for clear guidelines especially in regard to (a) standards of program design and performance to be expected in situations of continuing Bank support, and (b) the roles and responsibilities to be exercised by Bank program managers, task team leaders, and representatives on partnership boards. 1 1. Program Objectives, Activities, and Costs Program Origin and Objectives 1.1 Origin. The World Bank conceived the idea of a global development network in the late 1990s as a response to a perceived paucity of support for development research emanating from developing and transition countries. Following a series of meetings and consultations on researchers’ needs and how to meet them, the World Bank formally launched GDN in 1999 at an international conference in Germany. Established initially as an internal unit of the Bank, GDN was incorporated in March 2001 as an independent, non- profit institution, based in Washington, DC. In 2005, GDN relocated to New Delhi, India, and changed its legal status to that of an international organization. The following timeline highlights the key events in its organizational history. Table 1. GDN Timeline 1997 The World Bank establishes, as a internal unit, a global network of researchers and policy institutions 1999 While still a unit of the World Bank, GDN is launched at its first annual conference, held in Bonn, Germany 2000 GDN launches its first Global Research Project, “Explaining Growth” 2001 GDN is incorporated as a nonprofit organization independent of the World Bank and establishes an office in Washington, D.C. 2001 GDNet is launched 2001 GDN holds its third annual conference, the first in a developing country 2005 GDN headquarters are moved to New Delhi, India 2006 GDNet moves to offices in Cairo 2007 GDN’s second president assumes office 2008 GDN becomes an international organization 1.2 Goal and Objectives. From the outset, GDN has defined its overarching goal broadly. As stated in its charter, “GDN, a worldwide association of research and policy institutions, promotes the generation, sharing and application to policy of multidisciplinary knowledge for the purpose of development.”1 Underlying this mission is the premise that better research from within developing and transition countries is necessary for better policy-making and that researchers in developing and transition countries need both financial and advisory support to increase the output of high-quality research relevant to key policy issues. 1. While this statement appears on GDN’s Web site and in some other formal documentation, variations of it also exist, reflecting a lack of clarity in its outcome objectives, which is discussed below in the sections on GDN’s relevance and its effectiveness. In particular, while some statements insert capacity building into GDN’s overarching goal, others refer to capacity building as one of three program objectives. 2 1.3 To achieve this overarching goal, GDN has three core program objectives:  Generate high-quality, policy-relevant research in developing and transition countries.  Build research and policy outreach capacity among researchers in those countries to improve the quality and expand the policy influence of their work on a national and international level.  Promote greater linkages between researchers and the policy process to foster effective, evidence-based policy-making.2 Activities 1.4 GDN sponsors five core activities, each intended to promote the three program objectives to varying degrees. The five activities are:  Regional Research Competitions (RRCs)  Global Research Projects (GRPs)  Global Development Awards and Medal Competitions (GDAMCs)  Annual Conference  GDNet While GDN has undertaken these five activities from the outset, it has altered their level and share of expenditures over time. Specifically, the budget expenditures for GRPs and GDNet have increased, while expenditures for the Annual Conference and GDAMCs have declined slightly and those for the RRCs declined and then reverted to a higher level in recent years, as shown in Figure 1.3 THE REGIONAL RESEARCH COMPETITIONS (RRCS) 1.5 GDN provides grants annually to eight Regional Network Partners (RNPs) to support the work of development researchers in developing and transition countries. In turn, the RNPs, which are independently functioning entities, award sub-grants to researchers and institutions in their respective regions on a competitive basis.4 GDN’s stated aim of these annual RRCs is to support policy-relevant research projects that advance new knowledge, increase individual research capacity, and help to discover new talent.5 To foster the sharing of the resulting research, GDN requires all working papers from the research it helps fund to be posted on GDNet. 2. As each of these objectives has been stated variously in different documentation, the wording here is a best attempt by this review to capture GDN’s intent. 3. In the last 18 months GDN has also begun to develop two new sets of activities: (1) a portfolio of training programs to build individual skills and strengthen the link between research and policy and (2) a program of institutional capacity building in Africa. While these efforts are too new to review for their effectiveness, the significance of GDN’s expansion in these directions is discussed in the sections on relevance and efficiency. 4. See Table 2 on p. 7 for a list of the RNPs and their shares of the allocation of RRC funding from GDN in fiscal year 2007. 5. GDN Web site (www.gdnet.org). 3 Figure 1. GDN Expenditures on Five Core Program Activities, FY02–08 US$ Millions 3.5 Global Research Projects 3.0 2.5 2.0 Regional Research Competitions 1.5 Annual Conference 1.0 GDNet 0.5 Global Development Awards 0.0 FY02 FY03 FY04 FY05 FY06 FY07 FY08 Source: Annex E, Table E-1. 1.6 From inception through FY08, GDN has contributed in this way to over 800 research grants across developing and transition regions. This activity, funded largely by contributions to GDN from the World Bank, has accounted for some 29 percent of total GDN program expenses for its five main activities over the period FY02–08. 1.7 GDN uses a decentralized approach for the management of its RRCs. While it requires that regional partners follow standards of transparency and a competitive process in the awarding of grants, each RNP has latitude in selecting thematic emphases and setting procedures and guidelines for the use of the GDN-provided funds. This approach has resulted in wide variation in the range of topics addressed and the nature and extent of mentoring and other capacity-building support provided to researchers. For example, rather than making research grants to individuals as most of the RNPs do, LACEA, which is an association of individuals, uses its allocation of GDN funds to support a range of services considered beneficial to the research work of their individual members, such as an annual conference, topic-specific smaller meetings of researchers, and publication of a journal aimed at directing research to policymakers. In another example, CERGE–EI provides varied kinds and intensity of mentoring and feedback to researchers based on their levels of experience. 1.8 A new, related activity was launched in 2008 in which two or more RNPs collaborate in conducting joint research on a topic of common interest. An additional allocation of $500,000 has been earmarked for the initial round of these Inter-regional Research Projects (IRPs). As of the writing of this report, four proposals had been received. 4 THE GLOBAL RESEARCH PROJECTS (GRPS) 1.9 These projects — which GDN organizes and funds centrally — examine aspects of development through a comprehensive and comparative approach. Among the topics that GRPs have addressed are as follows: the impact of rich countries’ policies on poverty, varieties of governance, migration’s economic and social impacts, and bridging research and policy. Typically, the projects undertake case studies in multiple countries and involve teams of 30 to 80 researchers from across two or more regions. The projects have tended to be financed by one or two donors with earmarked funds. In total, they account for some 30 percent of GDN’s total budget expenses from FY02–08. Annex B provides a brief description of each of the GRPs undertaken to date. 1.10 Since 2007, GDN has engaged in four “Strategic Research Partnerships” that it views as consistent with its core program objectives and for which it serves as the implementing agency or partner agency. These Strategic Research Partnerships, which are all ongoing, have been initiated either by GDN or by a specific funder or group of funders. Examples of the focus of these new efforts are: institutional capacity building in Africa, improving public expenditure analysis, and promoting development impact evaluation. In the case of the latter example, GDN has established a partnership with the multidonor funded International Initiative for Impact Evaluation (3ie) and is providing it with an initial home. Although these Strategic Research Partnerships are too new to assess, it appears that while they offer potential opportunities to advance GDN’s objectives, they also pose a risk of diverting GDN from its other core activities. So far, GDN’s Board has endorsed some initiatives and rejected others which it viewed as too far afield from the program’s core business. But the Board has not taken the further step of articulating guidance or criteria for entering into such partnerships. GLOBAL DEVELOPMENT AWARD AND MEDAL COMPETITIONS 1.11 Launched in 2000, the GDAMCs provide about $300,000 annually in awards to researchers and development practitioners from developing and transition countries. There are three categories of annual awards: (1) funding for outstanding research in any of five pre- selected topics, with a first prize of $30,000 and second prize of $5,000; (2) medals for research to individuals under the age of 46, with a first prize of $10,000 and second prizes of $5,000 in each of the five topics; and (3) awards of varying sizes for innovative development projects with potential for major impact in local communities, managed by institutions in developing or transition countries. GDN intends these awards to be a main means of identifying and showcasing new talent and advancing multidisciplinary development research. With support from the Government of Japan through its PHRD trust fund administered by the World Bank, this activity has accounted for some 7 percent of total GDN program expenses from FY02–08. ANNUAL CONFERENCES 1.12 A leading activity in terms of budget resources and staff time, the Annual Conferences provide a global forum for the exchange ideas on a broad topic. For example, in 2007 the Annual Conference focused on “Security and Development” and in 2008 on “Natural Resources and Development.” The Annual Conferences also provide for networking among researchers from all regions of the world, policymakers, civil society representatives, and donors. They provide an opportunity for grantees and award-winning researchers and 5 practitioners to showcase their work. In addition, they serve as a setting for research workshops, meetings of the board of directors, donors, and RNP coordinators — separately and together. 1.13 Held each year in a different region, these typically involve about 500 participants. Funding for the Annual Conferences, which has come from a variety of donors over the years, accounted for some 20 percent of total GDN program expenses from FY02–08. GDNET 1.14 GDN launched this Web-based activity to serve three purposes: (1) assist researchers and research institutions in developing and transition countries to link to a global network to showcase their work; (2) provide them access to resources to support their work; and (3) help build the communications capacity of research institutions to enhance the policy influence of research. To meet these aims, GDNet serves as a depository of research output from the developing world. It provides registered developing country researchers with online access to a variety of tools and services, notably free access to databases and journals, information on research funding opportunities, and a researcher profile. In addition, in recent years GDNet has offered training and professional support in knowledge management to interested RNPs and helped with the establishment of “regional windows” that target information to regional constituencies. 1.15 In its early years, GDNet’s technical platform was handled out of the Institute for Development Studies (IDS) in Sussex to allow for a quick start-up of the online services by making use of existing capacity. In 2004, GDNet moved to Cairo, residing under the legal auspices of GDN’s regional network partner, the Economic and Research Forum (ERF). GDNet now has a staff of six: the Director, four Regional Coordinators and a Coordinator of Online Services. Funded by the United Kingdom, the Netherlands and the World Bank, this activity accounted for some 13 percent of total program expenses from FY02–08. Governance and Organization 1.16 GDN is designed and operated as a “network of networks,” as distinct from a central research organization. This organizational feature is manifest in its principles and structures of governance and management. 1.17 Principles of governance. The documents establishing GDN as an independent organization in 2001 specify that the network will be guided in all its activities and decisions by five principles of governance:  Independence: GDN will not be influenced in any of its activities and decisions by political considerations.  Openness: GDN shall operate in a transparent manner and be receptive to the views of its constituency.  Effectiveness: GDN shall function in a manner consistent with the efficient realization of its purpose.  Democracy: GDN shall strive for broad representation and participation.  Plurality: GDN shall encompass a diversity of disciplines and paradigms. 6 1.18 Organizational Structure. GDN’s structure comprises an Assembly, a Board of Directors, and a Secretariat (with a President and staff), with composition and responsibilities as follows.  Assembly: the Charter establishing GDN as an international organization introduces an Assembly comprised of any State or public international organization that is a signatory to the charter agreement. The Assembly has broad oversight authority, notably to ensure that GDN fulfills its purpose and to dissolve GDN and distribute its assets. As there were only three states that had ratified the charter as of the end of 2008, the Assembly has not yet been formally constituted or met and is not discussed further in this review.  Board of Directors: As the central decision-making body of the network, the Board consists of not less than 16 and not more than 20 directors, who are selected for a term of three years. It is largely a constituency-based board, currently comprised of 18 directors as described in more detail in chapter 4 (para. 4.3).The Board, which meets a minimum of twice a year, is responsible for appointing the President of GDN and directing the general operations of GDN, including setting the network’s strategic direction, approving its menu of activities and related budget, and overseeing its financial management and evaluation of performance.  Secretariat: As the central management and administrative hub of the organization, the Secretariat organizes and/or coordinates the implementation and monitoring of most GDN activities. It also manages the financial resources. The Secretariat is led by a President and currently has a staff of some 30 professionals and interns. Headquartered in Delhi, the Secretariat has additional offices in Cairo and Washington, D.C. In 2007, GDN experienced a leadership transition from its first to its second president; and January 2009 the second president announced his plans to resign (effective March 1, 2009) in order to assume a new position. 1.19 Regional Network Partners (RNPs). GDN works with 11 regional partners in all regions of the world, of which 8 are in developing and transition regions and 3 in developed regions (identified in Table 2). The 8 RNPs in developing and transition regions manage the annual regional research competitions. GDN has also engaged the RNPs in the identification of participants for global research projects and Annual Conferences. Recently, the RNPs have organized sessions at the Annual Conferences that showcase research on topics of importance to their regions and 8 of them currently manage regionally focused Web sites (or “regional windows”) in cooperation with GDNet. 1.20 Each RNP is itself a network (or association), linking numerous researchers and institutions within its region, supporting local research, and facilitating members’ contacts with policymakers. Each is also a legally separate entity that functions independently of GDN. Though the RNPs are interconnected through GDN activities and GDN is involved in some of the RNPs’ activities, the regional networks remain self-regulating bodies, and their structures, governance, and scope of activities vary. For example, the Cairo-based Economic Research Forum (ERF) comprises individual researchers in the Middle East, while the East Asian Development Network (EADN) is constituted of research institutions. While some served their regional constituencies for a decade or more before becoming affiliated with GDN, others were established by GDN to give its program global scope. 7 Table 2. GDN Grant Support to RNPs as Percentage of Their Total Annual Revenues, FY2007 Total annual revenues, GDN grants, Date all sources including RRC (B) as % Regional Network Partner founded (A) (B) of (A) AERC 1988 $12,447,656 $360,000 3% (Sub-Saharan Africa) CERGE–EI 1991 $1,173,514 $354,500 30% (Eastern and Central Europe) EADN 1998 $260,000 $260,000 100% (East Asia) EERC $1,469,479 $291,550 (Commonwealth of Independent 1995 20% States) ERF 1993 $3,851,999 $360,530 9.5% (Middle East and North Africa) LACEA 1992 $229,057 $187,000 82% (Latin America and Caribbean) SANEI 1998 $350,500 $350,500 100% (South Asia) ODN 2003 $100,000 $100,000 100% (South Pacific) EUDN 2000 NA 0 NA (Western Europe) GDN–NA 2002 NA 0 NA (North America) GDN–Japan 2000 NA 0 NA Total $19,922,205 $2,264,080 11.4% Source: GDN Secretariat. Note: It should be noted that the amounts of budget expenditures for RRCs in Figure 1 and Annex Table 6 are larger than the grant allocation figures in the present table because the former include other expenditures incurred such as travel expenses of GDN staff to RNP workshops and conferences and the allocation of GDN staff time for coordination of the RRCs and other connections with RNPs. 1.21 Consequently, the share of GDN grants in the total annual revenue of the RNPs varies considerably — ranging from as little as 3.0–9.5 percent for two RNP in existence prior to the establishment of GDN to 100 percent for three RNP established by GDN. The proportion of annual GDN grants to each RNP’s total annual revenues is shown in Table 2. 1.22 Other Partners. GDN’s donors and several specialized institutions comprise two additional groups of GDN partners. Donors as a group meet once a year alongside the Annual Conferences with the Board, the Secretariat, and the RNPs. The major donors and their level and areas of support are discussed in detail below in the section on GDN’s financial sustainability. The other “global partners” engage with GDN in a variety of ways, such as publication of GDN-supported research, technical support of its Web site, and collaboration on research, workshops, and conferences. 8 2. The 2007 Independent Evaluation Process 2.1 In 2007, a small team of private consultants conducted an independent evaluation of GDN’s relevance, efficacy, and efficiency.6 Their assessment built on a 2004 independent evaluation focused largely on GDN processes — both of which were commissioned by GDN’s Board.7 In addition, the 2007 evaluation drew on several other assessments of the outcomes of specific elements of GDN’s program that were internally conducted or internally commissioned. A separate independent evaluation of GDNet was commissioned by one of its main supporters, DFID, and completed in 2008 after the completion of the broader 2007 evaluation.8 (These evaluations appear in the list of references for this review.) 2.2 For a relatively young organization, this record of evaluation is notable. Also notable is GDN’s transparency and responsiveness with regard to the evaluations’ findings and recommendations. All assessments are posted on the GDN Web site, along with the management responses to the two major independent evaluations. 2.3 The 2007 evaluation was conducted in an appropriately independent manner. The GDN Board commissioned and approved the terms of reference for the 2007 evaluation, which were drafted by the Secretariat. A combination of Board members and Secretariat staff selected the contractor from several responses to a competitive call for proposals. The evaluation cost about $70,000, which was covered by GDN’s resources. 2.4 The program had no previous contact with the selected contractor. Nor had any of the investigators been involved in GDN activities. Conclusions in the evaluation report were arrived at independently, and, according to the contractor, there was no manipulation of findings or recommendations by GDN management or Board. Both the staff and Board gave feedback on a first draft that the evaluation team took into account at its discretion in the final drafting of its report. 2.5 While GDN was supportive of the evaluation effort and responsive to requests for information, not all information sought was readily available in usable form.9 In addition, the evaluation team found no results chain or logical framework with measurable indicators on which to base its assessment of GDN’s effectiveness. 6. The contractor was Marc D. Shapiro, principal investigator and Managing Partner of MDS Associates (a U.S.-based firm). He was assisted by Savi Mull, Senior Research Associate, and Tina Khanna, Research Assistant. Marc D. Shapiro, Savi Mull, and Tina Khanna, “Evaluation of the Global Development Network, 2007,” December 2007. 7. H. Peter Muth and Frederick H. Gerlach, “Independent Evaluation,” March 16, 2004. The two main independent evaluations are referred to herein as the 2004 evaluation and the 2007 evaluation. 8. This evaluation of GDNet was being finalized at the time of the writing of this IEG review and its early emerging findings and recommendations were shared with this reviewer and are reflected in this report. 9. Interview with Marc Shapiro, October 17, 2008. 9 Design and Methodology 2.6 The 2007 evaluation sought to be a comprehensive assessment of GDN’s relevance and its program effectiveness relative to its stated objectives. Specifically, the evaluation attempted to measure the outcomes, relevance and effectiveness of GDN activities against the program’s core objective of building capacity to generate, share, and apply knowledge. It examined, in limited fashion, the cost-effectiveness and sustainability of GDN. It also examined GDN’s operational effectiveness as a global network of research and policy institutions by giving attention to its interaction with its regional network partners, governance, and internal operational performance. 2.7 In undertaking this assessment, the evaluation used a combination of qualitative and quantitative methods, including:  Document reviews  Field visits to GDN offices and workshops  Interviews (some 90 face-to-face and by telephone)  Multiple surveys, notably of stakeholders (board members, donor representatives, heads of regional network partners, and Secretariat staff) and end-users (regional and global research grantees, awardees, and researchers registered on the GDN Knowledgebase).10 2.8 A notable feature of the evaluation was its effort to grapple with the well-known challenge of judging the effectiveness of program activities intended to build research capacity. To make that judgment, the evaluation used three measures:  It looked at research output through survey questions about the amount of research generated with GDN funds and reviews of the publishability of papers.  It piloted an effort to assess incremental improvements in skills of grant recipients attributable to GDN and a longitudinal review of change in quality from the proposal stage to the output stage of research projects.  It applied a statistical method to estimate comparisons between treatment groups (grantees) and control groups (a broader constituency of survey respondents) who shared similar characteristics. 2.9 Yet, the evaluation acknowledges serious limitations in the application of the latter two methods so it did not emphasize findings from them. Instead, it encouraged GDN to devote further effort to gauging the effectiveness of its research capacity building, noting that such an effort is a challenge shared widely by research capacity-building programs on a national, regional, or global level. 2.10 This IEG review, while acknowledging the broad scope and objectivity of the evaluation, notes three weaknesses. First, the evaluation had no benchmarks by which to assess GDN’s progress in achieving its program’s core outcome objectives. Second, because 10. Typical of experiences with online surveys, response rates to the stakeholder survey were much higher than for the end-users’ survey. Indeed, as acknowledged by the evaluation, the response rate of end-users is sufficiently low to suggest that responses are biased toward those most satisfied with GDN activities. 10 of limitations in the data available from GDN and the methods used in the evaluation, the assessment of effectiveness is not objective. It relies primarily on self-reporting by targeted beneficiaries of such things as professional capacity built and outreach to policymakers. Third, despite the importance GDN attaches to the conduct of high-quality research and its role in informing policy, the evaluation did not engage a professional economic researcher as part of the evaluation team and attempt to examine at least a sample of reports/publications that had been produced to determine the quality and newness of the GDN-funded research and the policy relevance of the work. These are significant shortcomings which leave incomplete the assessment of GDN’s effectiveness. Findings, Recommendations, and GDN’s Response 2.11 While acknowledging limitations to the available evidence, the evaluation concludes that its overall findings are “generally positive.”11 More specifically, its main findings indicate that:  Key stakeholders perceive that the program overall is “providing mostly unique, relevant, and valuable services.”  GDN has made some progress since the previous process evaluation in certain areas. For example, GDN has been able to diversify and increase funding in recent years after having failed to meet its ambitious funding targets during FY05–07.  Qualified evidence exists of research capacity built and knowledge created from GDN-funded activities, and limited, anecdotal, evidence of increased outreach to policymakers.  Though it would be too ambitious for GDN to aim to influence policy, it can claim to be building a base of policy-relevant literature and capacity of future researchers who can inform better policy-making.  Considerable work remains to be done to strengthen both program effectiveness and governance. 2.12 While concluding that overall GDN has been moving in the right directions, the evaluation offers 26 “priority” recommendations in three areas: (1) objective and reach, (2) outcomes and effectiveness of activities, and (3) governance, organizational performance, and financing. Notably, it recommends that GDN further clarify the scope and priorities of its objectives, strengthen its capacity-building and policy outreach efforts, improve its working relations with its regional partners, and bolster the functioning of its Board. (A full list of the priority recommendations is presented in Annex C.) 2.13 Overall the evaluation was well received by GDN’s main stakeholders. Interviewees for this IEG review generally found it to be fair and constructive. Most said that it showed progress since the 2004 evaluation, and, at the same time, the evaluation highlighted important issues that had not been adequately resolved in the intervening years. Because of the timing of the evaluation — in the period of GDN’s first leadership change since its founding — the findings were seen by the Secretariat as valuable input into the strategic thinking of its new leadership. In fact, many of the evaluation’s main points proved to be 11. 2007 evaluation, p. i. 11 similar to ideas under consideration in the preparation of the ongoing strategic review. Consequently, GDN’s management and board developed an action plan for addressing the bulk of the evaluation’s recommendations and incorporated the plan into the final version of the strategic review.12 3. The Effectiveness of GDN 3.1 This chapter presents IEG’s review of GDN taking into account the findings for the relevance of its objectives and design for meeting an internationally recognized need, the extent to which its activities have advanced its objectives, and the appropriateness and efficiency with which it has governed and managed its program. Overall, the IEG review finds that while GDN’s relevance is moderately strong and its record of outputs is strong, its achievement of outcome objectives is moderate, with notable shortcomings. Relevance 3.2 The review of GDN’s relevance draws on evidence from the three phases of the organization’s history: the consultative phase prior to the launch of GDN; the start up of GDN as an independent organization; and the more recent phase of GDN following its relocation to New Delhi. Thus IEG’s review covers the maturation of GDN over almost 10 years. 3.3 GDN’s goal and objectives. The relevance of GDN’s overarching goal was high at the outset of the program and remains high today. Its focus on research generated from within developing and transition countries is widely viewed as responsive to a need for such research to inform national policies and strengthen the voice of these countries in global policy-making. Historically, most development research has come from outside developing countries. Moreover, funding for developing country research is in short supply: little comes from governments or other domestic sources and international research support is most often for inputs into international policies and practices. 3.4 Prior to the launch of GDN, the World Bank held a series of consultations and conducted a survey of some 512 research institutions worldwide.13 Both the consultations and survey revealed strong support for a global network focused on development research from developing and transition countries and its translation into policy. More specifically, the views noted an increasing gap between the demand for policy-relevant research in developing countries and the supply of funds for this purpose. These views also emphasized that effective support for building a base of high-quality research would go beyond adequate funding. Such support would also require mechanisms for mentoring, training, systematic use of networks of researchers across countries and regions, and enhanced access to various kinds of online databases and other resources.14 12. The major action items identified by GDN’s management in its formal response to the 2007 evaluation are presented along with the list of recommendations in Annex C. 13. Erik Johnson and Diane Stone, “The Genesis of the GDN,” undated manuscript. 14. Survey cited in Ramona Angelescu and Lyn Squire, “Capacity Building and Policy Impact: The Experience of the Global Development Network,” paper presented at World Bank and CEU Conference, Budapest, June 12 3.5 The independent evaluations of GDN in 2004 and 2007 found that stakeholders continued to view GDN’s objective as highly relevant. As reported in 2004: “GDN’s programs and activities meet a clear demand of a global market for development-related knowledge….” The 2007 evaluation confirmed that all of GDN’s constituencies saw continuing value in a global network of development researchers and institutions. Overall, the evaluation concluded that GDN is “providing a highly valuable service of funding research by individuals in developing and transition countries” both regionally and cross- regionally. 3.6 Yet, this IEG review has been unable to find a clear and unambiguous definition of the “problem” for which GDN was established. GDN has a vision of good, evidence-based policy-making having an impact on the development of countries in all regions. It is also built on the rationale that there is need for more and better research from within developing and transition countries to inform national and international development policies. But its documentation is not clear about the extent to which the inadequacies it seeks to address relate to: (1) the volume of high-quality, policy-relevant research being generated, (2) the technical quality of the current research output, or (3) the policy relevance and outreach of that research. While all of these shortcomings are said to exist, GDN does not define, except at a high level of generality, the nature of the problem its program targets and where its specific comparative advantage as a global network lies is addressing the problem(s). 3.7 As a result, the translation of GDN’s overarching objective into its three program objectives has been a source of continuing debate among stakeholders and remains in need of clarification. Both the 2004 and 2007 evaluations found tension regarding GDN’s objectives, even though the more recent evaluation noted “somewhat greater agreement and lower tensions”15 among shareholders on the objectives. Specifically, although there is widespread agreement on capacity building as an objective, if not the primary program objective, GDN has lacked consensus among key stakeholders on three points:  The primacy of the capacity aim, especially in relation to the aim of generating high-quality research. For some, the two are so intricately related as not to be a problem; for others, there is something of a trade-off between capacity building and quality research output, which has not been clearly resolved in GDN’s statements of objective and related program planning.  The extent to which GDN’s research and support for research capacity building should have a multidisciplinary focus (rather than focusing predominantly on economics). While disagreements on this issue — both among board members and donors — appear to have subsided since the conclusion of the 2007 evaluation,16 interviews for this IEG review suggest that the current Board-approved stance (of 2005; and in Johnson and Stone. 15. 2007 evaluation, p. ii. 16. While the 2007 evaluation found “somewhat greater agreement and lower tensions regarding GDN’s objectives,” it noted that the “role of multidisciplinarity remains an area of disagreement, although no longer quite the source of tension as previously” (p. ii). 13 retaining economics as GDN’s “default discipline,” while giving importance to the selection of interdisciplinary research themes) is very recent and not sufficiently spelled out in the view of all who continue to worry about GDN spreading itself too thin.  How much emphasis to give to strengthening the policy outreach of researchers and research institutions. This remains the most unsettled issue. In the view of some, GDN should focus on building capacity and helping to generate high-quality research, which will find its way into policy because of existing high demand; in the view of others, GDN needs to give more direct support to policy outreach activities. 3.8 Design. Conceptually, GDN’s four design features are highly relevant to the pursuit of the organization’s overarching goal and program objectives, but these reveal substantial weaknesses when put into practice. The four design features, consistent with GDN’s objectives, are the following:  GDN activities are wholly focused on developing and transition country research. While many research programs that seek to expand the quantity and quality of development research, GDN’s program concentrates on enhancing the generation, sharing, and application of development research work from within the developing world.  The program is global in scope, involving researchers and partner institutions from all regions of the world. In the view of many, GDN’s global scope is its primary value. Some regional research networks existed at the time that GDN was founded and others were established subsequently (in part with the help of GDN). What GDN offers, uniquely, is a mechanism for connecting these networks and other researchers and fostering an exchange of knowledge and experience among the research communities. It also offers a global platform for increasing the exchange of ideas among developing country researchers and between them and others as well as raising the worldwide visibility of developing country research.  It is structured as a network of networks (as distinct from a global think tank). This feature recognizes that while there are large benefits to sharing knowledge and experience across regions, research capacity building and policy outreach support needs to be tailored to regional and country differences. While the alternative of GDN as a centralized global research institution that organizes and carries out its own research that could generate comparative and cross-country development research, it would not offer the same potential for building sustainable, policy-relevant research capabilities and voice throughout the developing world.  It facilitates a menu of activities as a way to achieve its three program objectives. GDN designed its set of activities on the grounds that no one action would be adequate to meet its core program objectives. Both the 2004 and 2007 independent evaluations of GDN and interviews for this review share this view and see all the activities as relevant to GDN’s purpose. 14 3.9 In practice, however, GDN’s design reveals three substantial weaknesses. The first weakness is the absence of a theory of change (or conceptual road map) and rigorous strategies for achieving each of its three program outcome objectives. This weakness is reflected in the apparent absence of a results framework linking inputs (resources) and outputs (activities) to intended outcomes or even intermediate outcomes (such as better research, enhanced capacity, and improved policy outreach). The 2007 evaluation notes that the absence of such a framework makes evaluation of GDN effectiveness difficult. More importantly, it makes it hard for GDN to make the most effective use of its resources, identify where donor support is most needed, and judge if it is being successful in advancing its outcomes objectives. 3.10 To date, GDN has put its time and attention into conducting a portfolio of activities on the grounds that these activities will, in varying ways, advance its three program objectives of generating high-quality, policy-relevant research, building enhanced research capacity, and linking that research to policy-making. But it lacks a strategy for achieving each of those program objectives in ways that builds on the state of knowledge about how to achieve these aims and that systematically integrates its activities into such strategies. For example, GDN sees its RRCs, GRPs, and GDNet activities as central to achieving its capacity-building and policy outreach objectives, but it does not systematically connect these activities to well-defined approaches for achieving the two objectives. 3.11 The second weakness is the imprecise configuration of its network structure. It is instructive that after almost 10 years of operation, the 2007 evaluation highlights as unresolved the issue of whether GDN is “a network of networks” or a central organization that provides grants to network grantees. Although GDN is built on the notion of a partnership among regional networks in the design and implementation of activities, the reality is that the operational roles and responsibilities of GDN’s Secretariat and the RNPs are not adequately delineated. 3.12 On the one hand, GDN has adopted a largely decentralized approach to the support of regional research and research capacity building, through its RNPs. This approach makes sense given the differences among regions in their research practices and capacities. While GDN requires that the regional partners follow standards of transparency and a competitive process in the awarding of grants, each RNP has latitude in selecting thematic emphases and setting procedures and guidelines for the use of the GDN-provided funds. However, the division of responsibility between the RNPs and Secretariat for quality control and monitoring and evaluation of the regionally based activities is not clear. This weakness is reflected in the shortage of information available as input into the 2007 evaluation’s assessment of the results of the RRCs. It is also reflected in the finding discussed below regarding regional variation in the adequacy of the mentoring and other capacity-building support provided to researchers as part of the RNPs’ management of their allocation of GDN resources.17 17. For example, rather than making research grants to individuals as most of the RNPs do, LACEA, which is an association of individuals, uses its allocation of GDN funds to support a range of services considered beneficial to the research work of their individual members, such as an annual conference, topic-specific smaller meetings of researchers, and publication of a journal aimed at directing research to policymakers. As 15 3.13 On the other hand, GDN takes a centralized approach to other of its other major activities, notably the GRPs and Annual Conferences. And the RNPs are of the view that their roles and responsibilities are not sufficient in the design and implementation of these activities to capture the interests and ongoing work of their regions. Despite this, GDN’s Board has recently “strongly emphasized that GDN is much more than a network of networks” and agreed that GDN should explore the possibility of undertaking some in-house research, “providing other activities are not hampered.”18 Where this exploration will lead and how it might involve the RNPs remains an open question. 3.14 The third and final design weakness is the underdevelopment of GDN’s networking services. GDN is based on the assumption that global efficiencies can be achieved through cross-regional knowledge-sharing and cross-regional capacity building, as well as shared efforts in setting research standards and mobilizing resources. Despite this, the services that GDN’s Secretariat has provided to realize these efficiencies have not been well defined nor, apparently, resourced. Three points illustrate this problem. The first point relates to the Secretariat’s role in relation to the varied institutional capacity of individual RNPs. Both SANEI and ODI have lagged other RNPs in their performance in managing RRCs. Yet, this has not been reflected in assistance provided to them by the Secretariat to build their institutional capacities; nor is the level of performance reflected in annual research grant allocations made by the Secretariat to the RNPs. The wide variation in the size and capacity of the RNPs and extent of GDN engagement with them raises questions of whether the organization’s strategy of globalization is paying off in terms of coverage and quality, and of whether its resources are spread too thin. Second, the Secretariat’s role in helping its RNPs mobilize resources appears to have been modest to date (leaving three RNP still 100 percent dependent on GDN funding) though this was one rationale for the formation of GDN. Third, the Secretariat’s role in ensuring adequate objectives-based M&E of core GDN activities is not clear — in particular, its role in ensuring that the necessary data is provided for assessing the outcomes (as distinct from the outputs) of the RRCs and GRPs. 3.15 GDN’s Strategic Review — prepared in 2007 under the leadership of its then new president and in light of the external evaluation — addresses both the need for a greater clarity of objectives and the strengthening of relations with RNPs. But the resignation of the previous president in March 2009 (after less than two years in that position) and his replacement in mid-August 2009 leaves open the question of how far and how fast any changes will be made. Efficacy 3.16 IEG’s review of GDN’s progress in achieving its objectives draws on the several internal and external evaluations of GDN undertaken over the period FY02–08 and interviews conducted in mid-FY09. Overall, the evidence on GDN’s efficacy from these sources is thin. It is based largely on self-reporting by grantees and other stakeholders (notably survey data and interviews) and focuses more on outputs than outcomes. The another example, CERGE–EI provides varied kinds and intensity of mentoring and feedback to researchers based on their levels of experience. 18. GDN documentation, January 2008. 16 limitations of the evidence reflect, in part, the inherent difficulty of determining the kinds of outcomes which GDN seeks, notably enhanced research capacity and policy influence. In addition, the limitations reflect GDN weaknesses in setting and tracking clear and monitorable outcome objectives; and shortcomings in the methodology of the recent evaluation, which chose not to use established approaches (such as tracer studies and quality reviews) to assess GDN’s capacity building and other aims. 3.17 Based on the available evidence, this IEG review finds that GDN has a strong output record in each of its five areas of activity. Evaluations also show some “moderate evidence” of achievements in advancing two of its three core objectives — enhanced research capacity and increased output of high-quality research (though no evidence of the policy relevance of this work); and very limited evidence of success in advancing its third objective of strengthening the link between research and policy-making. But the absence of specific GDN Table 3. Level of Activity Baseline Progress Target values Activity (fiscal year) (as of fiscal year) (FY09) Research Cumulative number of 1300 (FY05) 1960 (FY08) 2200 funded and mentored researchers Share of publishable 64% (FY02) 87% (FY07) 90% outputs Share of researchers 60% (FY05) 60% (FY07) 75% producing policy briefs Global Research Projects Total number of 4 (FY05) 6 (FY08) 7 projects Awards and Medals Number of award and 351 (FY02) 626 (FY08) 650 medal submissions Annual Conference Number of conference 400 (FY02) 500 (FY08) 600 participants Number of workshops 4 (FY02) 9 (FY08) 10 GDNet Web site traffic 30,000 (March 2004) 95,398 (June– 105,000 (June– November 2007) November 2008) Research documents 7,000 (March 2004) 13,677 (November 15,050 (November 2007) 2008) Researcher profiles 3,000 (March 2004) 6,925 (November 7,620 (November 2007) 2007) Source: DGF Progress Report, FY09 Note: Some baseline and progress indicators were collected in course of independent evaluations only. Going forward, the progress reports will need also to include total number of IRPs and number of training programs. 17 criteria and outcome targets renders these findings less robust than desirable. The remainder of this section elaborates on these findings by first reporting on GDN’s achievements activity-by-activity and second examining the combined effect of the activities on each of GDN’s three core objectives. ACHIEVEMENTS OF GDN’S FIVE MAIN ACTIVITIES 3.18 GDN increased outputs in all five areas of activity since becoming an independent organization in FY02, as shown in Table 3. 3.19 Researchers — the intended primary, direct beneficiaries of GDN’s efforts — have expressed high to very high ratings of overall satisfaction with these activities, as shown in Table 4. Yet, the evidence on the outcomes of each of these activities measured against GDN’s three program objectives is more modest. 3.20 RRCs. From inception through FY08, Table 4. Researchers’ Overall Satisfaction GDN has funded over 800 research grants with GDN Activities (n = 1,224) through RRCs. Overall, the efficacy of this Overall Satisfaction Rating Percent activity varies across regions and in terms of its contribution to each of GDN’s program Very low 1 objectives. The large majority of the GDN- Low 7 funded research has led to completed projects. Medium 38 Thus, the support has clearly added to the base High 39 of knowledge on development emanating from Very high 14 developing and transition countries, though the Total 100 2007 evaluation disputes the extent to which GDN can claim that this is new knowledge. Research grantees report “moderate direct effects” in enhancing research capacity at the individual level. In contrast, the 2007 evaluation found little corroborating evidence of capacity built from RRC processes as measured by quality improvements from proposals to research papers. It also found variation in stakeholder perceptions of the RRCs’ capacity-building impact across regions, seemingly correlated with the amount of feedback and mentoring provided to grantees in different regions.19 3.21 RRCs are also intended to improve the capacity of researchers to inform policy, but this has been a limited feature of most of the RRCs to date. Interviews for this IEG review reflect the prevailing support among shareholders for the RRCs, but indicate concern about the variation in effectiveness across regions. A main problem is the absence of information about the quality of the research and the extent it informs policy. 3.22 GRPs. So far, GDN has completed four GRPs, and, at the end of 2008, had four other GRPs ongoing.20 The efficacy of the GRPs varies across projects as well as across GDN program objectives. As described in Annex B, the four completed projects have varied in 19. 2007 evaluation, p. 12. 20. This is exclusive of the four Strategic Research Partnership projects that were in the early stages of start-up at the end of 2008. 18 size, time to completion, and level of output. So far, three of the completed projects have led to substantial output and dissemination of research in the form of papers and edited volumes. While the 2007 evaluation finds that GRP papers show greater originality, clarity, and effective methodology than RRC papers, it concludes that they are equally limited in their capacity-building effect. Stakeholders closest to the work of the GRPs are generally more positive than others about the projects’ effectiveness in building research capacity. Even so, they and other stakeholders see need for improvements in the design and implementation of projects in order to provide more technical input and training early in the project process, and suggest that GDN reduce the average size of the projects to make them more manageable. Most IEG interviewees were more critical of the GRPs, noting that they are, for the most part, neither high-quality research nor good capacity building. As one interviewee stated, although they are sold as capacity building, they provide for “capacity building in a hurry, which is not the right way.” 3.23 There is some perception that the effectiveness of these projects has been improving over time. For example, the recent project on health is seen by GDN leadership to embody what GDN has learned about how to make GRPs effective — notably a strong policy focus, a mix of more and less experienced researchers, and the active engagement of the RNPs. But the project is ongoing so there is as yet no indication of its actual success in informing policy. Overall, the experience so far leaves open the question of whether GRPs can give equal emphasis to the objective of capacity building, along with the generation of high-quality research and policy outreach. The GRP experience demonstrates GDN’s weakness in developing concise strategies for meeting the three program objectives and clearly aligning activities to those strategies. 3.24 Annual Conferences. As of 2008, a total of over 4,000 participants from some 100 countries had attended the eight Annual Conferences convened since 1999. About 70 percent of participants were from developing and transition countries. All categories of GDN stakeholders view the conferences as at least moderately useful in bringing researchers together across regions and giving them the opportunity to increase their own visibility. More specifically, survey respondents who have attended at least one conference see their value, in descending order, in providing opportunities for making presentations, enhancing research skills and knowledge, providing feedback on research, and increasing the policy relevance of research. Disagreement is stronger on the quality of the conference papers and proceedings and the extent to which they offer intellectual value.21 Some think the capacity building (as well as cost-efficiency) of the Annual Conferences might be increased by focusing less on the showcasing of “big names” from the global research community and more on organizing sessions to provide feedback on the work of GDN-funded researchers. But others worry that a greater focus on capacity building would not attract high-quality researchers to the meetings. “The conflict demonstrates the challenging trade-offs in effect implicit in GDN trying to pursue multiple objectives for the conference.”22 Interviewees for this IEG review share the view that the Annual Conferences have been more useful in providing a networking 21. The 2007 evaluation makes a comparison with another professional conference that took many years to become consistently strong in quality 22. 2007 evaluation, p. 22. 19 function than feedback or policy relevance, and share the view that the ambiguity in the purpose and design of the conferences impede their effectiveness. 3.25 GDAMCs. Nearly 4,300 individuals from over 100 countries have participated in this competition to date. In 2007, the competition attracted more than 600 applicants. There is limited evidence on the efficacy of this activity. GDN has provided quantitative data on the number of submissions for each year’s set number of awards and medals. It also emphasizes that the awards are the main way that GDN advances a multidisciplinary orientation to development research and seeks to advance both woman and younger researchers. But this IEG review found no information on the outcomes of those awards in terms of output or policy outreach from research awards, nor hard evidence on enhanced career development of awardees. 3.26 GDNet. This Internet activity is one of the two GDN main activities aimed primarily at increasing knowledge sharing. Notably, it now provides online access to some 14,000 research papers from developing and transition countries. It also provides a package of online services for developing country researchers and research institutions and seeks to assist them in better communicating their research to academics and policymakers. Thus, it aims, like the other activities, to advance all three of GDN’s core objectives. All groups of stakeholders surveyed in the 2007 evaluation rated the activity as moderately to very valuable, while recommending that it could be made more user-friendly. Some two-thirds of survey respondents who are GDNet-registered researchers found the activity of high value for the package of services it offers (such as its information on funding opportunities and ongoing research). Even so, they also rated it behind other search engines or their own institutions’ Internet services for access to journals and other research output. Most researchers also found GDNet more valuable in increasing their knowledge of their own subject area than in enhancing outreach and building contacts with other researchers or policymakers. Still, the 2007 evaluation found that stakeholders, overall, lack familiarity with GDNet. The evaluation therefore concluded that GDNet needs to increase its visibility to reach its full potential. It also concluded that if GDNet wants to continue to “differentiate itself as adding value through aggregating information on development from the southern perspective in ways not available otherwise, it has a difficult task ahead to do so.”23 In addition, it noted that as a tool to increase the policy communication and influence of developing country researchers, GDNet has a largely unrealized potential. 3.27 The findings of the DFID-commissioned evaluation of GDNet are consistent with those of the 2007 evaluation, but provide a more comprehensive assessment that makes three main points:  GDNet is providing a well-used and valued service that provides research funding information, access to online journals and data, communication among peers, and access to unpublished research. But it is much less highly valued as a unique service for the reasons for which it was primarily designed — that is, to support knowledge generation through access to information and knowledge resources. While it devotes much of its time to this function and exercises good quality control over its 23. 2007 evaluation, p. 24. 20 knowledge base of developing country papers and articles, it needs to develop a strategy to provide more distinctive value added.  While it aims to be an instrument to advance GDN’s objective of informing policy, little of what it currently does takes it outside the research domain. As an intermediary, the policy-relevance of its online information is limited by the policy relevance of the research that comprises its knowledge base. To fulfill its policy purpose, it needs to move from a passive to an active approach based on a clear theory of change and strategy on how research can better inform policy.  An important step in improving its usability would be greater audience segmentation of its information and services, and a strong focus on its recent innovation of regional windows developed in coordination with GDN’s RNPs.24 3.28 IEG interviewees share the view that GDNet is a key tool for advancing GDN’s core objectives, but that it has not yet fully developed its unique mission. This development would seem to require closer integration of GDNet and other GDN activities and more investment in GDNet’s operation. It is a positive sign, therefore, that in its management response to the external evaluation, GDNet has launched a redesign of its Web site, signaled its intention to continue expanding the online portfolio of services, and most importantly committed to the development of a strategic plan.25 3.29 Building on the activities’ record. GDN has developed a set of five core activities each of which is meant to advance all three of GDN’s core objectives though to varying degrees. Overall, it is on track in delivering annual outputs of these five main activities. Notably, GDN is helping fill a widely perceived need for support of more and better developing country research, facilitating increased cross-country analysis, and promoting increased communication among and visibility of developing and transition country researchers. This record led GDN’s own Strategic Review to conclude that, “GDN is now implementing comfortably its five core programs and can therefore consider expanding its activities.” In particular, as reported in Box 1, GDN is now planning to simultaneously develop a new portfolio of training programs linked to basic research skills, strengthen its efforts to strengthen the link between research and policy, and support the strengthening of research institutions.26 While it is too soon to evaluate any of these plans, it is apparent that the plans represent a substantial increase in the level and scope of GDN activity. Thus, a key question for GDN going forward is, how to ensure that the expanded portfolio reinforces rather than distracts from its original core activities in contributing to the achievement of GDN’s core objectives. OVERALL ACHIEVEMENT OF CORE OBJECTIVES 3.30 In comparison to GDN’s success in implementing a menu of activities, its record of advancing its three program objectives is far less robust. IEG’s review reveals an overall record of moderate progress on generating new policy-relevant research and enhanced 24. Forthcoming. 25. GDN Progress Report, July–December 2008. 26. GDN Strategic Review, p. 23. 21 Box 1. GDN’s Plans for Expanding Its Activities Since 2007, GDN has begun to develop initiatives for expanding beyond its five core activities in a several ways. As reported by GDN, its main “emerging activities” are:  Regional training program: Based on a survey conducted jointly by GDN and its RNPs that revealed constituency interest in research skills training, GDN supported training pilots carried out by three RNPs (CERGE–EI, EERC, and LACEA). Evaluation of the pilots by an external consultant concluded that GDN was well positioned to meet a growing demand for training by offering training activities that would complement other such activities in its partner regions. GDN is therefore planning to launch a scaled-up Regional Training Initiative for a period of two years starting in 2009 subject to a decision of its Board in February 2009.  Increased policy outreach. Building on consultations around its Bridging Research and Policy GRP, its Strategic Review, and its Management Response to the 2007l evaluation, GDN has launched a project in partnership with two RNPs (SANEI and AERC), funded by the Gates Foundation, to promote evidence-based policies in agriculture and trade. In addition, following a Board endorsement in July 2008 of a “concerted effort” to mainstream policy in all GDN activities, the Board has enlisted three former government officials to advise it on a strategy for enhancing the policy-relevance of GDN’s work.  Support to research institutions: Two donor-initiated projects move GDN from its current focus on supporting individual researchers to supporting the strengthening of research institutions. With support from DFID, GDN launched a project in December 2008 aimed at strengthening the analytic underpinnings of national policy debates on public expenditure management and impact on public service delivery. The project, begun with five initial partner institutions, seeks to engage a peer group of up to 20 institutions. GDN will partner with the AERC in a second project, supported by UNDP, to enhance knowledge management capacity in African public policy institutions and networks with a particular focus on the poverty and hunger MDGs.  International Initiative for Impact Evaluation (3ie). This multidonor funded initiative aims to provide and summarize evidence on what works and why in programs intended to improve peoples’ lives. GDN and 3ie have formed a partnership for a period of years. 3ie will operate as a grant program within GDN that finances impact studies and other evaluations in developing countries. GDN will serve as its temporary host institution, providing office space, as well IT and other administrative support, for some 3–5 years as 3ie matures and becomes a wholly independent operation. Source: GDN documentation and interviews. research capacity, and little progress on informing policy. It also sees a lack of strategic clarity in the pursuit of each of the program objectives as a major reason for the moderate progress to date in advancing GDN’s aims. 3.31 The generation of high-quality, policy-relevant research. GDN’s strongest record is in supporting increased amounts of development research from within developing and transition countries. GDN-funded research has also led to an increase in the dissemination of that work through papers, journal articles, and books. For example, GDN’s publication series, designed to give voice to researchers from developing and transition countries, has released 13 books, one of the earliest being an edited volume from GDN’s first GRP on “explaining growth.” In partnership with the series’ publisher, GDN is able to makes these works available for half price to individuals registered on its Knowledge Base and copies of 22 all publications under the series will soon be available for free and electronically to registered users of GDN’s Web site. In addition, GDN has expanded access to the work of the researchers it supports through its Web site and through publications. Each of these processes has quality-control mechanisms in place. But there is not sufficient cumulative evidence to be able to rate the quality or the policy relevance of the research output or to know if the GDN- sponsored research has improved in quality and policy relevance over time.27 3.32 Research capacity building. Combined findings on the effectiveness of GDN’s five activities indicate that, overall, GDN has been moderately effective in building research capacity at the individual level. In addition, survey data from 2005 and 2007 suggest a modest increase in GDN’s contribution to research capacity building over time. For example, grantees report stronger publication records emanating from their grants and prizes in 2007 as compared to 2005, as shown in Figures 2 and 3. 3.33 This data on publications self-reported by grantees is corroborated by an assessment conducted by independent referees of a selected set of proposals and papers funded by GDN. This assessment shows that as of 2007 only a minority of completed studies are deemed “not publishable” by the experts.28 In addition, GDN progress reports show that the “share of publishable outputs from GDN-funded research rose from 64 percent in FY02 to 87 percent in FY07. 29 Figure 2. Research Output (as percentage of the grantee survey respondents) 80% FY05 FY07 70% 60% 50% 40% 30% 20% 10% 0% Published at least one Published at least one Producted at least one article chapter in a volume working paper Source: GDN, “Management Response and Actions,” January 2008. 27. Some interviewees for this review also expressed concern that there is not evidence on the extent to which the research would have been produced without GDN support, but this counterfactual assessment is hard for any evaluation to make. 28. The GDN-funded proposals and papers reviewed were from one of GDN’s RNPs, CERGE-EI, and the recently completed GRP — The Impact of Rich Countries’ Policies on Poverty. 29. GDN’s management response to the 2007 evaluation notes, as a caveat, that a longer period has passed since GDN’s inception for publications to materialize, which may explain in part the considerably higher values reported in the 2007 survey. 23 Figure 3. Assessment of Publishability (as percentage of selected studies assessed by independent referees) 80% FY05 FY07 70% 60% 50% 40% 30% 20% 10% 0% Refereed Journal Others Not publishable Source: GDN, “Management Response and Actions,” January 2008. 3.34 Other survey findings indicate that GDN’s target beneficiaries (i.e., the RRC and GRP grantees and GDAMC winners) on average give the program credit for contributing, to a moderate degree, to their research capacity and career advancement. Findings also indicate an increase over time in beneficiary ratings in some areas, as shown by the shading in Table 5. Yet, the increases occur in only slightly more than half of the areas of impact and, for the most part, average values are in a middle range of 3 on a 5-point scale (1 being “very low” and 5 being “very high”). 3.35 Moreover, the 2007 evaluation found no corroborating evidence of positive capacity- building effects from RRCs or GRPs using other evaluative measures such as identifiable improvements in products from successive stages of proposal to final report. It also noted that the mentoring process during the course of the RRCs and GRPs — seen by GDN as a major feature of its capacity-building effort — had declined since 2005 and was of widely varying effectiveness across regions. This overall finding is consistent with the dominant thrust of responses about GDN’s capacity-building effectiveness in interviews for this IEG review. Notably, while all interviewees cited capacity building as a key program objective, if Table 5. Beneficiaries’ Ratings of GDN Contributions to Various Dimensions of Capacity Building Group Knowledge of Research Career Visibility Contact with subject contacts advancement policymakers RRC and GRP 3.8 4.0 3.3 3.3 2.8 grantees GDAM winners 3.4 3.7 3.4 3.2 2.9 GDNet 3.7 3.6 3.5 3.5 – registrees Source: GDN Management Response, January 2008, p. 4. Note: Bold numbers are higher values than in the 2005 pilot internal capacity building evaluation. Ratings: 1 = very low; 5 = very high. 24 not the primary objective of GDN, several stated that the capacity-building effort is not systematically designed. 3.36 As indicated, GDN is now planning to expand its capacity-building effort beyond research support for individual researchers by initiating a research training program and helping to strengthen research institutions (Box 1). In the view of this IEG review, if GDN is going to move in one or both of these expanded directions, it ought to do so not as a stand- alone measure, but as part of the development of a comprehensive capacity-building strategy. 3.37 One of the apparent reasons for GDN’s moderate effectiveness in research capacity building is the absence of an overarching capacity-building strategy to which the various GDN activities are linked. Also, GDN is not sufficiently clear about the prioritization of the knowledge generation and capacity-building goals of its major research-support activities and, therefore, the allocation of money and staff time to promote capacity building. As one interviewee said, “the aim is to get the best research possible from individuals in developing and transition countries and to enhance further existing capacity in a pragmatic and flexible way, so there is really no trade-off between generating high quality research and building capacity.” But GDN has not developed a systematic strategy for doing this. It claims that its research projects are meant to bring in and mentor younger talent. But its record suggests that it has not yet resolved how to ensure that capacity building takes place as needed within the different regions and separately-led GRPs. It has not decided how far to go beyond its aim of building capacity for technically sound analysis to capacity for influencing policy-making, as discussed below. Nor does GDN have sufficient monitoring and evaluation processes (such as periodic tracer studies) to know how well its capacity-building efforts work. 3.38 There is by now a large body of literature that shows that building research capacity is a long-term process that rarely has significant sustainable effects from one-off activities. Instead, training activities, in particular, have well-known conditions for success.30 GDN needs to draw on that accumulated knowledge and to commission an expert assessment of its comparative advantage in capacity building with the aim of assisting it in designing a sharper, more strategically designed effort. 3.39 Informing policy. The 2007 evaluation concluded that GDN’s aim of “influencing policy” was unrealistic for a program of its kind. Rather, it found that it can claim to be building a cadre of professionals capable of informing the policy process in support of better policy-making. Yet, it also reported that stakeholders rated GDN’s contribution to “facilitating contacts with policy-makers” lower on average than its other capacity-building effects and that the average rating had declined since 2005 (as shown in Table 5). Thus GDN’s efforts to strengthen the policy relevance of research training and output has been weak, reflecting a more academic than policy focused orientation. A similar conclusion emerged from the separate GDNet evaluation, as noted above. According to the GDNet evaluation, while GDNet can be a key means for enhancing the policy outreach of GDN- supported research, it needs to narrow the existing gap between its current outputs and this 30. See, for example, two IEG evaluations on World Bank capacity building and training activities: “Capacity Building in Africa: An OED Evaluation of World Bank Support,” April 2005 and “Using Training to Build Capacity for Development: An Evaluation of Project-based and WBI Training,” July 2008. 25 purpose. And, for that, GDNet needs to add a clear outreach function linked to an overall GDN strategy for better linking research to policy-making.31 3.40 GDN has recognized the need for a well-defined approach to informing policy. This recognition was the reason for its “Bridging Research and Policy” project (described in Annex B). But the project went for several years without showing significant results.32 A major issue for GDN is that informing policy is a difficult function for a global organization. So, in the view of the several independent evaluations and interviews for this review, GDN should probably concentrate on empowering and pressing RNPs and researchers to be more proactive rather than developing a centralized policy-informing function. But, as the GDNet evaluation emphasizes, this function needs to be designed in the context of a clear theory of change and strategy for achieving change. The recent decision by the Board that policy outreach should be mainstreamed in all GDN activities is a step forward. But the endorsement needs to be followed up by the development of a clear approach not just a lot more activity. 3.41 Thus, overall, this IEG review finds that GDN’s effectiveness in achieving its intended outcome objectives is hampered by the absence of: (1) clear prioritization among the multiple objectives in each of its core and expanded activities; (2) a well-defined strategy for advancing each of its core objectives which interconnects and shapes its activities and guides the allocation of resources; and (3) adequate outcome-based M&E processes for assessing the extent to which activities are advancing each of GDN’s core program objectives. Moving on all of these measures would seem essential to ensuring effective maintenance much less expansion of the program. Efficiency 3.42 It is not possible to make an overall assessment of GDN’s efficiency with the evidence on hand at the time of this review.33 The 2007 evaluation is generally positive regarding GDN’s handling of revenue fluctuations, administrative costs, and financial management. Also, GDN has made progress in increasing and diversifying the sources of its revenue, which is discussed in a subsequent section. 3.43 Amounts and uses of funds. GDN’s budget has reflected considerable volatility in the availability of annual revenues. Total budget expenses reached $10.4 million in FY03. GDN then experienced three years of financial declines, reaching a low of $6.6 million in 31. A small pilot collaboration between GDNet and the Middle East RNP focused on strengthening policy outreach did not succeed, but did identify some important lessons for developing GDNet’s role in the policy outreach area. 32. In the view of staff at the Overseas Development Institute in London, who were enlisted to help set up the BRP, the project framework became far too academic to achieve its aim. 33. The 2007 evaluation examined several aspects of GDN efficiency, but experienced limitations in evaluating cost-efficiencies because GDN “lacks benchmarks for comparisons, and activities involve different outputs and outcomes, some not readily quantified nor easily comparable” (p. iii). The difficulties also reflect the absence of GDN outcome metrics (even proxy or intermediate metrics) for judging its effectiveness. This absence of metrics is as a major challenge to be overcome before the next independent evaluation. 26 FY06 before moving upward again to $9.1 million in FY08.34 Annex E provides detailed information on revenues and expenses over the years from inception through FY08. 3.44 GDN dealt with its funding situation by scaling activities up or down based on the actual levels of funding available year by year, and thus was “able to fund its primary activities to a level deemed satisfactory to key stakeholders even if it is not near meeting the high absorptive capacity of its many potential beneficiaries for its conceivable activities.”35 According to the 2007 evaluation, this program financing is particularly noteworthy given that the increase in GDN budget revenues in recent years has come in the form of earmarked funds that constrained GDN’s ability to shift resources among activities. 3.45 Cost-effectiveness of individual activities. The 2007 evaluation tried to conduct cost- effectiveness analysis for GDN’s five major activities, primarily by measuring the average unit cost per output across each activity. The evaluation noted, however, that comparative criteria for judging the cost-effectiveness of the GDN activities were absent and it suggested that the next evaluation should be tasked with addressing this matter more fully. 36 3.46 Using the per unit cost approach, the 2007 evaluation concluded that RRCs generally are more cost-effective mechanisms in the strictest sense of the term (some $19,000 per team and $12,000 per team member on average) than GRPs (averaging between $54,000 and $135,000 per grant), though theoretically GRPs can lead to greater knowledge added in a concentrated subject area or lead to other follow-on activities besides publications. The evidence suggests that there may be greater efficiency gains to smaller GRPs. 3.47 Different measures for the Annual Conference lead to sharply differing conclusions about the activity’s cost-efficiency. For instance, the per-paper cost is high given the lack of the papers’ capacity-building effects, though participation costs are relatively reasonable (if one excludes the estimated participation costs of the conference side business meetings and workshops). As for GDNet, the issue is less one of cost-efficiency than awareness, user friendliness, and service to researchers. 3.48 Finally, the GDAMC, which are funded through an individual and decreasing earmarked grant that is not fungible across activities, were found not to be cost-effective relative to other activities (about $18,000 per medal, $62,000 per award, and $70,000 per innovative development projects). 34. It should be noted that these levels include expenses of $1.0 million, $1.2 million, and $0.3 million for FY03 to FY05, respectively, for a World Bank-supported activity for which GDN managed the disbursement of funds although not the program. It should also be noted that projected budget revenue for FY09 is over $16 million, or almost double that of FY08 (the last year covered by this review). This large jump in budget is due primarily to resources for new programs one of which is managed by GDN and the other, 3ie, which GDN hosts. 35. 2007 evaluation, p. vii. 36. For example, it noted that the next evaluation will have to judge the cost-efficiency of the otherwise cost inefficient ongoing projects — the Bridging Research and Policy and Health GRPs — based on activities and direct and indirect outputs from them. 27 3.49 GDN has made changes in the allocation of resources across its five core activities, which are responsive to previous criticisms, especially of the high cost of its Annual Conference in early years. But the changes in allocations are not entirely consistent with the relative valuation of the core activities as reflected in stakeholder and end-user surveys. In particular, the RRCs, which are reported to be of highest value, receive support from relatively few donors (Annex E); and they have actually declined in their share of GDN’s overall program expenditures, from about 38 percent in FY02 to about 30 percent in FY08. There have also been wide variations in the absolute as well as per unit expenditures for GRPs, without much accompanying evidence to date of direct and indirect outputs much less outcomes from them. 3.50 Administrative costs. While noting that it was not able to collect administrative data from other similar research organizations to ensure comparability, the 2007 evaluation finds that GDN has “generally reasonable” administrative costs and “generally predictable” delivery of services (with some notable exceptions with network partners and slow or delayed rollout of GDNet services largely beyond its control). Specifically, the evaluation reports that GDN’s programs and services constitute close to 90 percent of its budget — a low overhead level for any organization. These findings, which include the $190,000 costs for relocation from Washington to New Delhi, are improvements over those of the prior evaluation, especially as regards decreasing Annual Conference expenditures. 3.51 Subsequently, GDN has made its own cost-effectiveness assessment, which indicates that GDN operates at relatively low cost compared to several other research institutions of similar or larger size. Table 6 presents the cost comparisons as compiled by GDN. In addition, GDN’s assessment shows a decline in its expenditures for its administrative and support services from $640,425, or 8.60 percent of its total revenue in 2005 to $397,179, or 4.28 percent of total annual revenue in 2007.37 3.52 Financial management. An independent assessment of GDN’s financial management, commissioned by the World Bank and conducted in May 2007, concluded that GDN practices satisfied the Bank’s minimum financial management requirements. The Table 6. Cost-Effectiveness Comparison with Other Research Organizations Centre for International Economic Center for Development Policy Global Research GDN Research Development Brookings Centre (2007) (2003) (2004) (2006) (2007) Total Budget $9.26 m $5.15 m $11.56 m $70.75 m $170.00 m Administrative and 4.28% 4.45% 5.37% 10.04% 18.45% support services Source: Strategic Review, p. 25. Notes: (1) Dates indicate most recent available. (2) Administrative and support services include general oversight, business management, general record keeping, budgeting, finance and other management related activities, exclusive of direct staff and secretariat costs allocated to project activities. 37. Strategic Review, p. 25. 28 assessment found that staff have requisite functional experience and its financial management arrangements are capable of recording all transactions and balances, supporting preparation of regular reliable financial statements and safeguarding assets, and are subject to acceptable auditing arrangements. It, therefore, gave GDN’s practices an overall satisfactory rating, although it found that GDN did not have adequate processes in place to ensure proper use of sub-grants and that fiduciary risks in the use of sub-grants were significant. It recommended that GDN take corrective actions that GDN has since implemented. 4. Governance, Management, and Financial Sustainability 4.1 GDN’s governance and management structures work moderately well in guiding and implementing program activities. Also, the organization has improved its financial position in recent years by mobilizing larger and more diversified sources of funding. But further improvements in all three areas are needed, as acknowledged in the organization’s 2008 Strategic Review and the early efforts of GDN’s current president. Key issues include:  Strengthening board oversight of the business aspects of GDN operations  Improving working relations with GDN’s partners, notably the RNPs  Addressing the constraints posed by the prevailing mode of earmarked donor funding. 4.2 Governance. GDN’s main governing bodies are: (1) an Assembly comprised of governments that are signatories to the Charter establishing GDN as an international organization and (2) a governing Board of Directors. While the Assembly is the super-body above the Board — with the authority to dissolve the organization — the active strategic direction and management oversight of the organization is expected to be exercised by the Board. As of the writing of this review, only three governments had ratified the new GDN Charter, and the Assembly has not yet met. So the effectiveness of this arrangement is unknown. 4.3 The Board, which has functioned since the startup of GDN, convenes twice a year with one of the meetings occurring at the GDN Annual Conference. It is largely a constituency-based board — currently comprised of 18 directors, of which 10 are nominated by and represent GDN’s RNPs and three represent international professional associations. In addition, the Board comprises two senior representatives of GDN donor agencies — the World Bank and UNDP, while the remaining three members are “at large.” Unless a special exception is made, Board members rotate after two-year terms — which has posed problems in the past of high turnover resulting in limited institutional memory. 4.4 On the whole, GDN’s governance complies well in practice with its five “principles of governance (para. 1.17). The governance structure is — and is perceived by stakeholders to be — independent of governments. It is also perceived by stakeholders to be legitimately representative of the research community that GDN is designed to serve.38 Although some 38. This perception of legitimacy reflects in large part the fact that GDN consulted widely on the establishment of its governance structure and has followed an “inclusive and participatory selection process.” Independent 29 stakeholders criticize the continuing predominance of economists on the Board, there are now some directors from other social science disciplines. 4.5 The functioning of the Board has become increasingly effective over time. Many problems in the early years have by now been addressed. Notably, GDN’s move to independent status largely resolved issues stemming from its initial close ties to the World Bank — in particular confusion about the roles and responsibilities of the Bank and GDN’s Board.39 In 2005, to address inefficiencies in its governance operations, GDN established three subcommittees (audit, executive, and program) of the Board which confer in between full Board meetings.40 In particular, the Executive Committee, serves for the full Board in- between meetings. It is a group of appointed Board members who are willing to devote more time to the organization (not intentionally representative of the Board constituencies) each of whom is consulted either by email or telephone for advice/decisions throughout the year as need arises. Also in 2005, GDN created a donor advisory council.41 There is no structural link between the Council and GDN’s Board or Secretariat and the Council has no formal role, but it serves to foster donor coordination and to improve communications among donors and between GDN and interested funders. 4.6 Further improvements to strengthen the strategic direction and oversight roles of the Board, recommended in the 2007 evaluation, have since been implemented or taken under consideration by GDN management and Board.42 The main measures aim to achieve: (1) greater transparency in the process of choosing Board members, (2) processes to guarantee that some Board members have experience in managing organizations and boards, and (3) processes to balance the democratic thrust of two-year terms with the need for increased institutional memory on the Board. While it is too soon to know what effect these measures will have in further strengthening the functioning of the Board, interviewees state that key stakeholders are optimistic about current directions of change. The interviewees also offer three reasons for this optimism: (1) new Board leadership with strong research and policy credibility; (2) lessons learned about the profile of good board members for GDN’s type of program; and (3) the functioning of the Executive Committee. 4.7 Still, this review sees some unresolved shortcomings of GDN’s governance arrangements. Good governance entails separation of oversight/approval functions from management and program setting. The current composition of GDN’s Board, with 10 out of 20 board members representing the RNPs, combines both functions and serves neither well. On the one hand, the 10 RNP representatives are in the ambiguous position of representing the interests of their own regions while also exercising strategic direction and oversight of GDN performance overall (including the performance of their own RNP). On the other hand, the management heads Evaluation Group (formerly Operations Evaluation Department), “The World Bank's Approach to Global Programs: Phase 2 Report: The Global Development Network,” December 2004, pp 17-18. 39. Ibid. 40. Identified as a major weakness in the 2004 evaluation. 41. This step was in lieu of a recommendation of the 2004 evaluation that GDN add one or more donor representatives to the Board, which was rejected by all GDN stakeholders groups. 42. GDN Management Response presented in Annex C. 30 of the RNPs should ideally be involved in the research agenda-setting function, which is currently the responsibility of the GDN Board. One possible alternative might be to establish a separate research advisory body (including the heads of the RNPs among others) and to focus the business of the Board on broad strategic direction and program oversight. 4.8 Management. GDN’s day-to-day operations are managed by its Secretariat, comprising a President and some 30 staff. The number of staff represents a significant increase over the past few years in response to a number of pressures. Notably, GDN has bolstered its fundraising and human resource management capabilities in response to recommendations of the 2007 evaluation. In addition, GDN has added staff to strengthen its communication and policy outreach efforts and to accommodate the choice of GDN to be the host of the new 3ie. While most of the increase has been of staff in New Delhi, the GDNet office in Cairo, as noted above, has also expanded its functions and staff complement. 4.9 Over the years, stakeholders have expressed high regard for the professionalism, efficiency, and probity of the GDN Secretariat staff. That view is confirmed by the two independent evaluations and by interviews conducted for this review. Less certain is how well the staff structure matches the program priorities of GDN, and how staff time is allocated across activities. There is also an issue of whether GDNet is less well linked into the overall program of activities than would be the case if it was based in the Secretariat headquarters in New Delhi. But this may be less a case of location than of adequate program strategy. While in the past there was some concern that GDN’s president had strong research but only limited experience in the policy and fundraising realms, there were widespread expressions of support for the broad range of research, policy, and fundraising experience of the second president, appointed in 2007. It remains to be seen who GDN will find as its next president, following the unexpected departure of the current leadership. 4.10 Relations with regional network partners. GDN’s working relationships with its RNPs are an additional element of its governance and management arrangement. The prevailing view across GDN stakeholders is that the RNPs are the cornerstone of the program. “The RNPs are GDN’s strength.”43 Also, both the 2004 and 2007 independent evaluations found this structural element consistent with GDN’s purposes. Yet, the RNPs, other stakeholders, and the 2004 and 2007 evaluations share the view that the working relationship between GDN and the RNPs is not well developed and should be enhanced to get greater benefit out of the global network. 4.11 The prevailing view is that GDN needs to clarify and considerably strengthen its working relationships with the RNPs. In working across regions GDN cannot afford to have a uniform mission, but it should have a broader mission that encompasses flexibility and responsiveness to prevailing conditions in each partner constituency. Three areas in particular are in apparent need of improvement. First, GDN needs to do better in dealing with the variations in the capacity and performance of the different RNPs in building research excellence and informing policy — especially for the newer RNPs for whom GDN is a relatively significant source of support. Second, GDN needs to strengthen its global services to the RNPs where there is potential for effectiveness and efficiency gains through such undertakings as cross-regional capacity building, 43. Interview for this review. 31 standard setting, and fund-raising. As expressed in interviews for this review, while views vary among the RNPs as to the contribution to date of GDN’s regionally based activities and support, those that existed before GDN was formed point out, in particular, that GDN has not succeeded in boosting resource mobilization as originally intended. Third, GDN needs to adjust how it operates to increase the RNP’s voice in the design and implementation of GDN activities — thus making them true partners. 4.12 In the past year, GDN’s management and Board have made the strengthening and deepening of relations with RNPs a major focus of their Strategic Review and plans for the next several years. Notably, they have identified and begun to address a number of “challenges,” specifically to: enhance the horizontal cooperation among RNPs; improve vertical links between the GDN Secretariat and RNPs; further develop differentiated programs and institutional capacity-building support by region; and find ways to increase the allocations for RRCs and the resource base of the RNPs. In interviews for this review, RNP coordinators commended GDN’s recent efforts. In particular, they gave high praise for a first meeting of the GDN Secretariat and RNP coordinators in 2008; the introduction of new inter-regional research projects (an idea that emerged out of the meeting); and joint fundraising. Given findings about the relevance, comparative advantage, and effectiveness of GDN in promoting and disseminating research for better policy-making, there would seem to be no higher priority for the organization in the coming years than to move effectively in these directions. 4.13 Financial sustainability. GDN has made some progress in increasing the scale and diversity of its funding, especially in recent years. As shown in Table 7, as of 2008, GDN received a total of $9.3 million from 16 donors compared to $7.1 million from 10 donors in 2005. While one donor, Italy dropped out for reasons of its shifting assistance priorities, others have joined largely as a result of GDN’s efforts in marketing its program and achievements. The recent increase in financing included first-time support in 2007 from seven new donors to GDN. This is a notable improvement over the earlier period 2005–07, when it experienced a three-year decline in funds (due primarily to reduced support from the World Bank not compensated for by increased or new funding from others)and failed to meet the financial goals of its 2005–07 Strategic Plan. 4.14 Still, resource mobilization and financial sustainability remain major challenges for GDN — a common situation for all organizations like it that do not have an endowment or generate earnings from their own activities. Currently its “donor map” reflects:  Continuing dependence on some 2–3 donors (including the World Bank) for over 65 percent of total resources  Many small donors, entailing high transaction costs for GDN management  Low levels of support from private and developing country sources  High risks of losing some donors in any year (e.g., the government of Italy in 2006)  Very high rigidity in the uses to which funding can be put, with the bulk linked to specific projects and not fungible across activities for interaction with and institution building support to RNPs. (The extent of this earmarking of contributions is shown in Annex E: donor support by activity.)44 44. Strategic Review, p. 21. 32 Table 7. GDN Revenue from Donors (FY05–FY08) Donors FY05 FY06 FY07 FY08 AFESD AUSAID 22,716 233,141 280,000 480,000 Austria 260,728 DFID 537,374 945,399 916,621 ESRC (UK) 98,806 Finland 102,720 146,470 France 15,000 61,451 131,601 140,119 Gates Foundation 2,500,000 2,000,000 IDRC 99,097 162,617 172,565 72,808 India 99,360 Irish Aid 53,492 Italy 1,156,150 69,265 JBIC 120,040 Luxembourg 97,980 312,236 29,751 138,495 Merck 100,000 Netherlands 328,000 656,190 Norway 59,509 97,500 NZAid 60,609 PHRD 400,000 300,000 190,000 190,000 Saudi Arabia 50,000 Spain 146,415 Sweden 105,000 133,000 69,000 UNDP 390,000 USAID 144,530 67,573 World Bank 4,862,500 4,100,00 3,704,786 3,500,000 (DGF/Other) Others 9,600 16,689 40,137 In Kind Austria 308,400 337,000 IMF 49,000 World Bank 130,626 112,262 Total 7,058,239 2,486,696 9,128,166 9,341,227 Source: GDN Secretariat. 4.15 In planning for how to grapple with its financial situation, GDN’s Strategic Review highlights the organization’s need to: (a) expand its overall resource envelope by further increasing its donor base, attracting new donors, and improving donors’ perceptions of GDN’s demonstrated value added; (b) strengthen multiyear commitments and raising more fungible resources; and (c) build an endowment fund. The Strategic Review also sets out a 33 resource mobilization strategy for increasing resources by no less than a base case of 10–20 percent a year and a high case of 50–100 percent over the three years to 2010. In addition, it outlines four major risks to advancing its resource strategy (especially to achieve its higher case scenario). These risks, and the steps GDN sees as needed to meet them and protect GDN from future abrupt fluctuations, are listed in Box 2. Clearly, GDN is right to emphasize the extent of the financing challenge it faces. Box 2. Risk and Risk Mitigation Measures Major risks confronting GDN as identified in its Strategic Review are: (1) World Bank funding decreases (2) Share of fungible resources continues to decrease (3) Tensions grow between donor preferences for research and GDN/RNP comparative advantages (4) There is no regional commitment to collaboration Risk mitigation measures outlined in the Strategic Review are: (1) Demonstrate GDN’s value added and effectiveness (2) Secure multiyear funding from current donors (3) Diversify the donor base by targeting new donors with an emphasis on multiyear commitments (4) Actively explore synergies for collaboration and joint funding with centers of excellence worldwide (both North and South) (5) Explore possibilities for securing an endowment (6) Recruit a professional fund-raiser as GDN staff Source: Strategic Review, p. 22. 5. The World Bank’s Performance as a Partner 5.1 The World Bank was the founder and has been the major donor of the GDN. It has also been a member of the Board from the start, and Bank staff — retired and current — have so far provided the leadership of the Secretariat and participated in the organization’s Annual Conferences and other activities. Through these multiple roles the Bank has contributed crucial financial support and valued technical assistance to GDN. But there has been disagreement within the Bank about the effectiveness and continuation of DGF grant funding, and staff oversight has only been moderately adequate. Initiation and Incubation 5.2 The World Bank conceived of and initiated the GDN in the late 1990s as part of a broader interest in increasing the capacity of research and policy institutions in developing countries to produce high-quality, policy-oriented research that could feed into local policy- making. This interest was seen as consistent with the Bank’s stated objective of “putting countries in the driver’s seat” in designing their own development strategies. It was also seen as a way to generate research that could feed into the design of Bank projects, ESW, and 34 policy advice as well as generate independent analysis on key global issues affecting developing countries. As noted above, broad-ranging external consultations organized by the Bank confirmed the relevance of such an initiative and endorsed GDN’s establishment. 5.3 External consultations also indicated that the Bank had a distinct comparative advantage in initiating the network. There were three reasons for this. First, the Bank was already helping to support development research networks in different regions of the world and had experience in working with and helping create independent regional networks.45 Second it had both the financial resources and multicountry research expertise that could be useful to the success of a global research capacity-building effort. Third, experience suggested that World Bank support could help to leverage the funds of other donors. The first two of these strengths have been widely perceived as highly valuable in supporting the development and operations of GDN; but, as discussed below, the leveraging effect of the World Bank on other donors has not been as consequential as expected. 5.4 To jump-start the initiative and because there was no existing operational program within the Bank aimed at generating more and better research from within developing countries, GDN was launched as a unit of the Bank with a member of the Bank staff as the program’s first director. This arrangement led to some early outside criticism that the GDN was too Bank-dominated and some perceptions that the Bank was trying to use GDN to support research that served its own purposes and policy views. These concerns reinforced the Bank’s original intent to move GDN outside the Bank and to establish it as an independent organization. The initial program director moved as well, retiring from the Bank and becoming the independent GDN’s first president. The move was favorably endorsed inside and outside the Bank. It was a comparatively quick and efficient “spin off,” and it eased international worries that GDN was too much a “creature of the World Bank.” 5.5 Still, GDN’s relationship with the World Bank remains an evolving one, with several issues unsettled at the time of this review. One issue is the right amount and kind of substantive engagement between the Bank and GDN. GDN has, over the years, enlisted individual Bank staff (current and retired) as participants and advisers in conferences and research projects. While a minority view continues to see too dominant a presence of Bank thinking in all GDN work, the prevailing view of GDN stakeholders is that the Bank’s expertise has been extremely useful. Also, those responsible for GDN within the Bank and GDN’s current senior management think that, in an effort to prove GDN’s independence, the Secretariat in the early years distanced the organization too much from substantive engagement with relevant Network and Regional units of the Bank. As a result, substantive interaction between the Bank and GDN researchers and research projects is seen as less than optimal both for GDN research capacity building and Bank engagement with capable researchers and institutions in developing and transition countries. As GDN has matured and increasingly strengthened its own international reputation, both parties contend that the Bank 45. For example, it had contributed to the AERC since the late 1980s and helped create the EADN in June 1999. 35 and GDN would benefit from broader and deeper substantive interactions.46 Two other major issues are the Bank’s performance in providing strategic guidance and its financial support. Strategic Guidance 5.6 The Bank has played a role in providing strategic guidance to GDN through its membership on the governing board. To date, three senior Bank staff have served on the Board, and each has received high praise from other Board members and GDN management for their significant input and, often, leadership of strategic deliberations. For example, the Bank’s representatives were seen as instrumental in helping to resolve a dispute leading to the withdrawal of one of GDN’s RNPs from participation, shaping the two independent evaluations, and encouraging the move to New Delhi which helped GDN become a more cost-effective organization. 5.7 Yet, the lack of clarity in GDN’s program priorities and strategies raises questions about the adequacy of the Bank’s degree of engagement in the oversight and strategic direction of GDN. This issue is particularly fraught in the case of GDN, which was founded and has continued to be heavily funded by the Bank, and on which the Bank sits as a member of the board. GDN, from its early days, has relied on Bank funding but also sought to avoid the appearance of being Bank-dominated. At the same time, the Bank, by assuming a position on the Board, has had a responsibility to push for strategic clarity and direction, and it is implicated in the organization’s weakness in this regard. Its multiple partnership roles have also created a tension for the Bank’s representative on GDN’s Board between responsibility to the organization and to the Bank as donor. 5.8 Moreover, the Bank is not an active member of the GDN’s donor advisory council, established to improve communication among GDN donors, and between them and the Secretariat and Board, on GDN priorities and program planning. Indeed, one donor on the advisory council, while praising the Bank’s initiation and continued support of GDN, contends that the Bank is not a “team player” and not adequately engaged in fostering coordinated and coherent support for GDN. Funding 5.9 The World Bank has contributed a total of approximately $45.6 million to GDN over fiscal years 1998–2008. The bulk of this support has been in the form of DGF grants. In the fiscal years 1998–2001, when GDN operated as a unit within the Bank, DGF disbursed some $12.1 million on behalf of GDN directly to regional research networks and institutions as they became GDN partners. Thereafter, from the inception of GDN as an independent organization through FY08, the World Bank has provided some $31.5 million in core support to GDN through its DGF window 1 facility. This DGF grant support has amounted in total to some 54 percent of GDN expenses in the period since FY02, as shown in Table 8. 46. Collaboration of the World Bank Institute with GDNet in the conduct of capacity-building workshops on knowledge management in Burkina Faso, Egypt, and India is a recent example of useful, broadened interaction. 36 5.10 In addition, through the first five years of GDN’s independent status, the World Bank contributed about $2 million from its administrative budget to the organization’s operating expenses (of which $50,000 supported its move to New Delhi in 2005). This support reflected a commitment made by then-Bank President James Wolfensohn, formalized in a Memorandum of Understanding between the Bank and GDN, to underwrite up to $1.8 million in operating costs of GDN’s Secretariat in Washington for five years. The MOU also specified that this support would be reviewed at midpoint and would be continued provided that: (1) the assessment of the GDN activities is judged to be positive, (2) donor support exceeds $10 million a year and exhibits an upward trend, and (3) donor support for the Secretariat exceeds $0.5 million a year and is on track for meeting at least 50 percent of the Secretariat’s costs by the end of FY06. Although the financial conditions were not met, the Bank carried through with its commitment of five years of operating support. 5.11 As indicated by Table 8, GDN has not been successful in leveraging DGF support to generate additional funding equal to 85 percent of the program’s budget, which is the DGF target guideline.47 According to those guidelines, any single grant to a recipient “should generally not exceed 15 percent of expected funding over the life of Bank funding to a given program, or over the rolling 3-year plan period, whichever is shorter. Where grant programs belong to new areas of activities, ... the target for the Bank grant not to exceed 15 percent of total expected funding will be pursued after allowing for an initial start-up phase (maximum 3 years).”48 Table 8. World Bank Support to GDN from Its Inception as an Independent Organization through FY08 ($US millions) Fiscal Years 2002 2003 2004 2005 2006 2007 2008 Total /1 DGF grants /2 5.5 5.9 5.6 4.0 3.5 3.5 3.5 31.5 Administrative budget 0.5 0.4 0.4 0.4 0.3 2.0 Total Bank support 6.0 6.3 6.0 4.4 3.8 3.5 3.5 33.5 Total GDN expenses 7.3 10.4 8.6 7.2 6.6 8.6 9.1 57.8 DGF grants as percentage of 75% 57% 65% 55% 53% 41% 38% 54% total GDN expenses Sources: DGF grants — project internal order; Administrative budget — GDN internal memos; GDN expenses — annual audited statements. Notes: 1. From inception of GDN status as an independent organization in March 2001 2. Exclusive of $229,000 for separate HDN education initiative managed by GDN allocated in FY04–06 47. Note that the information in the table is the percentage of GDN expenses covered by DGF funding each year rather than the percentage of donor contributions accounted for by DGF because contributions fluctuate from year to year depending on donors’ funding cycles and the DGF grant is supposed to be spent in the fiscal year in which it is received. 48. DGF eligibility criteria, The Development Grant Facility: FY98 DGF Annual Review and Proposed FY99 Budget, October 28, 1998, p. 39. 37 5.12 Yet, in the case of GDN, Bank staff have repeatedly presented exaggerated expectations while seeking DGF funding. In 2005, an internal review of overall DGF funding priorities recommended that DGF support for GDN should be phased out within three years because of lack of progress in obtaining support from other donors and establishing GDN as a true global partnership. DEC and PREM, the two sponsoring units for GDN within the Bank, successfully argued to maintain funding at near current levels for another three years (FY06–08), while also recommending that the Bank should not cover more than 33 percent of GDN costs by FY08 (which would represent a significant decline from the level of 55 percent in FY05). In fact, the Bank’s share of GDN expenses did decline significantly in the three years, but not to the recommended level and a subsequent proposal for another three years of support (through fiscal year 2011) anticipates Bank support at a level of 23 percent of total revenue for the full cycle. 5.13 Thus, at this time the percentage of Bank support remains above the DGF guideline and GDN remains heavily dependent on Bank financial support. This heavy dependence is related not only to the dollar amount of Bank support. As noted above, the Bank is one of only three donors currently providing core program support (the other two at much lower levels) and its funding is the main source of GDN support through the RRCs to local researchers.49 5.14 From GDN’s point of view, a further decrease in Bank support in the absence of other support for non-project funding poses a serious risk for the organization. In addition, there is a risk for the individual RNPs who get World Bank support through GDN. For some of these RNPs, such as the AERC, the Bank had provided support directly prior to the establishment of GDN. While the shift in funding to the regional networks through a single DGF grant to GDN was a convenience for the Bank, these individual RNPs have felt this to be a lessening of the opportunity for evolving support from the Bank and as a risk of loss of support overall. Oversight 5.15 Bank oversight of its partnership with GDN resided in the PREM VPU from 1999– 2003, and thereafter has resided in the DEC VPU, which also serves as the sponsoring unit for GDN funding requests to DGF. Ongoing responsibility for monitoring financial and operational performance is carried out by a Global Program Task Team Leader in the responsible VPU, who also provides support to the World Bank representative on the GDN Board.50 5.16 In practice, Bank oversight and accountability have been weak throughout the life of the GDN program. There are three reasons for this weakness. First, the positioning for responsibility of GDN got off to a rocky start. PREM served as the first sponsoring unit even 49. Indeed, it is worth noting that prior to this funding arrangement, the Bank was directly funding regional research networks such as AERC. This direct support ended when the decision was taken to route support through GDN — an arrangement that at least AERC has found to be a negative outcome of the establishment of GDN. In AERC’s view, the change has weakened its working relationship with the Bank without any compensating enhancement in its ability to mobilize resources. 50. The TTL and the Bank’s representative were the same person for two years, 2006–07. 38 though the initiation, design, and primary interest in GDN came from DEC. While this initial arrangement had the potential benefit of establishing two overseers and champions for the partnership, this did not work well. Indeed, the two units differed on the rationale for the Bank’s interest in GDN. While DEC emphasized the value of GDN as a way to build analytical capacity in developing countries that could inform countries’ own policies, PREM looked for GDN contributions to Bank operations; and this led to differing views on the effectiveness of GDN activities and on the case for continuing support. In 2004 DEC became the responsible VPU, a change that helped align the interest in the program with the responsibility for overseeing it. But other weaknesses in oversight have persisted. 5.17 A second weakness has been the failure to keep adequate records, especially with each successive move of GDN. Indeed, this review was unable to retrieve most annual progress reports and funding requests to the DGF from GDN, DEC, or the DGF — each unit explaining that they should be available from one of the other units. 5.18 Finally, while oversight of GDN financial management has been satisfactory,51 there has been poor reporting of the Bank’s internal expenditures associated with GDN. For example, this IEG review was unable to find any reporting for GDN from 2002–08 in the Bank’s cost accounting system, meaning that the Bank’s representatives and task team leaders have been charging their time against some other activity, if at all. The present review was also unable to obtain a formal record of the $2.0 million which the Bank provided from its administrative budget to help cover the operating costs of the GDN Secretariat during the five-year period set out in the Bank-GDN MOU. Exit Strategy 5.19 The Bank does not have an explicit strategy for exiting its partnership with GDN, and those responsible for overseeing the program argue that it is too soon to set out such a strategy. GDN disengaged from the Bank legally and physically in 2001. Financially there has been a reduction in the share of Bank support, though the reduction has consistently failed to meet successive target levels set as a condition for continued support and fallen well short of the DGF 15 percent guideline. 5.20 In internal documents and interviews for this IEG review, current and past Bank task team leaders and representatives offered several reasons for continuing support prior to developing an exit strategy. Notably, they emphasized that:  GDN is still young. Typically, it takes 15–20 years for an organization of its kind to become fully established, and before the Bank can responsibly develop an exit strategy. Having moved successfully beyond its start-up phase, GDN needed to consolidate its programmatic and financial position following its move to India, shift to international organization status, and renovation under new leadership.  GDN objectives remain closely aligned with the Bank’s interest in fostering analytical and knowledge capacity in developing and transition countries. 51. World Bank commissioned financial management assessment, May 2007. 39  The research community in developing and developed countries values GDN and praises the Bank’s role in it. The community would strongly oppose a Bank withdrawal anytime soon, especially given the positive findings of the recent evaluation.  Although GDN has made substantial strides in recent years to increase and diversify its sources of funding, donors are unlikely to step in to compensate for a complete withdrawal of Bank support in the near term. There is danger this may, instead, create program and reputational risks for GDN and, by implication, for the Bank as its founder and major champion. 5.21 Overall, the funding history suggests a lack of realism on the Bank’s part about its ability to leverage other donor resources, and/or the lack of a sufficient effort by the Bank’s senior management and representative on the Board to help GDN mobilize resources. In addition, the funding history suggests a weakness in the way that the Bank approaches the idea of an “exit strategy” for an activity that by its very nature takes time to have effect. To date, thinking about an exit strategy seems to be focused on the timing of Bank withdrawal, rather than on how to assist GDN (as well as the RNPs, now funded indirectly by the Bank through GDN) in developing and implementing a sound strategy for achieving financial sustainability — including how to actively help GDN with the planning and resource mobilization effort that such a strategy would have to entail. While the Bank may not choose to play this role in all partnerships in which it participates, certainly for a global partnership of its “invention,” it needs to commit itself to a major engagement in securing the initiative’s financial sustainability. Setting and then ignoring guidelines for continued funding is not an adequate approach. 6. Lessons Learned 6.1 The review of GDN’s relevance and effectiveness highlights six main lessons, of which four are relevant to design and implementation of global programs and two are relevant to the Bank’s support of such programs. 6.2 Effectiveness depends on a theory of change and explicit strategies for achieving outcome objectives. GDN’s moderate performance to date highlights the importance of having a well-articulated theory of change and related operational strategies for achieving intended program results. As GDN shows, these design features are especially important for a program that aims to achieve results — such as capacity building and policy influence — that are not easy to quantify or attribute directly to program activities. GDN has a vision of the future it wants to help create. It has been explicit about its overarching and related program objectives — of generating high-quality, policy-relevant research, building research capacity, and informing policy. It has also been successful in carrying out a menu of activities in pursuit of those program objectives. But it lacks a road map (or logical framework) for how to get to its outcome objectives and strategies to guide and integrate the activities needed to achieve them. Thus, as evaluations show, there is limited evidence of GDN’s capacity-building impact despite its claim that capacity building is a central aim of its 40 RRC, GRP, and GDAMC activities; and the research activities as well as GDNet have yet to develop effective policy outreach functions. 6.3 Global networks require well-defined roles and responsibilities for their constituent parts. Global networks may link together many types of constituent parts, ranging from individuals to national, regional, and/or international organizations. In all cases, it is essential for success that the constituent parts know and accept their respective roles and responsibilities for achieving agreed objectives. GDN rests on the assumption that there is knowledge to be gained and cost-efficiencies to be achieved by promoting research collaboration and exchange of experience on a global scale across regions. But its experience also shows the shortcomings of operating with limited horizontal connection across regional network partners, limited vertical communication and support between the central secretariat and the regions, and inadequately differentiated programs of support from the center for the differing needs of the regions. GDN's maximum value added would come from operating as a true network of networks that addresses these shortcomings. 6.4 It is difficult to design and implement a coherent global program without a sufficient level of core funding to allow for the implementation of results-based strategies and the coordination of activities in support of those activities. As GDN’s revenues have grown, they have become increasingly project-tied and non-fungible. While these projects have enabled GDN to expand its support of developing world research and voice, the dominance of earmarked funding impedes GDN’s ability to support its regional partners and to design and link activities in ways that could best advance its threefold objectives of research generation, capacity building, and policy outreach. 6.5 There are weaknesses in a governance arrangement in which the governing board is responsible for setting the program agenda as well as strategic direction and oversight. Governing boards are responsible for ensuring that their organizations and programs run effectively and efficiently to advance their goals. This means watching that activities are consistent with core objectives, management is capable, and operations are based on adequate and sound financing. Where a board both sets and overseas a program agenda, it risks shorting its strategic stewardship role and obscuring the voice of the program’s target beneficiaries in defining what is needed to achieve core objectives. Although GDN’s governing board has become more efficient in recent years, due in part to the introduction of an Executive Committee and two other program committees, issues of broad strategic direction and financial sustainability would seem to require more Board attention, while choices of GRP and other research activities might be better left to a non-Board research advisory body (including the management heads of the RNPs among others). 6.6 A realistic commitment by the Bank is required in establishing a donor-dependent program with a long-term objective. GDN demonstrates that it can take time to attract donor support for a Bank-initiated program, if the program is not a broad multidonor partnership from the outset and, in particular, if the program aims to advance goals that are long-term and intangible (such as research capacity building). This suggests that timetables for the reduction of World Bank support need to be better calibrated to the nature of specific programs than is currently the case with DGF guidelines. It also suggests that exit strategies should be devised from the outset that not only set out a schedule for termination of Bank 41 support but also describe what assistance the Bank commits to provide to help the organization plan and achieve its long-term financial sustainability commensurate with the Bank’s interest in the program. 6.7 The Bank needs to develop strengthened guidelines for its participation, and the accountability of its performance, in global partnerships. The Bank’s accountability of its partnership role has been weak throughout the life of GDN. This weakness has been evidenced by repeated and exaggerated expectations in seeking DGF funding; poor record keeping; and understatements in progress reports of GDN’s lack of clarity and strategic direction in achieving its intended outcomes. This problem has been particularly severe in the case of a global program such as GDN, which has been founded and funded by the Bank and on which the Bank has served as a member of the governing board — thereby creating a tension for the Bank’s representative between responsibility to the organization and to the Bank as donor. The Bank’s engagement with GDN demonstrates the need for clear guidelines especially in regard to (a) standards of program design and performance to be expected in situations of continuing Bank support, and (b) the roles and responsibilities to be exercised by Bank program managers, task team leaders, and representatives on partnership boards. 43 References Angelescu, Ramona, and Lyn Squire, “Capacity Building and Policy Impact: The Experience of the Global Development Network,” paper presented at World Bank and CEU Conference, Budapest, June 2005. Craig, Barbara, and Fernando Loayza Careaga, “ An Evaluation of GDN Supported Regional Research Competitions,” 2002. Global Development Network, Annual Report, 2002 through Annual Report 2008. ———, “Evaluation of Capacity-Building Activities: Results of a Pilot Evaluation, “ March 2006. ———, “FY2006 Business Plan and Budget Support Request to the World Bank,” March 29, 2005. ———, “GDN Management Action Response,” January 2008. ———, “Progress Report, July 2008-January 2008,” January 2009. ———, “Response to External Evaluation,” October 7, 2004. ———, “Strategic Review: Issues Note for Board Discussion,” www.gdnet.org, January 2008. Independent Evaluation Group (formerly Operations Evaluation Department), “The World Bank's Approach to Global Programs: Phase 2 Report: The Global Development Network,” December 2004. ———, Using Training to Build Capacity for Development: An Evaluation of Project-based and WBI Training, July 2008. ———, Capacity Building in Africa: An OED Evaluation of World Bank Support, April 2005. Johnson, Erik, and Diane Stone, “The Genesis of the GDN,” undated manuscript. Loayza Careaga, Fernando, “Participatory Impact Assessment of the Global Research Program on Explaining Growth,” October 2003. Muth, H. Peter, and Frederick H. Gerlach, “Independent Evaluation,” March 16, 2004. Shapiro, Marc D., Savi Mull, and Tina Khanna, “Evaluation of the Global Development Network, 2007,” December 2007. Stone, Diane. with Simon Maxwell and Michael Keating, “Bridging Research and Policy,” paper presented at an international workshop funded by DFID, Warwick University, July 16-17, 2001. World Bank, “DGF Progress Report, Global Development Network,” FY08. ———, “Memorandum of Understanding (MOU) between the International Bank for Reconstruction and Development (Bank) and the Global Development Network (GDN),” May 20, 2002. ———, Development Grant Facility, “Annual Review and Budget,” fiscal years 1999 through 2009. 45 Annex A Annex A. Evaluation Framework for Global Program Reviews Note: This evaluation framework is a general framework that has been designed to cover the wide range of such programs in which the World Bank is involved, encompassing policy and knowledge networks, technical assistance programs, and investment programs. It is not expected that every global program review will cover every question in this table in detail. Table A-1. Assessing the Independence and Quality of the Evaluation Evaluation Questions 1. Evaluation process To what extent was the GRPP evaluation independent of the management of the program, according to the following criteria:  Organizational independence?  Behavioral independence and protection from interference?  Avoidance of conflicts of interest? Factors to take into account in answering these questions include:  Who commissioned and managed the evaluation?  Who approved the terms of reference and selected the evaluation team?  To whom the evaluation team reported, and how the evaluation was reviewed?  Any other factors that hindered the independence of the evaluation such as an inadequate budget, or restrictions on access to information, travel, sampling, etc.? 2. Monitoring and evaluation framework of the program To what extent was the evaluation based on an effective M&E framework of the program with:  Clear and coherent objectives and strategies that give focus and direction to the program?  An expected results chain or logical framework?  Measurable indicators that meet the monitoring and reporting needs of the governing body and management of the program?  Systematic and regular processes for collecting and managing data? 3. Evaluation approach and scope To what extent was the evaluation objectives-based and evidence-based? To what extent did the evaluation use a results-based framework — constructed either by the program or by the evaluators? To what extent did the evaluation address:  Relevance  Governance and management  Efficacy  Resource mobilization and financial management  Efficiency or cost-effectiveness  Sustainability, risk, and strategy for devolution or exit 4. Evaluation instruments To what extent did the evaluation utilize the following instruments:  Desk and document review  Consultations/interviews and with whom  Literature review  Structured surveys and of whom  Site visits and for what purpose: for interviewing implementers/beneficiaries, or for observing activities being implemented or completed  Case studies  Other Annex A 46 Evaluation Questions 5. Evaluation feedback To what extent have the findings of the evaluation been reflected in:  The objectives, strategies, design, or scale of the program?  The governance, management, and financing of the program?  The monitoring and evaluation framework of the program? Table A-2. Providing an Independent Opinion on the Effectiveness of the Program Every review is expected to cover the first four criteria in the following table: (a) relevance, (b) efficacy, (c) efficiency, and (d) governance and management. A review may also cover (e) resource mobilization and financial management and (f) sustainability, risk, and strategies for devolution or exit if the latter are important issues for the program at the time of GPR, and if there is sufficient information available on which to base an independent opinion. Evaluation Criteria and Questions Relevance: The extent to which the objectives and design of the program are consistent with (a) current global/regional challenges and concerns in a particular development sector and (b) the needs and priorities of beneficiary countries and groups. 1. Supply-side relevance — the existence of an international consensus that global/regional collective action is required. To what extent does the program reflect an international consensus on the need for action, on the definition of the problem being addressed, on priorities, and on strategies for action? Is the original consensus that led to the creation of the program still present? Is the program still needed to address specific global/regional public concerns? Take into account the origin of the program in answering these questions:  Is the program formally responsible for implementing an international convention?  Did the program arise out of an international conference?  Is the program facilitating the implementation of formal standards and approaches?  Is the program primarily donor-driven? Did donors establish the program with little consultation with developing countries?  Is the program primarily Bank-driven? Did the World Bank found the program and then seek other partners? 2. Demand-side relevance — alignment with beneficiary needs, priorities, and strategies. To what extent are the objectives consistent with the needs, priorities, and strategies of beneficiary countries as articulated in the countries’ own PRSPs, and in donors’ strategies such as the World Bank CASs, and the UN Development Assistance Frameworks? To what extent has the voice of developing and transition countries been expressed in the international consensus underlying the program? 3. Vertical relevance — consistency with the subsidiarity principle. To what extent are the activities of the program being carried out at the most appropriate level — global, regional, national, or local — in terms of efficiency and responsiveness to the needs of beneficiaries? To what extent are the activities of the program competing with or substituting for activities that individual donors or countries could do more efficiently by themselves? Pay particular attention to those programs that, on the face of it, are primarily supporting the provision of national or local public goods. 47 Annex A Evaluation Criteria and Questions 4. Horizontal relevance — the absence of alternative sources of supply. What is the comparative advantage, value added, or core competency of the program relative to other GRPPs with similar or complementary objectives? To what extent is the program providing additional funding, advocacy, or technical capacity that is otherwise unavailable to meet the program’s objectives? To what extent are the good and services being provided by the program in the nature of public goods? Are there alternative ways of providing these goods and services, such as by the private sector under regular market conditions? 5. Relevance of the design of the program To what extent are the strategies and priority activities of the program appropriate for achieving its objectives? What are the major activities of the program:  Policy and knowledge networking?  Financing country and local-level technical assistance?  Financing investments to deliver national, regional, or global public goods? (See Annex Table 7.) Has the program articulated an expected results chain or logical framework, along with assumptions that relate the progress of activities with the achievement of the objectives? Does the results chain identify the extent to which the achievement of the objectives depends on the effective functioning of bureaucracies, markets, or collectivities? If so, to what extent are these assumptions valid? For programs providing global or regional public goods, is the design of the program consistent with the way in which the individual efforts of the partners contribute to the collective outcome for the program as a whole — whether “best shot”, “summation”, or “weakest link?” Efficacy: The extent to which the program has achieved, or is expected to achieve, its objectives, taking into account their relative importance. 6. Achievement of objectives To what extent have the stated objectives of the program been achieved, or has satisfactory progress been made towards achieving these objectives? To what extent are there implicit objectives that are well understood and agreed upon by the partners and to which the program should also be held accountable? To what extent are there any positive, unintended outcomes of the program that have been convincingly document? To what extent have these assessments by the program or the evaluation been evidence-based? 7. Progress of activities, outputs, and outcomes. To what extent has the program or the evaluation measured the progress of activities, outputs, and outcomes? How did the program or the evaluation aggregate its outputs and outcomes at all levels — global, regional, national, and local — to provide an overall summary of its results? To what extent have factors such as changes in the location of the program, its legal structure, or governance processes affected the outputs and outcomes of the program? To what extent have there been outcomes that can be uniquely attributed to the partnership itself — such as the scale of or joint activities made possible by its organizational setup as a GRPP, or its institutional linkages to a host organization? 8. Linkages to country or local-level activities. To what extent has the program established effective operational linkages with country-level activities, taking into account that:  The desired nature of these linkages will vary according to the objectives, design, and implementation of each program?  Positive outcomes at the country or local level are generally a joint product of both global/regional and county- level activities? Annex A 48 Evaluation Criteria and Questions Efficiency or cost-effectiveness: Efficiency — the extent to which the program has converted or is expected to convert its resources/inputs (such as funds, expertise, time, etc.) economically into results. Cost-effectiveness — the extent to which the program has achieved or is expected to achieve its results at a lower cost compared with alternatives. 9. Efficiency To what extent is it possible to place a monetary value on the benefits arising from the activities of the program? To what extent has the program or the evaluation conducted impact evaluations of representative program activities? To what extent has the program or the evaluation analyzed the program’s costs in broad categories (such as overhead vs. activity costs), and categorized the program’s activities and associated benefits, even if these cannot be valued in monetary terms? 10. Cost-effectiveness To what extent is the program measuring up against its own business plans:  Has the program cost more or less than planned? How did it measure up against its own costing schedule?  Have there been any obvious cases of inefficiency or wasted resources? To what extent is the program delivering its activities cost-effectively in comparison with alternatives:  How do actual costs compare with benchmarks from similar programs or activities?  Are the overhead costs of governing and managing the program reasonable and appropriate in relation to the objectives and activities of the program? How does the program compare with traditional development assistance programs:  For beneficiary countries, has receiving the development assistance through the GRPP increased the transactions costs compared with traditional development assistance programs?  For donors, has delivering the development assistance through the GRPP reduced donor costs by harmonizing efforts among donors or by reducing overlapping work (such as through joint supervision, monitoring and evaluation)? Governance and management: Governance — the structures, functions, processes, and organizational traditions that have been put in place within the context of a program’s authorizing environment to ensure that the program is run in such a way that it achieves its objectives in an effective and transparent manner. Management — the day-to-day operation of the program within the context of the strategies, policies, processes, and procedures that have been established by the governing body. Whereas governance is concerned with “doing the right thing,” management is concerned with “doing things right.” 11. Compliance with generally accepted principles of good governance. To what extent are the governance and management structures and processes well articulated and working well to bring about legitimate and effective governance and management? To what extent do governance and management practices comply with the following seven principles:  Legitimacy — the way in which governmental and managerial authority is exercised in relation to those with a legitimate interest in the program — including shareholders, other stakeholders, implementers, beneficiaries, and the community at large?  Accountability — the extent to which accountability is defined, accepted, and exercised along the chain of command and control within a program, starting with the annual general meeting of the members or parties at the top and going down to the executive board, the chief executive officer, task team leaders, implementers, and in some cases, to the beneficiaries of the program?  Responsibility — the extent to which the program accepts and exercises responsibility to stakeholders who are not directly involved in the governance of the program and who are not part of the direct chain of accountability in the implementation of the program? 49 Annex A Evaluation Criteria and Questions  Fairness — the extent to which partners and participants, similarly situated, have equal opportunity to influence the program and to receive benefits from the program?  Transparency — the extent to which a program’s decision making, reporting, and evaluation processes are open and freely available to the general public?  Efficiency — the extent to which the governance and management structures enhance efficiency or cost- effectiveness in the allocation and use of the program’s resources?  Probity — the adherence by all persons in leadership positions to high standards of ethics and professional conduct over and above compliance with the rules and regulations governing the operation of the program? 12. Partnerships and participation To what extent has the program identified a complete list of stakeholders, or “stakeholder map”, including the agreed- upon or perceived roles and responsibilities of the categories of stakeholders identified? To what extent is this a routine programmatic function, updated regularly, and transparently available? Has the program adopted primarily a shareholder model of governance (in which membership on the governing body is limited to financial and other contributors), or a stakeholder model (in which membership also includes non- contributors)? To what extent, if any, is the program’s legitimacy being sacrificed in order to achieve greater efficiency, or vice-versa? 13. Programs located in host organizations To what extent is the location of the program in the Bank or other partner organization adversely affecting the governance, management, or other aspects of the program, such as compliance with the principles of transparency and fairness? For which functions is the program manager accountable to the host organization and the governing body of the program, respectively? Are conflicts of interest being managed appropriately? To what extent does the host organization play such a dominant role in the program, thereby reducing the incentives of other partners to participate effectively, or reducing the ability of the host organization to look at the weaknesses of the program objectively? Resource mobilization and financial management: Resource mobilization — the processes by which resources are solicited by a program and provided by donors and partners. Financial management — the processes that govern the recording and use of funds, including allocation processes, crediting and debiting of accounts, controls that restrict use, accounting, and periodic financial reporting systems. In cases where funds accumulate over time, this would also include the management of the cash and investment portfolio. 14. Resource mobilization To what extent has the program succeeded in raising financial resources commensurate with its objectives? And from what sources — the Bank, bilateral donors, foundations, etc.? To what extent has the program succeeded in diversifying its funding beyond a small number of donors? To what extent are the sources of funding for the program (including donor restrictions on the use of resources) affecting, positively or negatively:  The strategic focus of the program?  The outputs and outcomes of the program?  The governance and management of the program?  The sustainability of the program? Annex A 50 Evaluation Criteria and Questions 15. Financial management Are there any issues that have emerged during the course of the review in relation to:  The quality of financial management and accounting?  The methods, criteria, and processes for allocating funds among different activities of the program?  Financial management during the early stages of the program? Sustainability, risk, and strategy for devolution or exit: Sustainability — When applied to the activities of a program, the extent to which the benefits arising from these activities are likely to continue after the activities have been completed. When applied to a program itself, the extent to which the organization or program is likely to continue its operational activities over time. Devolution or exit strategy — a proactive strategy to change the design of a program, to devolve some of its implementation responsibilities, to reduce dependency on external funding, or to phase out the program on the grounds that it has achieved its objectives or that its current design is no longer the best way to sustain the results which the program has achieved. 16. Sustainability of the benefits of the program’s activities What is the risk, at the time of evaluation, that the development outcomes (or expected outcomes) of the program will not be maintained (or realized)? This depends on (a) the likelihood that some changes may occur that are detrimental to maintaining or realizing the expected outcomes, and (b) the affect on the expected outcomes if some or all of these changes actually materialize? 17. Sustainability of the program This will depend on a number of factors, such as the continued legitimacy of the program, its financial stability, its continuity of effective management, and its ability to withstand changing market or other conditions. To what extent is there still a sufficient convergence or accommodation of interests among the major partners to sustain the program financially? To what extent has the program developed institutional capacity such as performance- based management, personnel policies, learning programs, and knowledge management that help to sustain a program? In what areas could the program improve in order to enhance its sustainability, such as better marketing of the program’s achievements in order to sustain its reputation? 18. Prospects for continuation and strategies for devolution or exit To what extent should the program be sustained? Is the continuation of the program the best way of sustaining the results achieved? Should the design of the program be modified as a result of changed circumstances, either positive or negative? What other alternatives should be considered to sustain the program’s results more cost-effectively, in the light of the previous evaluation findings with respect to relevance, efficacy, efficiency, and sustainability:  Reinventing the program with the same governance?  Phasing out the program?  Continuing country or local-level activities with or without devolution of implementation?  Seeking alternative financing arrangements, such as revenue-generation, or self-financing to reduce dependency on external sources?  “Spinning off” from the host organization? 51 Annex A Table A-3. Assessing the Bank’s Performance as a Partner in the Program Evaluation Questions 1. Comparative advantage at the global/regional level. To what extent is the Bank playing up to its comparative advantages at the global/regional level — its global mandate and reach and convening power? To what extent is the Bank’s presence as a partner in the program catalyzing other resources and partners for the program? 2. Comparative advantage at the country level. To what extent is the Bank contributing multisector capacity, analytical expertise, and country-level knowledge to the program? To what extent has the Bank’s country operations established linkages to the GRPP, where appropriate, to enhance the effectiveness of both? 3. Oversight. To what extent is the Bank exercising effective and independent oversight of its involvement in the program, as appropriate, whether the program is housed in the Bank or externally managed? To what extent is the Bank’s oversight independent of the management of the program? To what extent does the Bank’s representative on the governing body have a clear terms of reference? 4. Risks and risk management. To what extent have the risks associated with the program been identified and are being effectively managed? For example, IEG identified the following risks in its global review:  Bank bears a disproportionate share of responsibility for governing and managing in-house programs?  Confusion at the country level between global program activities, Bank activities, and Borrower activities?  Representation of NGOs and the commercial private sector on program governing bodies?  Unclear role and application of Bank’s safeguards?  Trust-funded consultants and seconded staff representing the Bank on some program governing bodies? 5. Disengagement strategy. To what extent is the Bank engaged at the appropriate level in relation to the Bank’s new strategic framework:  Watching brief?  Research and knowledge exchange?  Policy or advocacy network?  Operational platform? To what extent is the Bank facilitating an effective, flexible, and transparent disengagement strategy for the program, in relation to the Bank’s objectives for its involvement in the program:  The program declares “mission accomplished” and closes?  The program continues and the Bank withdraws from all aspects of its participation?  The program continues and the Bank remains engaged, but the degree of the Bank’s engagement in some or all aspects (such as financing) declines over time? Annex A 52 Table A-4. Common GRPP Activities Policy and knowledge networking 1. Facilitating communica- This includes providing a central point of contact and communication among practitioners tion among practitioners who are working the sector or area of development to facilitate the sharing of analytical in the sector results. It might also include the financing of case studies and comparative studies. 2. Generating and This comprises two related activities. The first is gathering, analyzing and disseminating disseminating information, for example, on the evolving HIV/AIDS epidemic and responses to it, information and including epidemiological data collection and analysis, needs assessment, resource knowledge flows, and country readiness. The second is the systematic assembling and dissemination of knowledge (not merely information) with respect to best practices in a sector on a global/regional basis. 3. Improving donor This should be an active process, not just the side effect of other program activities. This coordination may involve resolving difficult interagency issues in order to improve alignment and efficiency in delivering development assistance. 4. Advocacy This comprises proactive interaction with policymakers and decision makers concerning approaches to development in a sector, commonly in the context of global, regional, or country-level forums. This is intended to create reform conditions in developing countries, as distinct from physical and institutional investments in public goods, and is more proactive than generating and disseminating information and knowledge. 5. Implementing Rules are generally formal. Standards can be formal or informal, and binding or conventions, rules, or nonbinding, but implementing standards involves more than simply advocating an formal and informal approach to development in a sector. In general, there should be some costs associated standards and norms with noncompliance. Costs can come in many forms, including exposure to financial contagion, bad financial ratings by the IMF and other rating agencies, with consequent impacts on access to private finance; lack of access to OECD markets for failing to meet food safety standards, or even the consequences of failing to be seen as progressive in international circles. Financing technical assistance 6. Supporting national- This is more directed to specific tasks than advocacy. This represents concrete level policy, institutional, involvement in specific and ongoing policy, institutional, and technical reform processes and technical reforms in a sector, from deciding on a reform strategy to implementation of new policies and regulations in a sector. It is more than just conducting studies unless the studies are strategic in nature and specific to the reform issue in question. 7. Capacity strengthening This refers to strengthening the capacity of human resources through proactive training and training (in courses or on-the-job), as well as collaborative work with the active involvement of developing country partners. 8. Catalyzing public or This includes improving regulatory frameworks for private investment and implementing private investments in pilot investments projects. the sector Financing investments 9. Financing country-level This refers primarily to physical and institutional investments of the type found in Bank investments to deliver loans and credits (more than the financing of studies), the benefits of which accrue national public goods primarily at the national level. 10. Financing country-level This refers primarily to physical and institutional investments of the type found in Bank investments to deliver loans and credits (more than the financing of studies) to deliver public goods such as global/regional public conserving biodiversity of global significance and reducing emissions of ozone-depleting goods substances and carbon dioxide, the benefits of which accrue globally. 11. Financing global/ This refers to financing research and development for new products and technologies. regional investments to These are generally physical products or processes — the hardware as opposed to the deliver global/regional software of development. public goods 53 Annex B Annex B. Global Research Projects (Completed and Ongoing) Total cost No. country or Year ($ Duration thematic GRP approved millions) (years) studies Research outputs Completed Explaining Growth FY00 1.52 3 89 South Asian Experience with Growth (edited volume; 2003) Explaining Growth: A Global Research Project, (edited volume; 2003) Economic Prospects of the CIS: Sources of Long Term Growth (edited volume, 2004) Sources of Growth in Latin America: What is Missing? (edited volume, 2006) Explaining Growth in the Middle East (edited volume, 2007) Diversity in Economic Growth: Global Insights and Explanations (edited volume, expected 2009) Understanding Reform: A FY01 1.74 3 42 Understanding Market Reforms Volume I: Philosophy, Politics and Multidisciplinary Approach Stakeholders (edited volume, 2005) Understanding Market Reforms Volume Two: Motivation, Implementation and Sustainability (edited volume, 2006) Understanding Economic Reforms in Africa: A Tale of Seven Nations (edited volume, 2006) Understanding Market Reform In Latin America: Similar Reforms, Diverse Constituencies, Varied Results (edited volume, 2006) Economic Reform in Developing Countries: Reach, Range, Reason (edited volume, 2008) Bridging Research and FY02 1.99 75 (includes Phase 1: 2001–03 Policy very short Phase 2: 2003–05 episode studies) Phase 3: 2006 Pilot BRP Initiative: 2007–08 Impact of Rich Countries’ FY04 1.46 3 23 Impact of Rich Countries’ Policies on Poverty: Perspectives from the Policies on Poverty: Developing World (edited volume expected 2009) Perspectives from the Developing World Annex B 54 Total cost No. country or Year ($ Duration thematic GRP approved millions) (years) studies Research outputs 1 Ongoing Promoting Innovative FY06 4.78 4 18 Programs from the Developing World: Towards Realizing the Health MDGs in Africa and Asia Development on the Move: FY07 1.78 3.5 7 Measuring and Optimizing Migration’s Economic and Social Impacts Varieties of Governance: FY08 0.11 3 TBD Effective Public Service Delivery Climate Change FY08 0.06 TBD 1/. Two more GRPs on Urbanization and on Private Sector Development have been approved by the Board at the recent meeting in Kuwait and are now in the research proposal preparation or pilot study phase. 55 Annex C Annex C. 2007 External Evaluation: Recommendations and GDN Management Response Recommendations Management response & actions OBJECTIVES AND REACH High Priority Clarify the extent to which GDN generally plans to focus on building Under Implementation: Researchers from all social science disciplines are encouraged capacity primarily in economics with other disciplines serving mostly to participate and this is clearly mentioned in calls for proposals to GDN's Awards and as an instrument to answer questions of interest or building capacity Medals Competition, RRCs and GRPs. Further discussion on this is planned as part of broadly across s the social sciences. the strategic review. Continue GDN's recent unofficial policy of encouraging the selection Under Implementation: Call for proposals encourage interdisciplinary contributions of themes in calls for papers that are more naturally interdisciplinary in through for e.g, the awards competition which accepts submissions ín five different subject supported by outreach outside of economics networks. categoríes. This will continue to be a component in the future as well. Low Priority Future Implementation: This is subjected to Board approval, and will be implemented Begin to develop measures of higher needs countries or even areas through RNPs within countries within regions to track. OUTCOMES, EFFECTIVENESS, RELEVANCE AND COST EFFECTIVENESSS, BY ACTIVITY PRCSs and GRPs: High Priority Under Implementation: Policy briefs, journal and volume publications, roundtable To disseminate GRPs' research findings to the broader policy and discussions are already undertaken to promote dissemination. Newer GRPs such as the development community. migration one "Development on the Move" include workshops with policy-makers to be To develop better delivery vehicles for RRCs, such as requiring policy conducted in each country at the start and end of the study. More emphasis on outreach briefs (accompanied by training to do so), for clarifying the policy to policy-makers planned in the new communications strategy. relevance of papers from RRCs. Future Implementation: Inclusion of policy briefs is stressed at the moment, but this component will be strengthened in the coming RRCs. Follow the findings of the "Workshop on Methodology for Comparative Under Implementation: GDN migration GRP "Development on Move" incorporates a Analysis" to promoting a diversity of methodologies and research diversity of methodologies and research designs. An Interdisciplinary Research Methods design through GRPs and other activities workshop will be held in conjunction with the conference in Brisbane (2008). These will continue in the future. RRCs and GRPs: Low Priority Consider holding some theme specific GRP workshops before or after Response: Will consider feasibility of the proposal. a disciplinary conference to emphasize capacity building of attendees Annex C 56 Recommendations Management response & actions Consider individual-level training programs and GDNet self-training Under Implementation: modules such as use of Internet resources as modules that  Training sessions are conducted for building capacity of participating researchers in accompany RRC workshops the latest GRPs (health and migration) and this practice will continue in the future.  Proposal writing toolkits are available on GDNet.  The newly developed skills-training-program to be launched in 200? in partnership with the RNPs will address this need. Restructuring the entire process of GRPs theme selection to increase Under Implementation: Themes are debated and approved by board members transparency and ranking of development contribution of the project representing different regions of the developing world.  Open calls for topic suggestions were held in the past (Impact project was selected in this manner); this practice will continue periodically in the future accompanied by specific guidelines/criteria for GRP topics that build on GDN’s comparative advantage Shortening the timelines and GRP project size Under Implementation: Current GRPs operate on a reduced timeline and size: this will continue through the proposed Inter-Regional Research Projects as well. Annual Conference: High Priority Response: Themes are determined on basis of (1) issues of current global importance To be consistent with specific track themes across years to increase (2) relevance to developing countries 3) particular importance to region where networking among clearer sets of audiences conference is located in that year. Future Implementation: To increase networking and cross-fertilization across regions beyond the conference, thematic groups with dedicated, interactive spaces on GDNet will be created. Explore publishing and distributing proceedings and papers in a post- Under Implementation: The following activities are already undertaken annually, conference CD (if possible, pre-conference) and on-line to extend the  Special conference issue of biannual newsletter Research Monitor includes excerpts long- term value of the conference of speeches, key messages, notes on workshop abstracts, etc.  CD of conference papers distributed during the conference  Conference proceedings published with Edward Elgar  Papers are widely available: online through the main website and the GDN library  Conference intranet site developed for 2007/ 2008 conference. Annual Conference: Low Priority Consider experimenting with alternative scheduling such as an 18 Under Consideration: Subject to Board approval. month schedule to allow secretariat and board to focus on additional priorities and activities, and also allow scheduling across regions. Consider additional consortia and participation in conferences aside Future Implementation: To be incl. in responsibilities of GDN's key research staff; from GDN's own and RNPs to promote networking the maintenance of global calendar of development events will ensure focus on key of GDN importance. 57 Annex C Recommendations Management response & actions GDN can pair itself with a relevant academic conference and use Response: GDN conference and related workshops last for one week so it is not feasible surveys to compare the added value versus the ABCD conference. to pair with another full conference. The ABCDE (later RBCDE) was held in conjunction with GDN's conference for 3 consecutive years  Also periodic surveys of conference participants are conducted to compare the added value of the conference to their career. To pursue both capacity building and highest-quality knowledge Under Implementation: The workshops held both pre-and post conference address this building objectives for the conference, especially by increasing concern through intense one-on-one mentoring; in addition exposing GDN's grantees to capacity building in at least selected sessions. internationally renowned speakers also acts as capacity building feature of GDN conferences. They also get feedback from the audience on their work presented during the conference. GDNet: High Priority To make GDNet a portal for Southern researchers, leading working Under Implementation: paper and database collections worldwide.  Site revamp focusing on incentives (access to online resources), networking, outreach and research comrnunications (promoting researchers profiles and work and collaborative working places (online project spaces) will underline the site revamp strategy and implementation  An aggressive marketing and outreach strategy together with a search optimization exercise will follow the new site launch to engage Southern Researchers in online networking activities  Review of current editorial and researchers' acceptance policy and processes  Continuous validation of current content and expanding sourcing effort in collaboration with regional window coordinators Change grantee contracts to include GDNet as a central location for Under Consideration: Contractual and technical changes can be easily accommodated open access to datasets funded by GDN process. However, there is often reluctance from the researchers to provide the datasets due to national legal constraints. Subject to availability, researchers' approval and legal clearance, the datasets will be made available online. Continue to pursue ways to expand journal availability through GDNet Response: Negotiations with J-Stor have been revived to allow eligible researchers considering the use of key stakeholders and board members to serve registered on the GDNet knowledgebase access to J-Stor Online Collection. Contractual as liaisons. arrangements are currently being finalized and technical gateway developed. Econ-Lit collection is currently being assessed for relevance and value. Feedback regarding recommendations for other online services from survey respondents and regional coordinators is compiled for assessment. GDNet: Low Priority Setting up an international model program of data access from national Response: This will require further assessment and validation subject to available institutions and aided by more senior researchers in the Global South. resources. It would also require a better understanding of other key players' efforts in this direction. Annex C 58 Recommendations Management response & actions GOVERNANCE, ORGANIZATION AND INTERACTION WITH REGIONAL PARTNERS AND STAKEHOLDERS Governance (Board of Directors): High Priority To develop position requirements for members who have a greater Response: Additional criteria for selection of representatives (such as previous managerial experience in managing organizations or boards and financial experience) will be presented to the Board for their consideration. Implementation is also background. constrained by the nominating organizations internal preferences and decisions. Increase the interaction with board members by extending the days of Under Implementation: Senior staff at GDN have frequent one-on-one interactions with board meetings and/or adding one or two virtual meetings per year via board members for specific projects and issues. Additional ways and means of the Internet. increasing interaction with board members will be explored, including virtual meetings. Increase the number of permanent committees such as programmatic Under Implementation: Program Committee exists. Additional committees or more and development committees to handle the business issues. extensive use of the Executive Committee to be considered by the Board. Clarify for prospective incoming board members about the appropriate Future Implementation: Next fiscal year. time expectations for board membership by frequently asked questions and using a nonbinding agreement about board member responsibilities. Providing new board members a strong orientation to the organization Future Implementation: Will be considered for newly appointed board members. within their first month on the board. Work with nominating organizations to make recruitment of board Under Implementation: Nominations are accepted through our Regional Network members more transparent. Partners and through boards of international associations represented on GDN; to our knowledge, calls for nominations are normally circulated among their members and a decentralized process is designed to ensure ownership and transparency. The need for transparency and wide consultation will be emphasized in the future. Make sure the board has a conflict of interest policy signed at least Future Implementation: Next fiscal year. once if not annually. Governance (Board of Directors): Low Priority Consider an additional donor representative to help ground truth Future Implementation: Proposal submitted to the Board for consideration. GDN's ideas from the perspective of funding. Management Processes: High Priority To remain as non-hierarchical as possible and involve partners to feel Under Implementation: Addressed in the recently completed Organizational Review. ownership. GDN should hire a consultant to work with the organization to develop Future Implementation: Lead Communications Officer position advertised; to be a coherent branding strategy and advise it regarding a recruited by April 2008. communications plan. Management Processes: Low Priority Under Implementation: Chief Economist has been recruited; provision of recruiting other Consider hiring a researcher if GDN wishes to focus more on researchers to be considered in early 2008. providing intellectual direction to new activities and oversight for conference activities. 59 Annex C Recommendations Management response & actions Consider hiring a fundraiser as a fulltime staff member or consultant Under Implementation: Discussed in the Strategic Review. based in the US or Europe. Engage more individuals to participate in planning processes. Under Implementation: Junior staff members have been included in various sub- committees of GDN's management structure and in the recruitment panels. Organizational Best Practice: High Priority Under Implementation: Human Resources consultant hired from October-December To undertake a more complete external review of human resource 2007 conducted a series of consultations and submitted a report on HR recommendation policies. Conduct a legal review of GDN's situation and options given its newly Under Implementation: Already conducted by GDN's Legal Counsel. acquired international organization status in reference to liabilities, taxes, etc. To formalize ways to how staff would handle problems with supervisors Under Implementation: Conflict resolution and anti-harassment advisor appointed; or top management. recruitment of HR Manager/Administrator will also support this process. Organizational Best Practice: Low Priority Establish clear and implemented policies on selection and supervision of Future Implementation: Will add to staff rules after new HR Manager comes on board. family or friends. Establish a policy on and training regarding sexual harassment. Under Implementation: Anti-harassment advisor has been appointed already. Review application of travel rules for consistency and cost Under Implementation: The travel rules already exist and proper care is taken to cat, r effectiveness. to the requirements of the GDN staff keeping in mind the cost effectiveness. Improve and more regularly update GDN's internal database to better Under Implementation: Projects Assistant is responsible for updating information on maintain consistency of information across staff. internal databases Undertake a market survey of similar organizations in India and Future Implementation: Planned for April-May 2008 to ensure changes to be implemented internationally to check on comparability of local and international salary by the next fiscal. rates. Examine the feasibility of adding at least a temporary additional local Future Implementation: This will be done for future conferences and has been done in staff if the conference remains annual. the past, e.g. Beijing 2007. GDN's Relationship with its RNPs: High Priority Clarify the extent to which GDN is a network of networks versus a Under Implementation: Clarity of goals and positioning, including relationships with centralized organization with networks as grantees. RNPs were examined as part of the concurrent strategic review. Widen the new pilot idea of working with RNPs in designing a coordinated Under Implementation: GDN-ERF Memorandum of Understanding provides a mode1 fundraising plan toward a dedicated work plan with joint fundraising. that will be explored in other regions. Provide greater institutional support for RNPs that rotate administrative Under Implementation: RNPs currently receive grants for overhead expenses, including centers to increase continuity during transitions. secretariat relocation. A short period of RNP leadership overlap will be encouraged. 6 Annex C 60 Recommendations Management response & actions Low Priority Look for partnerships and other ways to generate greater impact in the Future Implementation: This will be to some extent addressed through Inter-Regional Asia region in building research capacity. Research Projects and sharing of mentors across regions. Improve learning across regional networks through real or virtual meeting Under Implementation: Modalities to be discussed with RNPs for immediate outside of the annual conference. implementation. Augment input in decision-making process from RNPs on GRP themes or Under Implementation: GDN's Development on the Move project incorporated inputs candidate selection. from RNPs in the first phase on selection of country study teams and mentors. New GRP proposals to be discussed with the RNPs. Consider more detailed reporting to monitor the grant fund distribution to Under Implementation: All RNPs submit annual reports on grant recipient profiles as well researchers (that include gender, discipline of training, whether the as the breakdown of direct & indirect costs. Grantees' profiles are captured in the internal researcher is from high/low capacity area) and overhead costs (no specific database. examples of this) Consider adopting common reporting framework used by other Response: GDN's experiences suggest donors tend to use widely different reporting donors. formats. 61 Annex D Annex D. Members of the Governing Bodies Ernesto Zedillo (chair), Yale University and former President of Mexico Isher Ahluwalia, Indian Council for Research on International Economic Relations (represents SANEI) Mohammed Ariff, Malaysia Institute of Economic Research (represents EADN) Ernest Aryeetey, Institute of Statistical, Social, and Economic Research, University of Ghana (represents AERC) Abhijit Banerjee, Massachusetts Institute of Technology (represents the Bureau for Analysis of Research and Development) Eliana Cardosa, Escola de Economia da Fundacao Getulio Vargas (represents LACEA) Alan Gelb, World Bank (represents the World Bank) Ravi Kanbur, Cornell University Olav Kjorven, United Nations Development Program (represents UNDP) Masanori Kondo, International Christian University in Japan (represents GDN Japan) Mustapha Nabli, World Bank (represents the ERF) Vijay Naidu, Victoria University of Wellington (represents ODN) Pablo Andres Neumeyer, Universidad Torcuato di Tella (represents the International Economic Association) Jean-Philippe Platteau, University of Namur (represents EUDN) Vladimir Popov, New Economic School, Moscow (represents EERC) Vincenzo Porcasi, University of Triests (represents the government of Italy) Emma Porio, Ateneo de Manila University (represents the International Sociological Association) James Robinson, Harvard University and Weatherhead Center for International Affairs Lourdes Sola, University fo Sao Paulo (represents the International Political Science Association) Boris Vujcic, Croatian National Bank and University of Zagreb (represents the CERGE-EI) Annex E 62 Annex E. Sources and Uses of Funds Table E-1. Budget Expenses across GDN’s Five Core Program Activities (US$ millions) Program expenses FY02a FY03 FY04 FY05 FY06 FY07 FY08 Total Regional research 2.1 1.9 2.1 1.1 2.0 2.2 2.5 13.9 competitions Global Research 0.6 2.6 0.7 2.5 1.8 3.3 3.1 14.6 Projects Global Development 0.6 0.7 0.6 0.5 0.5 0.4 0.3 3.6 Awards Annual Conference 1.4 2.1 1.6 1.2 1.0 1.2 1.2 9.7 GDNet 0.8 0.9 1.0 0.7 0.7 1.1 1.2 6.4 Total 5.5 8.2 6.0 6.0 6.0 8.2 8.3 48.2 Source: Annual financial audits. a. From March 15, 2001 (inception) through June 30, 2002. 63 Annex E Table E-2. GDN: Donor Commitments by Activities, FY09 Accrual Basis (US$) Strategic Donor/ activity RRC GRPs AC GDNet Awards IRPs research and Secretariat Total other projects ACBF 30,000 30,000 AFESD 1,300,000 1,300,000 Austria 335,000 335,000 AusAID 50,000 110,000 10,000 15,000 185,000 DFID 656,668 1,329,040 145,960 2,131.,668 Finland 310,650 310,650 France 131,601 15,000 146,601 Gates Foundation 1,266,843 1091,260 298,040 2,656,143 IDRC 297,225 297,225 Luxembourg 887,000 79,000 37,500 1,003,500 PHRD 190,000 190,000 Netherlands 668,690 668,690 Norway 90,000 12,500 102,500 NZAID 60,000 60,000 Spain 195,000 60,000 45,000 300,000 UNDP 579,500 30,500 610,000 World Bank 2,150,000 160,000 125,000 200,000 75,000 500,000 145,000 230,000 3,585,000 Total 2,200,000 4,231,218 1,566,601 1,555,358 419,000 500,000 2,625,300 814,500 13,911,977 Source: GDN documentation. Note: Excludes $3,290,000 from 3ie for the activities and administrative costs of this program, which GDN hosts. Annex E 64 Table E-3. GDN Allocations to Regional Research Competitions (US$ thousands) % of % of % of % of % of % of % of RRC FY02 total FY03 total FY04 total FY05 total FY06 total FY07 total FY08 total AERC 300 16% 300 17% 300 16% 225 23% 275 15% 320 16% 330 17% EADN 275 14% 250 14% 250 13% - - 325 18% 260 13% 280 14% EERC 250 13% 250 14% 250 13% 150 15% 225 13% 260 13% 260 13% CERGE 250 13% 250 14% 250 13% 210 21% 225 13% 260 13% 280 14% ERF 250 13% 250 14% 250 13% 100 10% 225 13% 260 13% 270 14% LACEA 275 14% 200 11% 200 11% 125 13% 175 10% 220 11% 240 12% SANEI 325 17% 300 17% 300 16% 125 13% 275 15% 320 16% 260 13% OCEANIA - - - - 100 5% 50 5% 75 4% 100 5% 80 4% Total 1,925 1,800 1,900 985 1,800 2,000 2,000 % of GDN Revenue 23% 17% 22% 13% 27% 22% 21% GDN Revenue 8,460 10,880 8,550 7,450 6,750 9,260 9,620 % of GDN Core Funds 49% 54% 56% Core Funds 3,660 3,670 3,570 Source: GDN Secretariat. 65 Annex F Annex F. Persons Consulted Person Position GDN Secretariat Gobind Nankani President George Mavrotas Chief Economist Ramona Angelescu, Senior Political Scientist and program manager Rajesh Grover Controller Sherine Ghoneim Director, GDN Cairo Savi Mull Program Officer Shilpa Phadke Economist Carol Best-Aaron Coordinator, Washington Office Lyn Squire former President GDN Board Isher Ahluwalia Indian Council for Research on International Economic Relations Ernest Aryeetey Institute of Statistical, Social, and Economic Research, University of Ghana Olav Kjorven United Nations Development Program Pablo Andres Neumeyer Universidad Torcuato di Te Jean-Philippe Platteau University of Namur Ernesto Zedillo (chair) Yale University and former President of Mexico Carolina Hernandez (former) Emeritus, University of the Philippines Regional Network Partners Chalongphop Sussangkaru EADN Marcela Esclava LACEA Randy Filer CERGE-EI Heba Handoussa ERF William Lyakurwa AERC Donors Alan Winters DFfID (also former World Bank representative on the GDN Board) Abigail Mulhall DFID Rohinton Medhora IDRC Institute for Development Studies, University of Sussex Isabel Vogel Program Manager, Strategic Learning Initiative Overseas Development Institute (UK) and stakeholder meeting participants Simon Maxwell ODI John Young ODI Martin Prowse ODI Annex F 66 Person Position Wayne Diamond Oxfam Luke Mukubra DFID World Bank Alan Gelb DECVP and WB representative on GDN Board Iver Cederholm Senior Resource Management Officer, and WB TTL for GDN Guillermo Perry former WB representative on GDN board Paul Hubbard former Manager, DGF Secretariat Independent Evaluator Marc D. Shapiro MDS Associates 67 Annex G Annex G. Response of the Program to IEG’s Global Program Review April 2009 Introduction The Global Program Review (GPR) of the Global Development Network (GDN) was undertaken by the Independent Evaluation Group (IEG) of the World Bank in late 2008. The review assesses the relevance and effectiveness of GDN over its ten-year history and the Bank’s performance as a major partner. In making this assessment, the review drew on the findings of the GDN Independent Evaluation conducted in 2007. GDN acknowledges and welcomes the thoughtful and valuable insights generated by the review. The following section of this document presents the management response to the main findings and lessons learned. Section III captures the main areas in which strategic decisions have been and will be made on the basis of the current review’s recommendations. Management Response SUMMARY PROGRAM REVIEW CONCLUSIONS Key performance areas highlighted in the review include:  Conceptually, GDN is well-designed for pursuing its objective. Its reach is global in scope, involving researchers and partner institutions from all regions of the world.  GDN’s rapid and successful institutional transformation. Established in 1999 as an internal unit of the World Bank, GDN is now an independent, international organization located in New Delhi, India. Throughout this transformation, GDN’s main objectives and activities have remained constant.  The relevance of GDN’s overarching goal was high at the outset of the program and remains high today. Overall, GDN’s relevance is moderately strong, its record of outputs is strong, and its achievement of program outcome objectives is moderate. This is in consonance with the findings of the 2007 Independent Evaluation.  GDN’s output record is strong in each of its five areas of activity, namely: Regional Research Competitions (RRCs), Global Research Projects (GRPs), the Global Development Awards and Medals (GDAM) competition and the Global Annual Development Conferences (GDAC).  GDN has a notable history of external and internal evaluations in its relatively short existence.  Overall, evaluations show some ‘moderate evidence’ of achievements in advancing two of GDN’s three objectives — increased high-quality, policy-relevant research and enhanced research capacity — and (very) limited progress in advancing its third aim of informing policymaking (also stated in the 2007 Independent Evaluation).  GDN has handled resource allocations, administrative costs, and financial management efficiently. Annex G 68  GDN has improved its financial position in recent years by mobilizing larger and more diversified sources of funding, while some challenges persist.  The World Bank has provided important financial and technical support to GDN since its founding. Yet, Bank performance, as a partner, in strategic direction and oversight of GDN is less than wholly adequate. VALIDATION The insights from the review will play a defining role in guiding GDN to sharpen and reaffirm priorities, going forward from ten years of its operational existence. The review puts GDN’s activities in perspective in the achievement of its primary goal of generating, sharing, and applying to policy multidisciplinary knowledge for the purpose of development. Within this overarching goal, GDN appreciates the review’s useful reminder of its three core program objectives:  Generate high-quality, policy-relevant research in developing and transition countries  Build research excellence and policy outreach capacity among researchers in those countries to improve the quality and expand the policy influence of their work on a national and international level;  Promote greater linkages between researchers and the policy process to foster effective, evidence-based policy-making. More importantly, the review raises valuable insights into:  The need for GDN to develop its own explicit, state-of-the art systematic approaches for advancing each of its three program objectives and a corresponding log frame, focusing more on outcomes rather than outputs, especially on the policy front;  The potential for expanding the relationship with the RNPs — explore areas of potential for effectiveness and efficiency gains through undertakings such cross- regional capacity building, standard setting and joint fundraising;  The role of the World Bank in supporting global programs it has initiated, the duration of the support and the guidelines for continuation or withdrawal. GDN’s experience validates the review’s observation on the role of the World Bank as a global partner. GDN agrees that the research community in developing and developed countries values GDN and praises the Bank’s role in it. GDN recognizes the need for clearer guidelines for defining terms of engagement of the World Bank in global partnerships. GDN will welcome interest from the Bank in the form of a major engagement in securing its financial sustainability, and charting a clear and well-staggered exit strategy. FACTUAL UPDATE GDN would like to highlight information to the IEG for reasons of factual accuracy. Several developments since the Independent Evaluation was concluded in 2007 have not been adequately reflected in the current review. The following points merit clarifications on information provided in the report:  Relationship with the Regional Network Partners (RNPs): As clarified in the GDN Strategic Review in 2008, while GDN is a ‘network of networks’, this is not its sole 69 Annex G role and the organization is much more than a network of networks. GDN works primarily in partnerships with the Regional Network Partners to support building research excellence and generating development knowledge at the country/regional level. GDN’s partnership with the RNPs has involved strengthening both horizontal and vertical relationships and these remain a high priority for GDN. Bi-annual meetings are held with RNPs and involve their input into the larger research agenda in a peer driven and collegial manner. Through such meetings, GDN is able to engage in dialogues with RNPs and get valuable inputs without becoming too bureaucratic. However given the welcome differences across RNPs, GDN needs to work with other actors as well in support of its mission. The other complementary activities in GDN’s portfolio, the Global Research Projects (GRPs), the Global Development Awards & Medals (GDAM) competition, the Annual Conferences and the Strategic Research Partnerships which facilitate cross-regional fertilization and networking among researchers contribute to achieving GDN’s goals, with advice and input from the RNPs but not carried out directly by the RNPs in their respective regions. While GDN recognizes and validates the need to reinforce and continue to expand the relationships with RNPs, GDN also maintains direct contact with researchers and research institutions in the developing countries to allow for achieving the strategic objectives of building research excellence and shaping global policy debates.  Strengthening Institutional Capacity of RNPs: In responding to variations in capacity and performance of different RNPs, especially the newer RNPs, GDN has commenced dialogues with two of the RNPs — the Oceania Development Network (ODN) and the South Asian Network of Economic (Research) Institutes (SANEI) to strengthen their own capacity as well as the global network. Both have rotating Secretariats and institutional strengthening involves facilitating smooth transition processes for efficient, seamless operations. Discussions with the two RNPs include frequent interactions with their Executive Committees and employing a Consultant to design ways forward to strengthen institutional capacity and governance for effective institutional arrangements compatible with furthering the mission of building research excellence in the regions.  Financial Sustainability in FY09: Developments occurring in FY09 need to be reflected upon in terms of GDN’s financial sustainability. GDN’s dependence on funding from the World Bank has decreased. The funding from the Bank amounts to an estimated 25 percent of GDN’s total revenue in FY09, down from 38 percent in FY07. The recently concluded fundraising strategy involves a capital campaign plan, to be drafted within the next year to target funding from developing country governments and High Net Worth Individuals (HNWIs), further enhancing GDN’s financial position and sustainability, although the need for expanding the share of core versus project-based funding is clearly recognized.  Transition in Leadership: The succession process in GDN will take place under the able leadership of the new President, Dr. Gerardo della Paolera, effective 15th August 2009. Prior to his taking office in August 2009, GDN has clear interim management processes in place for a smooth transfer and Dr. Paolera was appointed by the GDN Board before the previous president’s departure. Dr. Paolera is known for his outstanding contribution in the past as the Founding President and Rector of the Annex G 70 Universidad Torcuato Di Tella (UTDT) in Buenos Aires, Argentina, and most recently at the American University of Paris, in both cases as President of these institutions as well as engaged in research. Both institutions grew phenomenally under his leadership. At present he is a Visiting Professor at the Central European University in Budapest and also a Visiting Fellow at the Paris School of Economics (PSE). He has a PhD in Economics from the University of Chicago, with a specialization in economic history.  Outreach and Knowledge Sharing: Besides playing an integral strategic role in generating and applying knowledge for development, GDN plays a significant role in sharing knowledge as well. GDN emphasizes that while the aspect of sharing has been discussed favourably, it has been addressed at the program level. In crystallizing the Strategic Objectives of GDN, a primary objective is to expand outreach of world class research to other researchers and to policy makers; to inform and shape policy debates in regional and global forums on selected topics, and to encourage policy makers’ exchange of ideas on selected issues. This is coupled with an effort to increase visibility with the objective of becoming a well known, reputable point of contact for reaching out to developing country policy makers and researchers.  Shaping Policy Debates: GDN underscores the expansion of its activities that fit well with its objectives outlined above. Recent activities have emphasized dissemination of policy research and engagement with policy makers. For instance, GDN is working on a newly funded project by the Bill and Melinda Gates Foundation on ‘Shaping Agricultural and Trade Policies in Sub-Saharan Africa and South Asia: Promoting Evidence Informed Policies’. GDN is also in the process of hiring several policy outreach staff and systematically requiring the production of policy briefs from completed research projects.  3ie: GDN would like to clarify that the International Initiative for Impact Evaluation (3ie) is a program within GDN titled GDN’s International Impact Evaluation Department. Lessons Learned & Action Items The review mentions lessons learned and GDN has carefully examined and considered the valuable recommendations made by the review and earmarked action items to incorporate them. Lessons Learned Action Items Effectiveness depends on a theory of change and GDN’s recent strategic planning process has strategies for outcome objectives involved the assessment of the effectiveness of its activities in achieving its strategic objectives and defining the theory of change. In doing so, GDN is developing a logical framework that would measure outcomes against the three core program objectives. The development of an evaluation design subsuming the logical framework will provide the basis for better preparation for the next Independent Evaluation of GDN. 71 Annex G Lessons Learned Action Items Global networks require well-defined roles and GDN is deepening partnerships with RNPs, responsibilities for their constituent parts reinforcing their role by empowering them with increased voice. GDN recognizes the value of intensive engagement with the RNPs and has augmented the relationship through bi-annual meetings, joint inter-regional research projects and a regional training program, the latter two being RNP driven. GDN will continue intensifying the relationships with RNPs. It is difficult to design and implement a coherent GDN acknowledges the challenge of resource global program without a level of core funding mobilization given the current world scenario and sufficient to allow for the implementation of intends to broaden the scope for additional results based strategies and the coordination of resource mobilization, particularly core funding. activities in support of those activities GDN had recently employed fundraising consultants to assess ways to increase financial stability and income sustainability, raising extra unrestricted funds for core activities and exploring the feasibility of an endowment fund. Results of the fundraising analysis indicate that GDN should embark on a capital campaign, the plan for which will be drafted and presented to the Board of Directors for approval. GDN will continue to target core funding outside of the World Bank. There are weaknesses in a governance The overarching, high-level strategic direction will arrangement in which the governing board is now be provided by the GDN Assembly. The responsible for program agenda-setting as well GDN Board has a clear role in strategic oversight as strategic direction and oversight and the RNPs will continue to be consulted with regularly (bi-annually) for program agenda- setting which is a mechanism that is working well. Other action items include:  GDNet: The GDN Strategic Review focused on GDN’s role in shaping policy debates rather than attempting to influence/inform policy debates. The GDNet strategy for 2010–2015 will be developed in light of: a) the GDN overall strategic direction; b) the GDNet goal towards communication of research knowledge generated in developing and transition countries; c) guided by the evaluation suggestion to develop a strong theory of change about how this can be implemented, d) and taking advantage of changing technologies. The GDNet strategy for 2010–2015 will involve strengthening regional ownership and working with RNPs to collectively develop institutional capacity for policy outreach.  GDAMCs: In order to document information on the outcomes of the awards and medals in terms of outputs and policy outreach, GDN will commission an evaluation of the Awards and Medals Competition in 2009. The evaluation will assess the outcomes of the research awards, the medals and effectiveness of the competition towards policy outreach and career advancement of awardees. The Global Program Review Series The following reviews are available from IEG. Volume #1, Issue #1: ProVention Consortium Issue #2: Medicines for Malaria Venture Issue #3: Development Gateway Foundation Issue #4: Cities Alliance Volume #2, Issue #1: Critical Ecosystem Partnership Fund Issue #2: Association for the Development of Education in Africa Issue #3: Population and Reproductive Health Capacity Building Program Issue #4: International Land Coalition Volume #3, Issue #1: Consultative Group to Assist the Poor Issue #2: Global Development Network GPR The Global Development Network (GDN) was launched in December 1999 as global pro- gram with its Secretariat in the World Bank. GDN is now an independent international organ- ization with its headquarters in New Delhi, India. From the outset, GDN’s program objectives have been to generate policy-relevant development research from within developing and transitional countries, to build the research and policy outreach capacities of researchers in those countries, and to promote the use of research in policy-making processes. To achieve these objectives, GDN has sponsored five core activities: (1) regional research competitions, (2) global research projects, (3) an annual conference, (4) a Global Development Awards and Medal Competition, and (5) GDNet—a Web-based source of knowledge, information, and services. A 2007 external evaluation concluded that GDN had provided relevant servic- es that showed evidence of research capacity built and knowledge generated, along with limited evidence of enhanced outreach to policy makers. IEG’s Global Program Review has also found GDN’s program objectives to be relevant, but progress on advancing these objec- tives to be more modest. This modest record reflects the lack of explicit, state-of-the-art strategies that incorporate GDN’s various activities into systematic approaches for advancing its objectives. In addition, GDN had not fully developed effective working relations with its 11 constituent regional network partners—an issue acknowledged by GDN as a priority in its own 2008 strategic review.