RPR 57637 THE ASSOCIATION FOR THE DEVELOPMENT OF EDUCATION IN AFRICA REGIONAL PROGRAM REVIEW Volume 2 Issue 2 THE WORLD BANK GROUP WORKING FOR A WORLD FREE OF POVERTY The World Bank Group consists of five institutions--the International Bank for Reconstruction and Development (IBRD), the International Finance Corporation (IFC), the International Development Association (IDA), the Multilateral Investment Guarantee Agency (MIGA), and the International Centre for the Settlement of Investment Disputes (ICSID). Its mission is to fight poverty for lasting results and to help people help themselves and their envi- ronment by providing resources, sharing knowledge, building capacity, and forging partnerships in the public and private sectors. THE INDEPENDENT EVALUATION GROUP Improving Development Results Through Excellence in Evaluation The Independent Evaluation Group (IEG) is an independent, three-part unit within the World Bank Group. IEG-World Bank is charged with evaluating the activities of the IBRD (The World Bank) and IDA, IEG-IFC focuses on assessment of IFC's work toward private sector development, and IEG-MIGA evaluates the contributions of MIGA guarantee projects and services. IEG reports directly to the Bank's Board of Directors through the Director-General, Evaluation. The goals of evaluation are to learn from experience, to provide an objective basis for assessing the results of the Bank Group's work, and to provide accountability in the achievement of its objectives. It also improves Bank Group work by identifying and disseminating the lessons learned from experience and by framing recommendations drawn from evaluation findings. The Association for the Development of Education in Africa March 3, 2008 Corporate and Global Evaluations and Methods http://www.globalevaluations.org ©2008 Independent Evaluation Group, The World Bank Group 1818 H Street NW Washington DC 20433 Telephone: 202-458-4497 Internet: http://www.globalevaluations.org E-mail: grpp@worldbank.org All rights reserved This volume is a product of the staff of the Independent Evaluation Group (IEG) of the World Bank Group. It is part of an ongoing series that reviews global and regional partnership programs in which the World Bank is engaged as one of the partners. The findings, interpretations, and conclusions expressed in this volume do not necessarily reflect the views of the Executive Directors of The World Bank or the governments they represent. IEG does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of IEG concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this publication is copyrighted. IEG encourages the dissemination of its work and permits these reviews to be copied or otherwise transmitted, with appropriate credit given to IEG as the authoring body. Cover photo: Children at a school in Burkina Faso. Photo by Ray Witlin, 1976, courtesy of the World Bank Photo Library. ISBN-13: 978-1-60244-106-4 ISBN-10: 1-60244-106-5 Printed on Recycled Paper Independent Evaluation Group Knowledge Programs and Evaluation Capacity Development (IEGKE) E-mail: grpp@worldbank.org Telephone: 202-458-4497 IEG Mission: Improving Development Results Through Excellence in Evaluation The Independent Evaluation Group (IEG) of the World Bank reviews global and regional partnership programs (GRPPs) in which the Bank is engaged as one partner among many for two main purposes: (a) to provide accountability in the achievement of the program's objectives by providing an independent opinion of the program's effectiveness, and (b) to identify and disseminate lessons learned from the experience of individual GRPPs. The preparation of a global or regional program review (GPR) is contingent on a recently completed evaluation of the program, typically commissioned by the governing body of the program. The first purpose includes validating the findings of the GRPP evaluation with respect to the effectiveness of the program, and assessing the Bank's performance as a partner in the program. The second purpose includes assessing the independence and quality of the GRPP evaluation itself and drawing implications for the Bank's continued involvement in the program. Assessing the quality of GRPP evaluations is an important aspect of GPRs, since encouraging more consistent evaluation methodology and practice across Bank-supported GRPPs is one of the reasons why IEG embarked on this new product in 2005. IEG annually reviews a number of GRPPs in which the Bank is a partner. In selecting programs for review, preference is given to those that are innovative, large, or complex; those that are relevant to upcoming sector studies; those for which the Executive Directors or Bank management have requested reviews; and those that are likely to generate important lessons. IEG also aims for a representative distribution of GPRs across sectors in each fiscal year. A GPR is a "review" and not a full-fledged "evaluation." It assesses the independence and quality of the relevant evaluation; provides a second opinion on the effectiveness of the program; assesses the performance of the Bank as a partner in the program; and draws lessons for the Bank's engagement in global and regional programs. The GPR does not formally rate the various attributes of the program. A GPR involves a desk review of key documents, consultations with key stakeholders, and a mission to the program management unit (secretariat) of the program if this is located outside of the World Bank or Washington, DC. Key stakeholders include the Bank's representative on the governing body of the program, the Bank's task team leader (if separate from the Bank's representative), the program chair, the head of the secretariat, other program partners (at the governance and implementing levels), and other Bank operational staff involved with the program. The writer of a GPR may also consult with the person(s) who conducted the evaluation of the GRPP. Each GPR is subject to internal IEG peer review, Panel review, and management approval. Once cleared internally, the GPR is reviewed by the responsible Bank department and the secretariat of the program. Comments received are taken into account in finalizing the document, and the formal management response from the program is attached as an annex to the final report. After the document has been distributed to the Bank's Board of Executive Directors, it is disclosed to the public on IEG's external Web site. ii Abbreviations and Acronyms ADEA Association for the Development of (Association pour le Développement de Education in Africa l'Education en Afrique) AfDB African Development Bank (Banque Africaine de Développement) AU African Union (Union Africaine) CIDA Canadian International Development Agency (Agence Canadienne de Développement International) COMEDAF Conférence des Ministres de l'Education de (Conference of Education Ministers of the L'Union Africaine African Union) DEA Donors to African Education (Bailleurs de fonds pour l'éducation en Afrique) DGF Development Grant Facility (World Bank) ECCAS Economic Community of Central African (Comité Economique des Etats de l'Afrique States Centrale) ECOWAS Economic Community Of West African Communauté Economique des Etats de l'Afrique States de L'Ouest EFA Education for All (Education pour Tous) FAWE Forum for African Women Educationists (Forum des éducatrices africaines) ERNESA Education Research Network in Eastern and Southern Africa ERNWCA Education Research Network for West and Central Africa FTI Fast Track Initiative (Initiative pour la mise en oeuvre accélérée) GPR Global Program Review (IEG) (Revue Globale de Programme) GRPP Global and Regional Partnership Programs (Programmes globaux et régionaux de partenariat) HEDCO Higher Education for Development (Education Supérieure pour le Développement Cooperation de la Coopération) HDNED Human Development Network Education (Réseau des Resources Humaines pour (World Bank) l'Education) IDRC International Development Research Centre (Centre de Recherche pour le Développement (Canada) International-Canada IEG Independent Evaluation Group, formerly (Groupe d'Evaluation Indépendent, Banque OED (World Bank) mondiale) IIEP International Institute for Educational (Institut international pour la planification Planning éducative) ILO International Labor Organization (Organisation Internationale du Travail) IOA Institutional and Organizational Assessment (Evaluation Institutionnelle et Organisationnelle) NEPAD New Partnership for Africa's Development NGO Non-Governmental Organization (Organisation Non-Gouvernementale) RPR Regional Program Review (IEG) (Revue de programme régional) SADC Southern African Development Community (Communauté de Développement de l'Afrique Australe) SC Steering Committee (ADEA) (Comité de Coordination) TTL Task Team Leader (Chargé de Projet) WG Working Group (ADEA) (Groupe de travail) Director-General, Independent Evaluation Mr. Vinod Thomas Director, Independent Evaluation Group (World Bank) Ms. Cheryl Gray Manager, Corporate and Global Evaluations and Methods Mr. Mark Sundberg Global Programs Coordinator Mr. Chris Gerrard Task Manager Mr. H. Dean Nielsen iii Contents ABBREVIATIONS AND ACRONYMS................................................................................................... II PROGRAM AT A GLANCE: ASSOCIATION FOR THE DEVELOPMENT OF EDUCATION IN AFRICA ..................................................................................................................................................V KEY BANK STAFF RESPONSIBLE DURING PERIOD UNDER REVIEW .........................................VI GLOSSARY..........................................................................................................................................VII PREFACE..............................................................................................................................................IX EXECUTIVE SUMMARY.......................................................................................................................XI 1. PROGRAM OBJECTIVES, ACTIVITIES, AND COSTS........................................................... 1 Objectives................................................................................................................................. 1 Activities................................................................................................................................... 1 ADEA Vehicles ......................................................................................................................... 3 Governance .............................................................................................................................. 5 Costs and Financing ............................................................................................................... 6 2. THE 2005 INDEPENDENT EVALUATION OF ADEA.............................................................. 9 Evaluation Process ................................................................................................................. 9 Monitoring and Evaluation Framework ................................................................................. 9 Main Findings and Recommendations of the Evaluation ................................................. 10 ADEA's Response ................................................................................................................. 10 3. THE EFFECTIVENESS OF THE PROGRAM......................................................................... 12 Relevance ............................................................................................................................... 12 Efficacy ................................................................................................................................... 13 Objective: Improving Educational Policies.......................................................................................13 Objective: Enhancing Development Agency Coordination ..............................................................16 Program: Promoting Policy Dialogue...............................................................................................17 Program: Promoting Analytical Work...............................................................................................19 Program: Strengthening Capacities in African Countries.................................................................21 Program: Promoting Regional and Sub-Regional Networking .........................................................22 Conclusion.......................................................................................................................................23 Efficiency................................................................................................................................ 24 Governance and Management ............................................................................................. 24 4. WORLD BANK'S PERFORMANCE AS A PARTNER........................................................... 27 iv 5. LESSONS LEARNED............................................................................................................. 29 ANNEX A. EVALUATION FRAMEWORK FOR GLOBAL PROGRAM REVIEWS............................ 31 ANNEX B. KEY DATES IN ADEA HISTORY ..................................................................................... 38 ANNEX C. ADEA WORKING GROUPS ............................................................................................. 39 ANNEX D. ADEA BIENNIAL MEETINGS, 1991­2006....................................................................... 41 ANNEX E. ADEA ORGANIZATIONAL CHART.................................................................................. 42 ANNEX F. TECHNICAL AND SUB-REGIONAL MEETINGS, 2003­2007......................................... 43 ANNEX G. USES OF ADEA FUNDS, 2001­2006 .............................................................................. 44 ANNEX H. IEG ASSESSMENT OF THE QUALITY OF ADEA'S ANALYTICAL WORK .................. 45 ANNEX I. PERSONS CONSULTED ................................................................................................... 46 ANNEX J. OFFICIAL RESPONSE OF ADEA TO IEG'S GLOBAL PROGRAM REVIEW ................ 48 Boxes Box 1. ADEA's New Vision Statement, Mission Statement, and Strategic Objectives ................. 2 Box 2. ADEA and African Policies on the Language of Instruction ............................................. 14 Box 3. "The Culture of Quality" and National Education Plans: A Disconnect?......................... 17 Tables Table 1. External Evaluation Recommendations and ADEA Response....................................... 11 Table 2. World Bank Task Team Leader Views of ADEA: Frequencies ....................................... 14 Table 3. ADEA Published Studies, by Working Group and Time Period ..................................... 20 Annex Table 1. Assessing the Independence and Quality of the Evaluation .............................. 31 Annex Table 2. Providing an Independent Opinion on the Effectiveness of the Program......... 32 Annex Table 3. Assessing the Bank's Performance as a Partner in the Program...................... 37 Figures Figure 1. Annual Contributions to ADEA (Restricted and Unrestricted)........................................ 7 Figure 2. ADEA Use of Funds 2001­2006 .......................................................................................... 8 Figure 3. Sample of ADEA Analytical Work..................................................................................... 20 v Program at a Glance: Association for the Development of Education in Africa Start date ADEA, originally called the Donors to African Education, was founded in 1988. It changed its name in 1997 to the Association for the Development of Education in Africa (ADEA). Original · Improving educational policies; Objectives · Enhancing development agency coordination. Activities ADEA addresses the original set of implicit objectives though: · The loosely coordinated actions of working groups; · Regular biennial meetings of the agency and ministry partners; · Meetings of the Steering Group and Bureau of Ministers; · Special or sub-regional meetings; · Training or capacity building activities, · Publication and information dissemination (including databases). WBG During ADEA's first three years (1988­1990), in-kind contributions by the WBG contributions and budget support amounted to about $445,000 (about 18 percent of total development agency support of $2.5 million). After 1991 Bank financial support came largely through the Special Grants Program (renamed Development Grant Facility in 1998). By the early years of the current decade (2000­2005), DGF funding amounted to about $300,000­350,000 per year (or about 6­14 percent of total development agency support). In 2006 DGF funding increased from $350,000 to $1.0 million, although $400,000 of this $650,000 increase was for activities already financed through other channels (a $250,000 on-grant to FAWE and $150,000 for the Working Group on Higher Education). Other See above. All development agencies participating in ADEA in recent years development have been expected to contribute at least $50,000 annually as a membership agency fee (many contribute more). In addition, partners earmark contributions to contributions specific activities or working groups through restricted funds. Total contributions to ADEA in the current decade have increased from $2.5 million in 2000 to $6.5 million in 2006. During that period restricted funds have exceeded unrestricted (membership) funds by about 2­3.5 to 1. Location An informal Secretariat was housed at the World Bank in Washington, DC, from 1988­91. Then a formal Secretariat was formed and hosted by UNESCO's International Institute of Educational Planning (IIEP) in Paris. Beginning in 2008 the Secretariat will be hosted by the African Development Bank in Tunis, Tunisia. Since 1997 all Biennial Meetings have been held in Africa. Governance and ADEA is governed by a Steering Committee consisting of 21 development management agencies and 10 African ministers of Education (the Bureau of Ministers). Its day-to-day operations are managed by its Secretariat, consisting of an Executive Secretary and 5­6 staff members. World Bank officials (Task Team Leaders, TTLs) have also held a place on the Steering Committee; in 2005 a World Bank official (Sector Manager) was elected by the Steering Committee to be its Chair. Latest program- Evaluation of the Association for the Development of Education in Africa level evaluation (ADEA), December 2005, conducted by the private firm, Universalia of Montreal, Canada. vi Key Bank Staff Responsible during Period under Review Position Person Period Task Team Leader Peter Moock FY90­FY95 Avril Van Adams FY96­FY98 Adriaan Verspoor FY99­FY00 Birger Fredriksen FY01­FY04 Dzingai Mutumbuka FY05­FY08 Jee-Peng Tan FY08 ­ present Director of Human Maris O'Rourke February 1997 ­ September 2000 Development Network, Marlaine Lockheed September 2000 ­ June 2001 Education Team Ruth Kagia March 2001 ­ present Vice President of HD Network Eduardo A. Doryan June 1999 ­ August 2005 Jean-Louis Sarbib July 2003 ­ July 2006 Joy Phumaphi February 2007 ­ present Trust Fund Operations Arif Zulfiqar, Director June 1999 ­ present Global Programs & Margret Thalwitz, Director May 2004 ­ present Partnerships Program Manager Position Person Period Program Manager (Executive Peter Moock 1988­1992 Secretary) Chris Shaw 1992­1995 Richard Sack 1995­2001 Mamadou Ndoye 2001 ­ present vii Glossary Devolution or exit A proactive strategy to change the design of a program, to devolve some of its strategy implementation responsibilities, to reduce dependency on external funding, or to phase out the program on the grounds that it ha achieved its objectives or that its current design in no longer the best way to sustain the results which the program has achieved. Donor Any organization or entity that makes a financial or in-kind contribution to a program that is reflected in the audited financial statements of the program. Therefore, this includes not only "official donors" but also developing countries that contribute annual membership dues, seconded staff, or office space, provided that these are formally recognized in the financial statements of the program. Another term, often used synonymously but more inclusive, is "development agency." Efficacy The extent to which the program has achieved, or is expected to achieve, its objectives, taking into account their relative importance. The term is also used as a broader, aggregate measure -- encompassing relevance and efficiency as well -- of the overall outcome of a development intervention such as a Global and Regional Partnership Programs (GRPP). Efficiency The extent to which the program has converted or is expected to convert its resources/inputs (such as funds, expertise, time, etc.) economically into results in order to achieve the maximum possible outputs, outcomes, and impacts with the minimum possible inputs. Evaluation The systematic and objective assessment of an ongoing to completed policy, program, or project, its design, implementation, and results. The aim is to determine the relevance and achievement of its objectives, and its developmental effectiveness, efficiency, impact, and sustainability. Governance The structures, functions, processes, and organizational traditions that have been put in place within the context of a program's authorizing environment to ensure that the program is run in such a way that it achieves its objectives in an effective and transparent manner. It is the framework of accountability and responsibility to users, stakeholders and the wider community, within which organizations take decisions, and lead and control their functions, to achieve their objectives. Impacts Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended. Independent evaluation An evaluation that is carried out by entities and persons free from the control of those involved in policy making, management, or implementation of program activities. This entails organizational and behavioral independence, protection from interference, and avoidance of conflicts of interest. Indicator A quantitative or qualitative factor or variable that provides a simple and reliable means to measure achievement, to reflect the changes connected to an intervention, or to help assess the performance of a development actor. Legitimacy As a criterion for assessing governance and management, the way in which governmental and managerial authority is exercised in relation to those with a legitimate interest in the program -- including shareholders, other stakeholders, implementers, beneficiaries, and the community at large. Logical framework or A management technique that is used to develop the overall design of a program or project, logframe to improve implementation monitoring, and to strengthen evaluation, by presenting the essential elements of the program or project clearly and succinctly throughout its cycle. It is a "cause and effect" model which aims to establish clear objectives and strategies based on a results chain, to build commitment and ownership among the stakeholders during the preparation of the program or project, and to relate the program's or project's interventions to their intended outcomes and impacts for beneficiaries. Management The day-to-day operation of the program within the context of the strategies, policies, processes, and procedures that have been established by the governing body. viii Monitoring The continuous assessment of progress achieved during program implementation in order to track compliance with a plan, to identify reasons for noncompliance, and to take necessary actions to improve performance. Monitoring is usually the responsibility of program management and operational staff. Outcomes The achieved or likely short-term and medium-term effects of the outputs of a development intervention. Oversight One of the core functions of the governing body of a program: Monitoring the performance of the program management unit, appointing key personnel, approving annual budgets and business plans, and overseeing major capital expenditures. Partners Stakeholders who are involved in the governance or financing of the program (including the members of the governing, executive, and advisory bodies). Public goods Goods which produce benefits that are non-rival (many people can consume, use, or enjoy the good at the same time) and non-excludable (it is difficult to prevent people who do not pay for the good from consuming it). If the benefits of a particular public good accrue across all or many countries, then the good is deemed a global or international public good. Relevance The extent to which the objectives and design of the program are consistent with (a) the current global/regional challenges and concerns in a particular development sector and (b) the needs and priorities of beneficiary countries and groups. Shareholders The subset of donors that are involved in the governance of the program. Therefore, this does not include individual (particularly anonymous) donors who choose not to be so involved, or who are not entitled to be involved if their contribution does not meet the minimum requirement, say, for membership on the governing body. Stakeholders The parties who are interested in or affected, either positively or negatively, by the program. Stakeholders are often referred to as "principal" and "other", or "direct" and "indirect". While other or indirect stakeholders -- such as taxpayers in both donor and beneficiary countries, visitors to a beneficiary country, and other indirect beneficiaries -- may have interests as well, these are not ordinarily considered in evaluations unless a principal stakeholder acts as their proxy. Sustainability When the term is applied to the activities of a program, the extent to which the benefits arising from these activities are likely to continue after the activities have been completed. When the term is applied to organizations or programs themselves, the extent to which the organization or program is likely to continue its operational activities over time. Transparency As a criterion for assessing governance and management, the extent to which a program's decision-making, reporting, and evaluation processes are open and freely available to the general public. This is a metaphorical extension of the meaning used in physical sciences -- a "transparent" objective being one that can be seen through. Value for money The extent to which a program has obtained the maximum benefit from the outputs and outcomes it has produced with the resources available to it. Source: Sourcebook for Evaluating Global and Regional Partnership Programs: Indicative Principles and Standards. Independent Evaluation Group ­ World Bank, 2007. ix Preface ADEA was established by the World Bank in 1988 in response to an influential policy paper issued the previous year on education in sub-Saharan Africa, Education in Sub-Saharan Africa: Policies for Adjustment, Revitalization, and Expansion. Originally the Association was created as an informal network of development agencies in Africa (Donors to African Education, DAE) having an informal secretariat at the World Bank, but by 1994 (when its Principles of Association were formalized) it grew into a professional association of both development agencies and African ministries of education. In a further evolution, in 1997 it changed its name to the Association for the Development of Education in Africa (ADEA). With this evolution came institutionalization: whereas in the early years administrative support was provided by the World Bank, by 1992 a free-standing Secretariat was created for this purpose, placed in Paris where it was linked to UNESCO's International Institute for Educational Planning (IIEP). At the request of the World Bank members of the Steering Committee, an external evaluation was undertaken 2004­05, given the fact that the Association had been financially supported by Development Grant Facility (DGF) funds for more than five years. The program commissioned a private company, Universalia, headquartered in Montreal, Canada, to undertake the evaluation, which submitted its final report to ADEA in December of 2005. This Regional Program Review (RPR) assesses the quality and independence of the 2005 evaluation of ADEA, provides a second opinion on the effectiveness of the program, assesses the performance of the Bank as a partner in the program, and draws lessons for the future operation of the program. IEG chose ADEA for an RPR because of the importance of the sector in the Bank's mission of poverty reduction, because IEG has done relatively few reviews of regional programs, and because IEG had not conducted a global or regional program review in the field of education before. This Regional Program Review included: A review of ADEA documents, reports, newsletters, proceedings (of meetings), databases, and published materials (books, and pamphlets); A visit to the ADEA Secretariat and interviews with all ADEA professional staff, plus relevant officials at its host institution IIEP; Interviews with representatives from development agency partner institutions: UNESCO, AfDB, UNICEF, and the Rockefeller Foundation; Interviews with Bank team leaders and members of the Steering Committee past and present; Interviews with academics and NGO representatives who have conducted research on education in Sub-Saharan Africa; Telephone survey of 9 World Bank TTLs covering education in sub-Saharan Africa (5 who have attended ADEA events and 4 who have not), distributed by sub-region. Interviews with 4 ministers or ex-ministers of education of selected African countries (including the current and a past chair of the Bureau of Ministers). x IEG gratefully acknowledges all those who made time for interviews, in particular ADEA governance members, management, and staff. A list of people consulted can be found in Annex I. Copies of the draft RPR were sent for comments to the ADEA Secretariat, to the Bank's global program task team leader, to AFTHD (which is responsible for the Bank's engagement with ADEA), and to other Bank units that have responsibility for the Bank's involvement with global and regional programs more generally. Their observations have been taken into account in finalizing this GPR. The formal response received from ADEA management can be found in Annex J. xi Executive Summary Program Objectives, Activities, Governance, and Costs 1. When ADEA was created in 1988 as one response to the Bank's landmark policy paper on the crisis in sub-Saharan African Education, it established the general goals of stimulating discussion of education policies in Sub-Saharan Africa and improving donor coordination in the sector. During the next 17 years the Association frequently articulated more specific goals and aims, but never established a formal set of objectives and related outcomes measures. This became an issue during the 2005 external evaluation, which recommended that ADEA develop "a strategic plan and business plan to operationalize ADEA's vision, set clear expectations, and monitor progress towards results." The ADEA Secretariat drafted such plans during 2006 and the ADEA Steering Committee formally adopted a new vision statement, mission statement, and five strategic objectives in October 2007 (Box 1). Since this occurred after the present review was conducted, this review has assessed the effectiveness of ADEA against its two original goals, with one modification. The review will substitute "improvement of educational policies" for "discussion of educational policies" since it is clear from ADEA's documents that policy strengthening is what the discussion (dialogue) has been all about. Thus, the present review has regarded (a) improving educational policies and (b) enhancing development agency coordination as the imputed strategic objectives of ADEA before October 2007. 2. From the outset the Association aspired to influence educational policy and development agency coordination in the region by stimulating policy dialogue informed by research (analysis of existing problems and needs, and innovative solutions). Gradually it also became aware of the need to build greater capacity for policy formulation and implementation and to promote enduring networks. In 2003 ADEA released a "visioning document," in which the following four challenges were set out as the work of the Association: (a) Promoting policy discussions (or dialogue): informed discussion across and within countries, between government and non-government organizations, and between country representatives and development agencies with the purpose of identifying common problems, formulating regional and national priorities, demonstrating and comparing innovative solutions, and building consensus; (b) Promoting analytical work: applied research using the "praxis approach" to promote understanding of educational conditions and constraints; identify, evaluate and compare solutions to problems; and analyze educational outcomes, typically undertaken within working groups or in preparation for a regional or sub-regional meeting; (c) Strengthening capacities in African countries: explicit or implicit efforts to improve local capacity for strategic planning and related technical skills, generally undertaken by working groups, or in the context of anchoring ADEA to African institutions; xii (d) Promoting regional and sub-regional networks: efforts to build new self-standing regional education development organizations (e.g., through graduating working groups), to link ADEA working groups to African institutions; and to partner with and contribute to regional networks and movements, such as those connected to the African Union. 3. The vehicles through which this work was undertaken have over the years consisted of the following: (a) working groups (originally 9, now 11 plus 4 ad hoc groups, considered to be the "laboratories" of ADEA; (b) biennial meetings (the premier regional event involving all partners every other year); (c) Steering Committee and Bureau of Ministers meetings (held every six months to provide guidance and decision making); (d) technical or sub-regional meetings (3­4 day events at request of Ministers on urgent topics); (e) capacity building (usually done by working groups, or through exchanges and study tours); and (f) publications and information dissemination (including management of data bases). 4. ADEA's Steering Committee is comprised of ten African ministers of education and representatives of most multilateral, bilateral and private development organizations that work in the education sector in Africa (currently 21). The Steering Committee is both ADEA's governing body and its primary facilitator of coordination among partners. The ten ministers of education who sit on the Steering Committee constitute the ADEA Bureau of Ministers. They are chosen by the Caucus of Ministers which consists of all the ministers of education in sub-Saharan Africa. Administrative support for ADEA is provided by the ADEA Secretariat, which since 1992 has been hosted physically and administratively by the International Institute for Educational Planning (IIEP) in Paris, a UNESCO affiliated organization. In 2008 the Secretariat will move to Tunis, Tunisia, where it will be hosted by the African Development Bank. When the Steering Committee is not in session, decisions about ADEA are made by the Bureau of ADEA (also referred to as the Bureau of the Steering Committee), which consists of the Steering Committee Chair, the Alternate Chair (also head of the Bureau of Ministers), the Vice-Chair (ex-officio and head of the host agency), and the Executive Secretary of ADEA. 5. ADEA is funded almost entirely through its 20­22 (the number has fluctuated recently) development agency partners, with some support coming from the African ministries. In recent years a minimum agency contribution of $50,000 has been required for joining and remaining in the partnership. These amounts are "unrestricted" or core funding and can be allocated by ADEA according to its needs. Contributors also pledge "restricted" funds, which are generally earmarked for particular working groups. Restricted funding has exceeded unrestricted by as much as 3 to 1 in recent years. Funding (especially restricted) fluctuates from year to year, which places constraints on long-term planning. Some countries (namely, Canada, the Netherlands, Finland and Sweden) have worked to moderate this by signing multiyear funding agreements, but this has not become the norm. The estimated funding for calendar year 2006 -- about $6.6 million, some of which is pledged for subsequent years -- is the highest level of funding ever (a 150 percent increase in funding over that for the year 2000 in nominal terms), reflecting an expansion in ADEA programs and donor support. xiii The 2005 Independent Evaluation of ADEA 6. The ADEA Steering Committee commissioned an independent evaluation in 2005, which was undertaken by Universalia, a Canadian management consulting firm based in Montreal. This was the third evaluation of ADEA, the first being in 1994, when the Association was still known as the DAE, and the second in 2000, which focused explicitly on the ADEA working groups. The cost of the evaluation was covered by exceptional funds provided by the World Bank (DGF) and CIDA. 7. The present IEG review found the 2005 evaluation to have been executed independently and objectively. While the evaluation sought and responded to feedback from ADEA stakeholders on factual matters, there was no sign of manipulation of findings or recommendations by them. One of the findings of the Universalia evaluation was that ADEA had never formulated a well defined strategic plan nor established a clear framework for monitoring and evaluation (with performance indicators to track progress). This made the evaluation less results-oriented than it might have been. Also, on some points the evaluation simply reports stakeholders' perceptions of Associations' organizational effectiveness instead of contextualizing such inputs and using them together with other evidence to draw conclusions (triangulation). 8. Universalia presented 23 findings and eight main recommendations, which ADEA partners interviewed for this review generally considered to be fair and constructive. The eight recommendations covered strategic planning, improvement of management and governance, successor planning for the Executive Secretary, and a feasibility study for the relocation of the Secretariat to Africa. ADEA took the recommendations seriously, having drafted a new strategic plan for 2008­2012 during 2006 and completed succession planning and the feasibility study. The governance improvements are still pending, since some of these were dependent upon first adopting the new strategic plan, which occurred in October 2007. The Effectiveness of the Program 9. Relevance. The founding aims of ADEA of improving education policies and enhancing donor coordination, taken by this review as its strategic objectives, were derived from the 1987 Education Sector Policy Paper and thus highly relevant to the conditions portrayed in it (inadequate education policies, and growing but uncoordinated and often conflicting donor influence). However, over the years perceptions and circumstances have changed, creating a need for revisiting the continuing relevance of these objectives; for example the need to make the association more country-driven, to create a more specific set of evaluable objectives, and to respond and contribute to the agendas of new regional organizations, such as those sponsored by the African Union. In general ADEA was appropriately designed to address its original strategic objectives. However, its design has not included a clear monitoring and evaluation framework for tracking how well these objectives have been met, and management structures are becoming inadequate for the growing size and complexity of ADEA (see also Governance and Management). 10. Efficacy. With respect to the strategic objective of improving educational policies, there is some anecdotal evidence that ADEA programs have had an impact at the country level, especially on girls' education and other areas where working groups have been xiv especially active. However, since ADEA has not set up systems to assess the attainment of such outcomes, it is not possible to make a full judgment about the achievement of this objective. Similarly, on development agency coordination, there are indications of improvement, including more interaction among donors, more co-sponsoring of regional programs and the availability of an ADEA database covering agency projects (PRISME), but no systematic evidence. Reinforcing a recommendation from the 2005 external evaluation, this review indicates the need for better strategic planning, including the specification of objectives, and tracking of outcomes indicators. 11. With respect to programs, ADEA has had a strong record in promoting policy dialogue, and has become the region's premier forum for evidence-based policy discussions -- possessing unmatched "convening power." The policy dialogue space that it has carved out (its "social capital") is characterized by high levels of trust, informality and openness, and the lack of political and ideological constraints. Its promotion of analytical work is evidenced by increasing volumes of studies for its biennial meetings and the output of its working groups. The volume and breadth of the work has clearly increased, but there are questions about its quality. Some working groups have produced few publications, calling into question whether they are contributing to regional education development. On strengthening capacities in Africa countries, there is evidence that ADEA has made substantial contributions, especially through the efforts of the most vigorous working groups, through regional and sub-regional meetings, and through the anchoring of ADEA programs in African institutions. Nevertheless, the reach of ADEA's capacity building has been limited, and mainly available to those directly involved in WG activities or attending regional and sub-regional meetings. In promoting regional and sub-regional networking ADEA's accomplishments have also been notable, as a "networker of networks," but not as strong as had been hoped given that only one of its working groups (FAWE) has "graduated" into a self-reliant regional entity. Also, new ad hoc working groups have less potential for spawning sustainable regional networks than the original variety. 12. Efficiency. There have been no cost-benefit analyses of ADEA products, given the difficulty in placing a monetary value on Association outcomes, and no cost-effectiveness ratios, given the lack of quantitative measures of results. However, analysis of ADEA financing has led to some concerns about efficiency. Of the two forms of ADEA funding, unrestricted and restricted (earmarked for specific programs like working groups), the latter has predominated, particularly in recent years. This restricts the capacity of ADEA management to allocate funds according to performance. Also, there are ongoing concerns, initially raised in the 2000 evaluation, that working groups are not held accountable for their funds. Solutions to these endemic problems are needed if ADEA is to operative efficiently. 13. Governance and Management. There are strong indications that ADEA has a legitimate status as a regional entity, both among development agencies and government ministries. Its governance model has a mix of shareholder (for development agencies) and stakeholder (for ministries) elements, which works to enhance legitimacy. The fact that agencies have twice as many seats on the Steering Committee than the ministries could raise equity issues, but in recent years priorities have been set by the ministries, suggesting that this imbalance does not undercut ministry influence. The bigger issue is the endemic problem of frequent turnover among ministry representatives, which needs to be addressed. Still ADEA's 31 member Steering Committee provides vigorous leadership all on a voluntary xv basis. However, it is overstretched, forcing it to focus more on procedural issues than strategic planning and programming. In addition, decision making by consensus constrains the Steering Committee's capacity to make difficult decisions. As noted in the 2005 external evaluation, the Secretariat is also overloaded, given dramatic increases in ADEA's scope and programs, suggesting the need for improved managerial resources and efficiency. World Bank's Performance as a Partner 14. The World Bank has played numerous roles in ADEA, from co-founder, to initial host institution, to lead financial supporter, to source of leadership (SC Chair and some Executive Secretaries) and major consultants. These crucial roles, often intersecting in complex ways, have made the Bank an indispensable partner. The Bank's analytical work and seed money leveraged other agency support for ADEA (DAE), and its presence continues to encourage partner involvement. Conversely, ADEA programs and positions have often reinforced (added value to) the Bank's programs at the country level in Sub- Saharan Africa. Such added value is not universal, however; it appears mostly in countries where a Minister and/or Bank team has been deeply and personally engaged with ADEA programs. 15. When backstopping of ADEA moved from Bank headquarters to its Secretariat in Paris, Bank involvement shifted to the oversight mode. Over the 17 years of oversight this function has been handled by five task team leaders, all recognized for their professionalism and vision. The present task team leader has also been selected by the SC as its Chair. This raises potential conflict of interest issues (not uncommon with global and regional programs), which need to be acknowledged and mitigated. Bank disengagement from ADEA is not being seriously discussed because of ADEA's unique position in fostering regional cooperation and progress in reaching Education for All (EFA) and the Millennium Development Goals. In addition, the Association is entering a crucial transition period (moving its Secretariat to Africa, adopting of the new strategic plan, and replacing a retiring Executive Secretary in 2008) during which the Bank's financial and professional support will be important to keep up momentum and to leverage continued partner agency support. However, exit strategies for some underperforming working groups are needed and should figure into the new strategic plan. Lessons Learned 16. The following lessons have emerged from the Bank's partnership with ADEA. Some of these are previously identified lessons which have not yet been heeded, some are in the process of being acted upon, some are modifications of previous lessons, and others are new. · Strategic planning for ADEA's future, which has focused mostly on desired processes and programs, also needs to specify organizational objectives in terms of desired outcomes (e.g., EFA outcomes, such as improved access or learning outcomes). This is a main point in the 2005 external evaluation which has not yet been heeded. In addition to desired outcomes, there is a need for measurable performance indicators. Even some of the so called "intangibles" that are prized in ADEA could be tracked in some way to show how ADEA is measuring up to its ideals. This calls for an agreement among ADEA partners as to the type of results-oriented monitoring and evaluation framework xvi that would be reasonable and credible, given ADEA's current and emerging objectives, resources, and organizational structure. · ADEA working groups need to be realigned with the Association's new strategic plan and subject to a new set of evaluation criteria which the Association needs to apply with firmness (a work already in progress). This would presumably lead to phasing out or graduation of some current working groups. The new pattern of forming ad hoc working groups could also be reviewed and modified as appropriate, given that its acknowledged benefits are accompanied by certain disadvantages, such as the groups not being anchored in Africa and their placing an administrative burden on the Secretariat. · Analytical work supported by ADEA is growing in volume but is of uneven quality. As ADEA moves into new realms of applied research it needs to set higher quality standards, and to concentrate as much on the quality (usefulness) of the research as on quantity (output). This recognizes the important capacity and ownership-building benefits of drawing new, local researchers into the process as co-investigators with more senior researchers. For research used as background documents in biennial and regional meetings and those published by ADEA, it would be desirable to establish a formal policy and procedures for document review. · Creative solutions to chronic overburdening of the ADEA Steering Committee must be found so that the Committee has time to exercise more substantive leadership, quality oversight, and identification of cutting edge issues. The formation of a smaller, constituency-based Executive Board would be a step in this direction. · Rapid turnover of Steering Committee members, particularly those from the Caucus of Ministers, is undermining the strength, confidence, continuity and institutional knowledge of their inputs into policy dialogue and decision making. Ways need to be considered to bring more permanent ministry officials into ADEA decision making circles and to find roles for emeritus committee members who are still prominent and influential in education circles. · ADEA programs and products are having a significant impact on those in the ADEA inner circle, but do not reach very far beyond this. Ways need to be found to extend the reach of ADEA programs and outputs, using country focal points or other innovative vehicles. 1 1. Program Objectives, Activities, and Costs Objectives 1.1 When ADEA was created in 1988 as one response to the Bank's landmark policy paper on the crisis in sub-Saharan African Education, it established the general goals of stimulating discussion of education policies in Sub-Saharan Africa and improving donor coordination in the sector. During the next 17 years the Association frequently articulated more specific goals and aims, but never established a formal set of objectives and related outcomes measures.1 This became an issue during the 2005 external evaluation, which recommended that ADEA develop "a strategic plan and business plan to operationalize ADEA's vision, set clear expectations, and monitor progress towards results." The ADEA Secretariat drafted such plans during 2006 and the ADEA Steering Committee formally adopted a new vision statement, mission statement, and five strategic objectives in October 2007 (Box 1).2 Since this occurred after the present review was conducted, this review has assessed the effectiveness of ADEA against its two original goals, with one modification. The review will substitute "improvement of educational policies" for "discussion of educational policies" since it is clear from ADEA's documents that policy strengthening is what the discussion (dialogue) has been all about. Thus, the present review has regarded (a) improving educational policies and (b) enhancing development agency coordination3 as the imputed strategic objectives of ADEA before October 2007. Activities 1.2 From the outset the Association aspired to influence educational policy and development agency coordination in the region by stimulating policy dialogue informed by research (analysis of existing problems and needs, and innovative solutions). Gradually it also became of aware of the need to build greater capacity for policy formulation and implementation and to promote enduring networks. In 2003 ADEA released a "visioning document," in which the following four challenges were set out as the work of the Association: (a) Promoting policy discussions (or dialogue) (b) Promoting analytical work (c) Strengthening capacities in African countries (d) Promoting regional and sub-regional networks. 1. In its 2003 "visioning document" ADEA described its reluctance to commit to a limited set of objectives as an asset ("institutional intelligence"), justified by the association's need to maintain an informality plus an openness to diversity and the changing needs of African education (ADEA, 2003). 2. In the March 2007 Steering Committee meeting, some suggestions were made for revisions. The strategic plan was revised and then adopted during the October 2007 Steering Committee Meeting. This undertaking was not without controversy. Many of those interviewed preferred to keep the ADEA charter informal and open, and worried that naming specific objectives and performance indicators would reduce its flexibility and responsiveness. The ADEA Executive Secretary responded to this by committing to build flexibility into the annual work plans. 3. The present review uses the term "development agency" instead of "donor" (the term used at ADEA's founding), since many of the agencies contributing to development in Africa (e.g., the World Bank and UN agencies) are not, technically speaking, donors. 2 Box 1. ADEA's New Vision Statement, Mission Statement, and Strategic Objectives The ADEA Steering Committee formally approved the following at its most recent meeting in October 2007. Vision statement: ADEA becomes a leader in the processes of dialogue, sharing, and learning for the qualitative transformation of education in support of Africa's development. Mission statement: To act as a catalyst for innovative policies and practices conducive to change in education through the pooling of thinking, experiences, lessons learned, and knowledge. Strategic objectives: 1. Foster the development of a shared understanding for coordinated and effective action by the main stakeholders to tackle the major challenges facing educational development in Africa. 2. Contribute to the achievement of continental and regional integration objectives in the education sector, as defined by the AU and its NEPAD program through the Plan of Action for the Second Decade of Education for Africa (2006­2015) and Africa's Science and Technology Consolidated Plan of Action. 3. Produce the knowledge and lessons learned from successful experiences in areas that are critical to the qualitative transformation of education in Africa in order to reinforce the policies and capacity for reform. 4. Disseminate as widely as possible the messages and findings accumulated by ADEA from policy dialogue, analytical research and exchanges of experience in order to support knowledge sharing, inter-learning, and capacity building. 5. Improve ADEA's performance in the areas of management, planning, evaluation, and accounting and financial reporting while at the same time strengthening its institutional capacity and organizational mechanisms 1.3 Promoting policy discussions. ADEA was built on the premise that educational policies can be improved through informed discussion among development partners. At the regional level this involves dialogue across countries, among government and non-government organizations, and between country representatives and development agencies. At the national and sub-national levels it involves the all-important discussions among actors and stakeholders (government, non-government and development agency). The purposes of such dialogue are to identify common problems, to formulate regional and national priorities, to demonstrate and compare innovative solutions, and to help build consensus at least at the national level, but often at the regional level and across agency types (e.g., development agencies and ministries of education). Such dialogue is expected to build partnerships across organizations at the regional and country level. It can be especially constructive for the (mostly) small countries in Africa, which cannot possibly try out a full range of solutions to serious educational problems, and for which learning from others makes sense. In ADEA policy dialogue takes place within thematically oriented working groups, during regional and sub-regional meetings (most importantly the biennial meeting); during Steering Group and Ministers' Meetings; and through newsletters and publications. Given that most countries in the region have prioritized basic education ("Education for All") it is at that level that most ADEA policy dialogue has taken place, but interest in post-basic education has been building in recent years. 3 1.4 Promoting analytical work. One of the founding principles of ADEA is that policy dialogue needs to be informed by rigorous analytical work, work that provides and understanding of the nature of problems and constraints, identifies solutions and their effectiveness, and compares strategies and outcomes. ADEA has come to promote a "praxis" approach to research which consists of cycles of action, grounded research, reflection, and improved action (including policy making). The Association promotes analytical work within its working groups, which are expected to act as laboratories in identifying and testing new solutions, and as preparation for thematic discussions in regional meetings at which research reviews and case studies are presented. ADEA makes the results of analytical work available in these forums and through its communications programs (archiving, publication and dissemination). 1.5 Strengthening capacities in African countries. This was not an explicit concern at the founding of ADEA, but by the mid-1990s partner governments began to note the serious constraints that poor institutional capacity were placing on policy reform efforts and policy implementation. Thus, at the request of the ministries, ADEA adopted capacity building as one if its core challenges, a position reinforced by a 2002 needs assessment in which ministries placed capacity building high on their list of needs. There is no specific ADEA program and budget for this, but in many or most of the working group and regional meetings there are explicit or implicit efforts to improve local capacity for both strategic programming and related technical skills (e.g., initiating and managing reform, creating management information systems, conducting research, and managing finances).4 Most of the technical capacity building that it has undertaken has been by or in connection with various working groups, with those on Finance and Education, Communications for Education and Development, and Education Statistics being particularly active. Improved capacity for strategic programming and reform management has often been facilitated by regional/sub-regional meetings and cross-national exchanges. In addition, ADEA's earnest efforts to shift coordination of working groups to African institutions are also a form of institutional capacity building, the best case of this being the Forum for African Women Educationists (FAWE). 1.6 Promoting regional and sub-regional networks. This activity or "challenge" has been implicit in ADEA since the beginning, especially in the concept of working groups and their eventual "graduation" to the status of self-standing regional organizations. During the mid-1990s a concern for anchoring ADEA activities in African education spurred an effort to link ADEA working groups to African institutions, and more recently there are ADEA efforts to partner with and contribute to other regional networks, such as those connected to African Union and various sub-regional economic communities and research associations. ADEA Vehicles 1.7 From the early years of the DAE to the present the main vehicles used by ADEA to advance its work have been: (a) loosely coordinated working groups; (b) Steering Group and Bureau of Minister meetings; (c) biennial meetings involving all agency, ministry and civil society partners; (d) special or sub-regional meetings; and (e) publication and information 4. Since ADEA is primarily a policy and knowledge network, it does not have the charter and/or capacity to undertake wide-spread capacity building, and if it did, it would be in danger of encroaching on other institutions that have this function. 4 dissemination (including databases). Each feeds into one or more of the activity areas above, and is organized as follows. 1.8 Working Groups. At the outset nine working groups were formed, covering topics considered by the agency partners to be highest priority in Africa, many directly connected to findings and recommendations in the Sub-Saharan Africa policy paper. (See Annex C for a chronology and description of ADEA working groups.) Initially, working groups were all sponsored by one or more of the development agency partners, and given an open agenda. With the push towards more anchoring of ADEA in Africa beginning in 1994, increasing numbers of working groups (starting with the one on Female Participation) forged connections to other institutions in Africa.5 By 2007 over half of the working groups were linked to African institutions. New working groups have been created along the way and a few closed or retired (the 2000 evaluation recommended that a large number be either dismantled or graduated but few have been) and that no new ones be created (one has been). Instead, a new kind of working group, now called an ad hoc working group, has been set up to deal with urgent cross-cutting themes over a limited time period. 1.9 Steering Committee and Bureau of Ministers Meetings. ADEA is governed by a Steering Committee consisting of partner development agencies plus a group (Bureau) of African ministers of education (see the ADEA Organizational Chart in Annex E). Steering Committee meetings are held twice a year to provide general guidance and decision making on ADEA directions, work plans, and budget allocation. The Bureau of Ministers generally meets twice a year adjacent to a Steering Committee meeting and uses this time to consider emerging issues, plan sub-regional activities, and launch regional programs such as inter- country quality nodes and peer reviews of national education programs. These events are intense, covering a large array of management, policy and budgetary issues, but are also expected to provide opportunities to critically reflect on emerging problems, strategies and promising new practices in African education. 1.10 Biennial Meetings. The major event in the ADEA calendar is the biennial meeting. This event convenes all donors to ADEA and their teams, most ministers of education in Sub- Saharan Africa, and a variety of stakeholders from academia and civil society groups. Its purpose is to stimulate open and frank policy dialogue among the partners, generally around a particular theme for which papers and case studies have been prepared in advance. (A list of biennial meetings, their dates, venues and themes, is provided in Annex D.) The first of these meetings was held in Manchester England with a focus on reports and papers from one of the working groups (that on Textbooks and Libraries). In early years such meetings were called "plenary meetings," to distinguish them from meetings having a more limited scope, like Steering Committee or working group meetings. The next two bienniales (as they began to be called) were held in France. After that, in keeping with the movement towards more African ownership of ADEA, bienniales have been held in African national capitals. Products from the bienniales are published proceedings, the case studies and research presented, an ADEA newsletter summarizing biennial reports and discussion highlights, and follow-up activities. 5. The Working Group on Female Participation became connected to the Forum for African Women Educationists (FAWE) and eventually "graduated" from ADEA (although it has maintained connection as an associate member). 5 1.11 Publications and Information Dissemination. ADEA produces, publishes, and promotes a wide range of articles, books, databases, papers, and reports that relate to the development of education in Africa. The complete catalogue of ADEA publications and documents contains more than 200 titles. Many of these are available online, and others can be ordered from the ADEA Secretariat and from the working groups. These publications include: · Quarterly ADEA Newsletters reaching 12,000 addresses and covering topics of the day (results from biennial or Steering Committee meetings, summary of WG group findings and recommendations, topics of special interest); · Occasional ADEA News Briefs, covering breaking news on African education and ADEA happenings · A new (2005) series of books covering country case studies in education; · Books and materials reporting research and proceedings from case studies and bienniales. 1.12 In addition ADEA maintains three databases relevant to the development of education in Africa, including (a) Profile: a database of ADEA working groups and programs in 48 countries of sub-Saharan Africa; (b) PRISME, a descriptive database of foreign aid to education in sub-Saharan Africa; and (c) CV Experts, an online curriculum vitae information system of African specialists in education. Finally, the ADEA communications section manages the annual Africa Education Journalism Award, created by ADEA in 2001 to recognize the best articles on education in Africa written by African journalists and published in the African press, and invites an increasing number of receptive journalists from Africa and other parts of the world to its gatherings. Governance 1.13 ADEA's Steering Committee composed of ten African ministers of education and representatives of most multilateral, bilateral and private development organizations that work in the education sector in Africa (currently 21). The Steering Committee is both ADEA's governing body and its primary facilitator of coordination among funding agencies, among African ministries of education, and between these two components. The ten ministers of education who sit on the Steering Committee constitute the ADEA Bureau of Ministers. They are chosen by the Caucus of Ministers which consists of all the ministers of education of sub-Saharan Africa. Bureau Ministers are selected by region (two each for Western Africa, Eastern Africa, Central Africa, Southern Africa and the Indian Ocean) on the basis of alphabetical rotation by country. They serve for a four-year period. Both the Bureau of Ministers, which meets twice a year, and the Caucus of Ministers, which meets during ADEA's biennial meetings, are forums for dialogue and consultation on the challenges and opportunities for the development of education in Africa. The Steering Committee meets twice a year to decide on budgets and work programs, including those proposed by Working Groups. Each Working Group in turn has its own steering committee. 1.14 Administrative support for ADEA is provided by the ADEA Secretariat, which since 1992 has been hosted physically and administratively by the International Institute for Educational Planning (IIEP) in Paris, a UNESCO affiliated organization.6 In 2008 the 6. According to the Revised Principles of Association of the "Association for the Development of Education in 6 Secretariat will move to Tunis, Tunisia, where it will be hosted by the African Development Bank.7 The ADEA Secretariat facilitates the coordination of the working groups, promotes information exchange and communication, organizes the biennial and Steering Committee meetings, develops and updates databases on education in Africa, and coordinates programs and activities such as the intra-African exchange program. The Secretariat consists of an Executive Secretary and 5­6 full time professionals and part-time consultants. When the Steering Committee is not in session, decisions about ADEA are made by the Bureau of ADEA (also referred to as the Bureau of the Steering Committee), which consists of the Steering Committee Chair, the Alternate Chair (also head of Bureau of Ministers), the Vice- Chair (ex-officio and head of the host agency), and the Executive Secretary of ADEA. Costs and Financing 1.15 ADEA is funded almost entirely through contributions from its 20­22 development agency partners (the number has fluctuated recently), and African Ministries of Education through in-kind and cash offerings.8 A minimum agency contribution of $50,000 has been required for joining and remaining in the partnership. These amounts are "unrestricted" or core funding and can be allocated by ADEA according to needs. Contributors also pledge "restricted" funds, which are generally earmarked for particular working groups. Restricted funding has exceeded unrestricted by as much as 3 to 1 in recent years. As can be seen in Figure 1 funding (especially restricted) fluctuates from year to year, which places constraints on long-term planning. Some countries (namely, Canada, the Netherlands, Finland and Sweden) have worked to moderate this by signing multiyear funding agreements, but this has not become the norm. Given such multi-year pledges, ADEA annual budgets generally show carry-overs from year to year. The estimated funding in calendar year 2006 -- about $6.6 million, mostly covering that year, but for some donors stretching to subsequent years -- is the highest level of funding ever (a 150 percent increase in funding over that for year 2000 in nominal terms), reflecting an expansion in ADEA programs and development agency support.9 Africa" (Paris: 1995), IIEP acts as host agency for the ADEA Secretariat, so that ADEA funds are administered as a "funds-in-trust project" within the UNESCO special account for IIEP, operating under the responsibility of the IIEP Director and Governing Board. The Director of the host agency serves as the ex-officio Vice Chair of ADEA's Steering Committee. As host agency, IIEP serves in a custodial role, monitoring the policies and programs defined by the Steering Committee. The ADEA Secretariat operates under the administrative regulations of the host agency, with the Secretariat staff to be hired a personnel of the host agency. (In practice, Secretariat members have been hired as short-term staff members.) Over the years IIEP has accorded ADEA virtual autonomy over its affairs and budget. 7. The feasibility study was conducted to consider various alternatives for the Secretariat's move to Africa (R. Sack, Kinyanjui, K., and Kinzounza, K., "Feasibility Study on the Transfer of the ADEA Secretariat to Africa," December, 2006). This commented on AfDB's strong commitment to education (based the number of projects 0and professional staff members in the education sector), but the IEG mission to the AfDB in Tunis was informed by AfDB staff that its education program was being downsized, given the AfDB's prioritization of infrastructure under its new leadership, and that there might be few or no projects in the pipeline for education in the near future. ADEA has benefited from its being hosted by an agency (IIEP) with a strong commitment and reputation for excellence and leadership in educational research and policy studies. ADEA may not receive the same kind of support from AfDB if the latter moves towards a relatively weak commitment to the education sector. 8. African ministries of education have been expected to contribute $1000 a year, but in 2007 the Bureau of Ministers recommended raising that to $2,500. 9. It can be seen from Figure 1 that the increase in funding for 2006 was almost completely in the form of unrestricted funds, which is a positive development, given the recurrent "structural deficit" that ADEA has faced, 7 Figure 1. Annual Contributions to ADEA (Restricted and Unrestricted) US$ Millions 7.0 Restricted 6.0 Unrestricted 5.0 4.0 3.0 2.0 1.0 0.0 2000 2001 2002 2003 2004 2005 2006 Audited Estimate Source: ADEA, 2006 Report of Activities. 1.16 The use of funds in ADEA (Figure 2) appears to be consistent with its charter. A majority (47­55 percent in recent years) of the funds are devoted to the operation of the working groups. Funding support for policy dialogue fluctuates between 7 and 21 percent depending upon whether it is a biennial meeting year. Funding for publications and communications have in recent years been around 5­6 percent of the total, except in 2004 -- the year following the report-rich bienniale on education quality -- when it rose to over 12 percent. During the past three years (the only years in which this has been broken out) the percent of budget allocated to the Secretariat has been 16­17 percent. 1.17 ADEA has generally been known for sound financial and budget management. For example, in 2004 it executed its initially approved budget plus a 9 percent overrun which was approved by the Steering Committee and covered by the mobilization of additional funds. However, the financial situation deteriorated unexpectedly in 2005, when IIEP presented ADEA claims for a significant payment arrears, based on its 2002­04 accounting.10 An independent audit found that the arrears was not as great as originally presented, but this still put a strain on ADEA programming, especially by the Secretariat, whose funds were especially curtailed, during 2005 and 2006. A large increase in development agency pledges of unrestricted funds in 2006 (especially from the World Bank's Development Grant Facility) seems to be improving the situation, but results will only be known for sure when audited financial accounts are available. in which there is less funding than needed for programs covered by the association's "general" (discretionary) account. ADEA's campaign to increase unrestricted funds and correct its structural imbalance is expected to be given a boost by the new strategic plan (providing the basis for longer-term commitments by agencies) and by the ministries' decision in 2007 to increase their membership fees from $1,000 per year to $2,500. 10. This was to cover an apparent debt incurred by ADEA over 2002­2004 but not perceived by IIEP because of UNESCO's transition to SAP and other delays in reconciling accounts for spending in Africa. 8 Figure 2. ADEA Use of Funds 2001­2006 US$ Millions 3.5 3.0 2001 2002 2003 2004 2005 2006 2.5 2.0 1.5 1.0 0.5 0.0 Ad hoc Publications Secretariat Working Coordination Overhead Policy Regional & Working & Communi- Groups & Facilitation & Reserve Dialogue Sub-regional Groups cations Cooperation Kind of Activity Source: ADEA, 2006 Report of Activities 1.18 Financial support from the World Bank has been significant over the entire nineteen- year period of ADEA's existence. In the first three years, when the support services resided at the Bank, its contributions were largely in kind (i.e., staff time). One analysis estimated the Bank funding during that period to be about $445,000 compared to the overall development agency support of $2.5 million, or about 18 percent of the total. After 1992, Bank support came largely through grants from its Special Grants Program (renamed the Development Grant Facility in 1998). By the early years of this decade (2000­2002), DFG funding amounted to about $300­350,000 per year, or about 12­14 percent of the total. In the middle years of the decade, financial support for ADEA from other agencies increased, but DGF support remained the same (reducing its share to less than 6 percent). In 2006, the DGF agreed to an increase in DGF funding (from $350,000 to $1.0 million), although $400,000 of this $650,000 increase was for activities previously funded though other channels (a $250,000 on-grant to FAWE and $150,000 for the Working Group on Higher Education). The additional increase was to help ADEA support new ad hoc working groups mainly linked to EFA, for example, on HIV/AIDS and education, and on the quality of education, the latter as a follow-up to a very successful biennial meeting on this topic and the preparation for a second one in 2006. The Bank's task team leader advocated these increases on the basis of the increased value added of ADEA programs to the Bank's educational sector objectives in the region. Depending on the estimates used for total commitments to ADEA in 2006, this represented an increase in the proportion contributed by the Bank to around 12­15 percent -- a return to the share provided in the early years of the current decade. 9 2. The 2005 Independent Evaluation of ADEA Evaluation Process 2.1 The 2005 independent evaluation of ADEA was undertaken by Universalia, a Canadian management consulting firm based in Montreal. This was the third evaluation of ADEA, the first being undertaken in 1994, when the Association was still known as the DAE, and the second in 2000, which focused explicitly on the ADEA working groups. The 2005 evaluation was undertaken under the authority of the ADEA Steering Committee at the request of the World Bank's Development Grant Facility, which requires programs receiving its financial support to be evaluated every 3­5 years. 2.2 The terms of reference for the evaluation were drafted by the ADEA Secretariat and reviewed by the Steering Committee. They were finally approved by the Bureau of ADEA (Steering Committee Chair and Alternate Chair, the Director of the IIEP, and the ADEA Executive Secretary). The contractor was selected from among candidate groups that were nominated by Steering Committee Members. The short-listed firms were sent Request for proposals (RFPs) which were reviewed by the Bureau of ADEA. Universalia was judged to have the strongest proposal. The evaluation cost about US$100,000, which was covered by exceptional funds provided by the World Bank (DGF) and CIDA. 2.3 According to the current Executive Secretary of ADEA, the Association had never had contact with this contractor before. Moreover, none of Universalia investigators had ever been involved with ADEA in the past. All conclusions in the evaluation report were arrived at independently; there was no manipulation of the findings or recommendations by ADEA stakeholders in any way. After the evaluation report was drafted it was reviewed by the ADEA Secretariat and Steering Committee over a two-month period, and revisions were made in response to corrective inputs. Monitoring and Evaluation Framework 2.4 One of the main findings of the Universalia evaluation was that ADEA did not have a well defined strategic plan and therefore had no clearly stated objectives or expected results and related strategies, and no clear framework for monitoring and evaluation (with performance indicators to track progress). The use of a logical framework has been somewhat antithetical to ADEA's work, which has been characterized as informal and responsive to changes in the African education scene. Consequently, much of ADEA's work and outputs have not been systematically evaluated for their value added. This made the external evaluation less results-oriented than it might have been. Universalia did acknowledge that some useful results had been produced, but even in its evaluation report, these were hard to find. For example, the main operational arm of ADEA is its working groups, but the evaluation's review of the working groups' contributions and products was only the sixth of six items in its section on program effectiveness. 2.5 Universalia's evaluation adopted a methodology known as the Institutional and Organizational Assessment (IOA) framework, developed jointly by itself and the International Development Research Centre of Canada (IDRC). The framework focused the evaluation on three main features of organizational performance, relevance, effectiveness, and efficiency (similar to IEG's approach). The evaluation also covered governance and 10 resource mobilization and management (in its section on efficiency), and touched upon exit strategy, especially for working groups. Sustainability and risk were not explicitly covered. 2.6 In undertaking its assessment of relevance, effectiveness, and efficiency it used a mix of quantitative and qualitative methods, including: Document review; Questionnaire (administered to a potential pool of 177 stakeholders -- 74 of whom responded); Interviews (45 face to face); and Field visits to Africa (including Benin, Burkina Faso, Niger, Senegal, Uganda, and Kenya). Main Findings and Recommendations of the Evaluation 2.7 The Universalia evaluation presented 23 findings, from the confirmation of ADEA's success in building "social capital" around educational change to the diverse perspectives on the ideal geographical location for ADEA's Secretariat. About a third of the findings dealt with the Association's effectiveness and another third with its efficiency. Some of the findings on efficiency covered funding and financial management, including the relative proportions of restricted and unrestricted funding, but the predominance of the former was not highlighted as much as it could have been. 2.8 This review notes that many of the evaluation's findings document stakeholder judgments on various questions, rather than the evaluator's own independent judgments, based on evidence collected. For example, one finding is that "ADEA is perceived to be effective in facilitating policy change" (italics added). This finding is largely derived from questionnaire results in which three-quarters of the stakeholders surveyed endorsed this position, many of whom have some stake in a positive ADEA image. A more nuanced assessment would have at least differentiated between the stakeholders having insider (producer) and outsider (consumer) status. It would have also been good to provide more evidence of how ADEA events or programs explicitly influenced a policy or program at the national level. Similarly stated findings are: "ADEA is regarded as an effective medium for building partnerships," and "ADEA's Secretariat is generally considered to be very effective at ... disseminating relevant information." 2.9 Also, given the importance ADEA attaches to the conduct of analytical work and its use in informing practice, it was surprising that the evaluation did not highlight or catalogue the reports/publications that had been produced, including the main messages and conclusions from the 2003 bienniale, which would soon appear as The Challenge of Learning (2005). In addition, nothing was written in the evaluation about the quality of the studies, reports, and reviews that have been sponsored by ADEA. 2.10 Universalia's findings led to the formulation of 8 recommendations, presented in Table 1, along with ADEA's responses. ADEA's Response 2.11 Interviewees asked about the external evaluation generally found it to be fair and constructive, some saying that it brought up important issues that had been just below the 11 Table 1. External Evaluation Recommendations and ADEA Response (before the Steering Committee meeting in October 2007) Recommendations ADEA Response 1. Based on the principles set out in the 2003 Vision 1. Some changes in organization are underway (see Document, the Steering Committee should clarify point 2 below), but this will not be fully undertaken what type of organization structure is most until the new strategic plan has been finalized. appropriate for ADEA's future development. 2. The Steering Committee should revisit its mode of 2. The Association has proposed that its finance sub- operation in order to address important matters committee fuse with the Steering Committee such as management oversights, decision-making, Bureau11 to become an Executive Board in charge ownership, and fiduciary responsibility. of making certain decisions for the Association. A consultant has been appointed to work with the Secretariat in working out the mandate and the boundaries of the new Executive Board. 3. In order to provide a greater sense of direction and 3. The Secretariat spent much of 2006 drafting a new focus, the Steering Committee should ensure the strategic plan which was reviewed and commented development of a strategic plan and business plan by the Steering Committee during its October 2006 to operationalize ADEA's vision, set clear and March 2007 meetings. There has been some expectations, and monitor progress towards results. controversy in ADEA over the creation of a formalized strategic plan, given the tradition of operating with informality, but the Executive Secretary has indicated his intension to build some flexibility and spontaneity into the plan. The plan will be finalized before the second SC meeting in 2007. 4. The Secretariat should be strengthened both in 4. ADEA is looking at ways to strengthen its terms of its analytical contributions and logistical Secretariat commensurate with its expanding oversight functions. Resources attributed to the mission. This will not be fully actualized until after Secretariat should be commensurate with the the new work plan is finalized and approved but at expectations of ADEA's constituency. least there is a decision to strengthen the financial and administrative management post in the Secretariat and to recruit a senior person to fill it in 2008. 5. The Steering Committee, with the support of IIEP, 5. Given the fact that ADEA will be disassociating itself should modify its human resource procedures in with IIEP during the next year, addressing this issue order to regularize the status of Secretariat staff. is untimely. It would be appropriate to address it with the new back-up agency. 6. Over the course of next year (2006), the Steering 6. Succession planning for successor to the current Committee should plan for the succession of the Executive Secretary who will retire in 2008 is going Executive Secretary. ahead. There is now a ten step plan, and, as of January 2007, five had been completed. The process should complete by summer '08. 7. The Steering Committee, in collaboration with the 7. A new communications plan developed by the Secretariat, should develop policy guidelines to Secretariat was briefly presented at the March `07 improve dissemination of ADEA's work. SC meeting but there was no discussion of it. 8. The Steering Committee should carry out a 8. The feasibility study concerning relocation was feasibility study for the relocation of the Secretariat presented at the March 2007 Steering Committee and a detailed cost-benefit analysis of the current meeting and a decision reached to relocate to Tunis arrangements with IIEP. in association with the African Development Bank. A cost-benefit analysis of arrangements with IIEP was not undertaken, given ADEA's impending separation from IIEP. 11. The Steering Committee Bureau consists of the SC Chair, Alternate Chair, Ex-Officio Vice-Chair, and the Executive Secretary of ADEA. 12 surface. The ADEA Steering Committee has taken its recommendations seriously and has initiated many changes in response to it, especially with respect to recommendation #3 on the drafting of a strategic plan; #6 on planning for a successor to the current executive secretary; #7 on the creation of a new communications plan; and #8 on the conduct of a feasibility study for relocating the Secretariat to Africa. 2.12 With respect to the other recommendations -- #1 on changing the organization structure, #2 on changing the Steering Committee's mode of operation, #4 on strengthening of the Secretariat, and #5 on changing the human resources procedures -- much preliminary planning and debate has gone into them, but these could not be completed until after the new strategic plan was finalized in October 2007 or, in the case of human resources, detailed arrangements with the new host organization (AfDB) are worked out. 3. The Effectiveness of the Program Relevance 3.1 Relevance of Objectives. The founding aims of ADEA of improving education policies and enhancing donor coordination, taken by this review as its strategic objectives, were derived from the 1987 Education Sector Policy Paper and thus highly relevant to the conditions portrayed in it. For example, the paper had demonstrated how inadequate policies in education were behind much of the poor performance and stagnation of the education system that were apparent in the late 1980s. Also, it revealed a crisis of underfunding in basic education, which had created a growing dependency on external funding in many countries. This had led to a chaotic mix of development agency requirements and preferences, often creating more problems than solutions. Bringing development agencies together in partnership with ministries of education to discuss needed policy reforms in education was a much needed step in addressing these problems. 3.2 However, over the years perceptions and circumstances have changed, creating a need for revisiting the continuing relevance of these objectives. For example, a growing awareness of the donor-driven origins of many educational initiatives in the early 1990s led to ADEA's determination to anchor the Association more in African institutions and in country-defined needs -- out of which grew a stronger emphasis on institutional capacity and regional network building, and less explicit emphasis on donor coordination.12 Other perceptions focused on the vagueness of the ADEA agenda, which had produced a highly divergent set of expectations for the Association, and widely varying perceptions on how effective the Association had been (see 2005 external evaluation) -- problems which the new strategic plan for 2008­2012 is presumably intended to address.13 Finally, new regional initiatives launched by the African Union -- e.g., the Decade for Education in Africa (2006­2015) and the New Economic 12. Donor coordination in Africa, as it other parts of the world, is increasingly being seen in relation to the creation and support of country or regional sector plans and not as a separate issue. 13. The 2005 evaluation's recommendations that a strategic plan be produced "outlining clear expectations" and the monitoring of "progress towards results," was one of the factors leading to ADEA's decision to draft a new strategic plan in 2006. But the new strategic objectives approved by the Steering Committee in October 2007 are still mostly about what ADEA expects to do (activities and outputs) rather than what it expects to accomplish (outcomes). 13 Partnership for Africa's Development (NEPAD) -- have set out ambitious regional goals for education that ADEA needs to respond or adjust to or face growing irrelevance.14 3.3 Relevance of design. In general ADEA was appropriately designed to address its original strategic objectives of improving policies and enhancing donor coordination. Especially relevant were its policy dialogue and analytic components which were to operate through its semi-autonomous working groups and regional/sub-regional meetings. However, as noted in the 2005 external evaluation, its design has not included a clear monitoring and evaluation framework for tracking how well these objectives have been met.15 Also, ADEA has failed to establish accountability systems vis-à-vis its working groups to assure their productivity and to document their outcomes and policy impacts. This is in part a consequence of its outdated financing design, which allows most donors to contribute far more earmarked funds than unrestricted ones.16 Finally, the management structures are becoming inadequate for the growing size and complexity of ADEA, a point brought up in the 2005 external evaluation and more fully elaborated in the Governance and Management section of this review. Efficacy 3.4 To assess the efficacy of the Association, this review examined outcomes on the two strategic objectives of (a) improving educational policies and (b) enhancing donor coordination. It also reviewed the outputs of the programs which contribute to the fulfillment of these objectives. This sub-section will examine the efficacy of both the objectives (outcomes) and the programs (outputs). OBJECTIVE: IMPROVING EDUCATIONAL POLICIES 3.5 As mentioned in the previous section, there is not enough systematic evidence to show a clear pattern of ADEA impacts on the objective of education policy improvement in Sub- Saharan Africa. However, testimonies of positive impacts abound. For example, one World 14. ADEA contributed to the development of AU's "Decade for Education in Africa" and has put it, along with NEPAD priorities, into the background section for its new strategic plan. One difference between the draft ADEA objectives and those under these AU initiatives, is that the latter are quite outcomes oriented (e.g., calling for use of African languages of instruction, systems to monitor and evaluate learning achievement, achieving UPE by 2015, among other goals), whereas those of ADEA are more process oriented. 15. Responding to the external evaluation recommendation calling for a framework for assessing the achievement of its objectives has created a dilemma for ADEA. Since ADEA is not a conventional investment project like its development agency partners finance at the country level, it does not have easily measured inputs, processes, outputs and outcomes. Indeed, as a regional forum pressing for policy dialogue and voluntary change at the national level, its outcomes can be quite evasive and effuse. This is not to say that tracking ADEA-stimulated policy changes in various countries would be impossible, but given the complexity of doing so (such as dealing with the issue of attribution) and given existing methodologies, this has so far been viewed as a less worthwhile use of its relatively modest regional program funding. Responding to this recommendation in a cost-effective way will require some creative thinking. 16. It cannot be assumed that if partner agencies were prevailed upon to reduce their restricted funding that they would shift the difference into unrestricted funding, since for some agencies one of the main reasons for supporting ADEA is to have the theme of their favorite working group on the agenda. The rebalancing of the two main forms of financing may need to come primarily from partners increasing their subscription fee $50,000 (an amount set over a decade ago) to $100,000 (an amount already contributed by some). Also, there appears to be a new commitment in ADEA (in the new strategic plan) to making the working groups more accountable to the SC, which will make it less possible for them to get funding irrespective of their productivity. 14 Box 2. ADEA and African Policies on the Language of Instruction For decades primary school children in African countries have been required to study in colonial language which most of them do not use at home or even understand. Propelled by the growing awareness of how little children learn under such circumstances, the Ad Hoc ADEA Working Group on Quality of Education introduced the themes of mother tongue and bilingual education into its agenda for the 2003 ADEA Biennial Meeting, "Improving the Quality of Basic Education in Sub-Saharan Africa." Participants in the meeting concluded that "African languages were a necessary choice for the new century," and proposed further "stock-taking" research on the subject and more intensive discussions. As a response, ADEA commissioned a set of state-of-the-art reviews and case studies in 2005, aimed at presenting evidence-based recommendations for language-in-education policies for use by policy makers and other stakeholders. The result was an ADEA volume, Optimizing Learning and Education in Africa -- the Language Factor: stock-taking review on mother-tongue and bilingual education in Sub-Saharan Africa, which contained critical reviews of the existing literature by six experts and new case studies from 13 African countries on effectiveness (in relation to learning outcomes) of bilingual education programs (mother tongue plus a metropolitan language), efficiency, program design elements, and reasons for successes and failures. This volume became the focal point of a three day thematic conference co-sponsored by ADEA (together with UNESCO and GTZ) in Windhoek, Namibia in August 2005, attended by over 60 international experts and African Ministries of Education from about 20 countries. Once revised in response to discussions in that conference the report was presented during the 2006 bienniale in Gabon, where it had a prominent place on the agenda. This visibility and evidence-based advocacy is making an impact on the African education community, at least at the rhetorical level. For example, the action plan adopted by the Conference of Ministers of Education of the African Union under its 2006­2015 new Decade for Education in Africa includes an item directly or indirectly influenced by these efforts, namely: "strengthening of curricula and teaching materials, with special emphasis on mathematics and technology, the use of African languages in bilingual education, and the gearing of curricula to local contexts." Nevertheless, evidence of policy change at the national level is hard to find. Only one instance of country level follow-up is mentioned in the 2006 ADEA Report of Activities: expert advice to Burkina Faso in formulating a national language policy. Some interviewees for this report indicated that enthusiasm for bilingual education may be cooling down these days, perhaps as a result of some well publicized negative experiences with bilingual education and/or the stock-taking report's strong advocacy of an excellent but demanding "late exit" bilingual education model (requiring 6­7 years of instruction in the mother tongue), which some governments may find too demanding. 3.6 Bank report indicated that as a result of participating in ADEA many countries had improved their record in undertaking evidence-based policy adjustment, had recognized the need for national political commitment as a pre-requisite for reaching EFA goals, had increased their commitment to girls participation in schooling and girl-friendly schooling, had recognized the need for a new "culture of quality" to enhance basic education outcomes, and had grasped the need for closer government cooperation with NGOs, teachers' unions, and parents' associations in reforming educational policy and practice.17 Written for a management review, these statements were not supported by a strong body of evidence. Some impacts were reported to occur with a time-lag, such as the changes in rural education since 2000 following from visits by African Ministers of Education to Guatemala and El Salvador in 1999.18 17. See World Bank, 2005, Partnership Approval and Tracking System, Global Programs and Partnerships, "Association for the Development of Education in Africa." 18. Communication from the ADEA Secretariat in December 2007. 15 3.7 The 2005 external evaluation cited anecdotal evidence in its review of ADEA's impact on improved policies. The prime example was the widespread view of how the WG on Female Participation (now FAWE) influenced the commitment to girls' basic education across the continent. Other fields where ADEA programs were said to have influenced policies at the national level are public finance and education, teacher policies (e.g., use of contract teachers), improved statistical systems for education, national policies on early childhood education, and policies on language of instruction and bilingual education (Box 2). 3.8 All the Ministers interviewed for this review could point to policy changes in their countries that were stimulated by ADEA discussions, but also suggested that impact was greatest where a minister or official had direct involvement in an ADEA policy dialogue (e.g., as a contributor to a biennial event). ADEA appears to have had less success in influencing ministers and ministerial agendas among those not directly involved in ADEA events. Reasons for this are not fully clear, but they appear in part to be a consequence of inadequate dissemination of ADEA information and products (especially within countries), and widespread failure to establish recommended ADEA focal points in ministries and national coordinators for working groups.19 3.9 For this review nine World Bank education Task Team Leaders (TTLs) covering all sub-regions of Sub-Saharan African (about half having direct experience with ADEA) were interviewed (Annex H). Of the 9 TTLs, only two indicated that ADEA objectives and activities were relevant to their country's current development priorities (Table 2). Those two also indicated that they were highly familiar with ADEA, buttressing the point that perceptions of ADEA's impacts and relevance are, at least in part, a function of direct experience with the Association.20 3.10 Looking across country programs more generally, there are signs of a disconnect between the rhetoric and lofty intentions of ADEA gatherings on the one hand and the realities on the ground on the other (Boxes 2 and 3). If ADEA had been tracking policy impacts more systematically, it could presumably have become more aware of such disconnects and worked to overcome them. 19. In addition to disseminating ADEA information and products at the national and sub-national levels, a country focal point or coordinator could also track the extent to which ADEA agenda items and products (including those from working groups) are being used in the country and what their impact has been, thus becoming part of a relatively low-cost monitoring framework. 20. Given the alleged high value added of ADEA to the World Bank's Education Sector Work in Africa (see World Bank, 2005), it is curious that more TTLs are not well versed in ADEA's purposes and programs. This may be in part a consequence of the small amount of the Bank's administrative budget allocated for ADEA activities ($50,000 per annum), which has gone mostly to support the Working Group on Higher Education (personal communication from Birger Fredriksen). The Education Sector in Africa might consider giving higher priority to briefing its African task team leaders about ADEA and involving them in its activities in some way. 16 Table 2. World Bank Task Team Leader Views of ADEA: Frequencies (n=9) A A A A great Don't substantial modest negligible deal know Question amount amount amount 1. Familiar with ADEA /1 4 3 2 2. Used ADEA products or services in 3 2 4 country operational work 3. Country counterparts show interest in 2 4 3 ADEA products and services 4. Objectives and activities relevant to country's current development 2 3 3 1 priorities Don't Very Moderately A little Not at all know 5. Aspects of ADEA considered most useful: a. Facilitating communications 4 2 1 2 among practitioners b. Generating and disseminating 2 2 2 1 2 information & knowledge c. Improving donor coordination 1 2 4 2 d. Advocating approaches to 2 2 1 4 development in the sector e. Supporting national level policy, 2 1 2 4 institutional & tech reforms f. Strengthening capacity and 1 2 4 2 training /1 For this item the response categories were: Very, somewhat, heard of it, never heard of it. OBJECTIVE: ENHANCING DEVELOPMENT AGENCY COORDINATION 3.11 For this objective, there is no systematic information on outcomes. The overarching assumption appears to be that improved interagency coordination will result from all major agencies being ADEA partners, that are sitting together on the Steering Committee, interacting during biennial and other regional/sub-regional meetings, reaching a consensus on some reform agendas, and co-sponsoring working groups and ADEA events. ADEA's promotion of government-owned, sector-wide, education development plans, supported by development agencies in a coordinated manner, can be assumed to have positively influenced agency coordination. In addition, donors have often joined hands in co-sponsoring various events and gatherings with ADEA (see section on Regional and Sub-Regional Networking). And a database of development agency education projects in Africa has been maintained by ADEA (PRISME) which has been used as a tool for enhanced coordination.21 21. PRISME is currently downloadable to remote sites, but not yet available on line. The entries were updated in 2005 and include projects with start-up dates as recent as 2004. Information includes only project title, planned start-up and completion dates, and planned amounts. There are no project descriptions or actual disbursement data. 17 Box 3. "The Culture of Quality" and National Education Plans: A Disconnect? The 2003 ADEA Biennial Meeting, entitled "The Challenge of Learning: Improving the Quality of Basic Education in Sub-Saharan Africa," ended with a pledge to create a new "culture of quality" for education on the continent. Enthusiasm for this agenda was such that pro-quality policies and programs emerged as the top priority in ADEA and among its partners. This is demonstrated by the fact that quality improvement returned again as the main theme of the 2006 bienniale. This has meant increased attention to how and how well students are learning, since as an ADEA newsletter article states, "Systems that adopt a culture of quality as the basis for their operational practice, focus on learning as the central goal of their education system..." (Jan­March, 2004, p. 8). Given this one might expect to see learning outcomes as a major concern in government plans and programs. As a test of this, a new set of government poverty reduction programs were examined to see to what extent they included a focus on education, and in particular on improved learning outcomes. The set consisted of new Poverty Reduction Support Credits (PRSCs), produced by African governments subsequent to the 2003 bienniale and funded by the World Bank. Of the 11 PRSCs in Africa approved in the Bank's fiscal years 2004 and 2005, only 3 mention poor learning outcomes as a problem, and only two had improving learning outcomes as an objective (both from the same country). To be sure, seven revealed intentions to cover some aspect of quality improvement (teacher training, books etc.), but without a focus on learning outcomes, it is unclear whether such inputs will lead to "the central goal" of the education systems. 3.12 Thus, there is fragmentary evidence that agency coordination has improved, but neither ADEA nor the 2005 external evaluation provided any systematic evidence to document it. When TTLs were asked if they noted an ADEA influence on donor coordination at the country level (Table 2), only one gave a positive response while six indicated the influence to be little or nil, leaving some doubt as to whether coordination often extends beyond ADEA gatherings.22 If ADEA takes this objective seriously, explicit efforts are needed to track improved coordination both at the regional level (reaching consensus on how to deal with the region's most pressing educational problems) and at the country level (working together on country-led efforts in a harmonized fashion). PROGRAM: PROMOTING POLICY DIALOGUE 3.13 Based on key informant interviews and other evidence, it is clear that ADEA has become the premier forum in Africa for educational policy dialogue. Evidence for this is its drawing power during key policy dialogue events, especially its biennial meetings. The last biennial meeting, held in Libreville, Gabon in March 2006 was attended by over 500 participants (the largest number ever), including 60 education ministers and deputy ministers. In such meetings ministers are brought into dialogue with representatives (some high level) of all major development agencies and NGOs active in education on the continent. Most of the 21 agencies contribute regularly to ADEA's budget, which includes policy dialogue as its second largest line item (after working groups). Increasingly in recent years policy dialogue has also taken place during the growing number of thematic and sub-regional meetings (see Appendix F for a listing.) 22. In fairness, ADEA never formally articulated an aim to promote development agency coordination at the country level, although this could be expected as a by-product of the agencies' meeting with African Ministries of Education every six months. Another avenue for country-level agency coordination would be through ADEA's growing cooperation with FTI, which has agency coordination at the country level as an explicit goal. 18 3.14 Many informants, including the 2005 external evaluator of ADEA, have commented on the unique features of the policy space that the Association has carved out, its openness and informality, the high level of trust that delegates feel, the lack of political or ideological constraints, and the frequent use of research and experience-based reflection to back up policy positions. Such intangibles (sometimes referred to as ADEA's "social capital") are both difficult to measure and to replicate in other settings.23 A final confirmatory point is ADEA's "convening power." Individual actors (agencies or networks or countries) may have difficulty calling attention to a particular issue without creating some suspicion as to their motives, but according to the ministers of education interviewed for this review, if an issue makes it onto the ADEA agenda, it is taken seriously by the education community in Africa. 3.15 ADEA's important role in policy dialogue was also noted by World Bank TTLs (Table 2). Seven of the nine TTLs responded to a question about what aspects of ADEA they found most useful. Four of the seven indicated that "facilitating communications among practitioners in the sector" (i.e., policy dialogue) was very useful (more than for any other category). These were the same four that indicated that they were "very familiar" with ADEA. In other words, those having a less positive view of ADEA's usefulness in facilitating communications were those less familiar with the Association. 3.16 As mentioned above, the main ADEA vehicle for policy dialogue on specific topics has been its working groups (WGs). It is difficult to generalize about the efficacy of the WGs, since they vary greatly in their activity levels and impact on policy discourse. The year 2000 external evaluation, which focused on WGs, concluded that 5 of the 11 then active WGs were having high or medium/high impact, one a medium impact, and four low or no impact (the most recent one was too new to rate). The more recently created ad hoc WGs have a stronger record, since their work feeds directly into the bienniales and has a limited scope and time line.24 The 2000 evaluation concluded with 14 recommendations on how the output of the regular WGs could be improved, including closing some, graduating others, and enhancing WG team building, funding flexibility and reliability, support to coordinators, and the mobilization of country-level coordinators and focal points. According to the 2005 external evaluation, only two of these recommendations were fully implemented, five were partially, and 6 were not implemented. In fact, ADEA has always had difficulty making hard decisions about its WGs.25 Results of the latest ADEA Steering Group Meeting (March 2007) 23. Even so, it is conceivable that some of the intangibles could be tracked in some way, giving stakeholders a clearer indication of ADEA stature in the region, and how it might wax and wane. 24. As can be seen in Annex C, ad hoc working groups were created in the fields of HIV/AIDS and Education, Quality Education, Policy Dialogue, and Post-Primary Education, over the period 2001-2004 (one per year). This was done at the initiative of the ministries of education. The first three have been coordinated by the ADEA secretariat and the fourth by the World Bank. These groups are expected to deal with cross-cutting issues and have a limited time span. For example, the one on quality education produced studies and preparations for the two bienniales on education quality (2003 and 2006) and is now slated for dissolution, and another is focused on preparing for the Biennial on post-primary education in 2008. It is not clear whether this kind of group will be the format of the future, since ad hoc working groups do not have the potential for creating permanent networks or institutions (like FAWE). There may continue to be a mix of standing and ad hoc groups, but this will need to be addressed as WGs are restructured, now that the 2008­2012 strategic plan has been adopted. 25. Among the reasons for this are that the ADEA Steering Committee makes decisions by consensus and that most of the funding for WGs comes through specific earmarks (restricted funds) from sponsoring development agencies, which may be determined to keep their preferred programs on life support. 19 may change that. WGs will now be asked to align their activities with the new strategic plan and their continued existence will be reviewed against a new set of criteria which are being drawn up and applied before the next SG meeting in the 4th quarter of 2007. This could also be an opportunity for creating a more systematic program of progress monitoring and self- evaluation among the working groups. PROGRAM: PROMOTING ANALYTICAL WORK 3.17 ADEA sponsors analytical work mainly as background material for biennial meetings and within working groups. Over the years ADEA has come to support a participatory and applied variety of analytical work undertaken at the national or sub-national level, but embedded in a regional and comparative framework, in order to identify similarities and differences across countries and to create the basis of reflection and innovation (a kind of analytical work often referred to as "praxis"). 3.18 The most productive time for such research has been just before biennial meetings, and their research-based policy dialogue sessions. It is beyond the scope of this review to review ADEA's productivity of such analytical work from the beginning. Instructive would be to consider the analytic work, quantity, quality and impact during the past two bienniales, those held in 2003 and 2006. The bienniale in 2003, "The Quest for Quality: Learning from the African Experience," brought forth a record amount analytical work, including 22 case studies, 3 reviews of development agency experiences, 15 background papers, and 4 thematic syntheses, all of which were included in one form or another in the large conference volume (the main basis for policy discussions), The Challenge of Learning: Improving the Quality of Education in Sub-Saharan African (edited by Adriaan Verspoor). 3.19 The 2006 bienniale, which was also mainly focused on effective learning ("More and Better Education"), saw even higher numbers of studies presented (near 60), including 17 on the theme of improved literacy, 18 on the conditions and factors of effective schools in Africa, and 19 on the scaling up of early childhood education.26 Among the papers presented at this bienniale were case studies and synthesis reports on the characteristics of effective schools in various African countries. In many cases, relatively inexperienced local researchers were partnered with experienced researchers, giving the undertaking of a capacity building sub-agenda. 3.20 One of the prominent researchers interviewed for this review commented on the uneven quality of the studies undertaken as inputs into biennial policy dialogues, warning that some studies were "not worth reading." Prompted by this, this review took a near 20 percent random sample of studies presented at the 2006 bienniale and subjected them to a quality review, using a rating scale adapted from an instrument used by the World Bank in a "quality assurance" review.27 Figure 3 shows the overall rating outcomes for the 10 studies selected (listed by number). The results of this analysis do reveal an unevenness in the 26. The total number of studies "produced" for the bienniale was said to be over 80, but around 60 were listed for presentation in various sessions. 27. The dimensions covered in the instrument were: (a) task objective (clearly stated? value added nature of the problem); (b) quality of the analysis (methods appropriate to objectives? quality of evidence presented?); (c) quality of conclusions (appropriately specific and consistent with analysis/evidence?); and (d) quality of recommendations (clearly stated? flow well from conclusions? implementable and feasible?). On the 1­4 rating scale, 1 was high and 4 was low. 20 Figure 3. Sample of ADEA Analytical Work Average Rating 3.5 3 2.5 2 1.5 1 0.5 0 #22 #47 #24 #15 #35 #43 #7 #36 #50 #10 Report Number quality of the papers, with a handful of papers (3 of 10) receiving high rating on all or most dimensions, a few (3 of 10) getting mostly weak ratings, and the rest (the majority) somewhere in between. In many of the papers a mention was made of the modest resources available for the studies and brief time available. For the next bienniale, ADEA might want to consider putting more funds into the preparation of fewer papers, and the establishment of a peer review process. 3.21 Concerning analytical work by ADEA working groups, the record is unclear, given the lack of comprehensiveness in ADEA's data base. All of the working groups have sponsored a considerable amount of work, but only a small fraction of it has been published and archived by the Secretariat. Nevertheless, an examination of ADEA's publications archives does provide a rough estimate of productivity across working groups (or at least productivity of materials considered by the Secretariat and/or the WG to be worthy of publishing). Table 3 shows the number of study titles, by both WG and time period.28 Of the 89 studies listed, nearly two-thirds were completed in the last six years (since 2000), indicating either higher research productivity in recent years or recent improvements in ADEA effectiveness in archiving products or both. 3.22 Clearly the most productive WG in terms of documenting analytical work has been FAWE (now graduated). Its studies comprised almost 1/3 of the total, and were both early (before 2000) and more recent. The Finance and Education WG, often also mentioned as a candidate for graduation given its strong ties to the Council for the Development of Social 28. This list does not include the analytical work produced for the 2006 bienniale, which included a large number of new studies on nonformal and early childhood education. 21 Table 3. ADEA Published Studies, by Working Group and Time Period (in descending order by Working Groups) ADEA Working Group Before Year 2000 Year 2001­2006 Total Female Participation (FAWE) 14 14 28 Finance and Education 2 9 11 Books and Learning Materials 3 7 10 Education Sector Analysis 4 6 10 Nonformal Education 2 7 9 Higher Education 5 2 7 Early childhood education 0 6 6 Distance Ed and Open Learning 0 3 3 Teaching Profession 3 0 3 Education Statistics 0 2 2 Total 33 56 89 Source: ADEA Publication Archive Science Research in Africa registered the next highest number of titles (12 percent), almost all in recent years. Three other WGs -- Books and Learning Materials, Education Sector Analysis, and Nonformal Education -- had around 10 titles (or 10 percent of the total) each. Three WGs showed very little published research productivity, namely those on Distance Education and Open Learning, the Teaching Profession, and Educational Statistics.29 Higher Education (showing only 7 study titles) was lower than might be expected, given the fact that universities thrive on research and publications. The general overview is that about two- thirds of the publications archived by ADEA stem from the output of four working groups. The productivity of many of the others is low enough to call into question whether they are making a contribution to African education development. PROGRAM: STRENGTHENING CAPACITIES IN AFRICAN COUNTRIES 3.23 Capacity strengthening was not an original ADEA objective, but various initiatives, starting in the mid 1990s put it on the Association's agenda. The visioning paper in 2003 called capacity strengthening one of ADEA's continuing challenges. This was reinforced by the results of a 2003­04 needs assessment conducted by ADEA among Ministers of Education, in which ministers showed particularly strong demand for capacity-building activities. Unfortunately, there has been no clear statement of what capacity building or strengthening means in this context. This has led to varying expectations of what ADEA can do and equally varying assessments of its effectiveness, as mentioned in the 2005 external evaluation. The present review will briefly consider three kinds of capacity building mentioned in ADEA documents: (a) technical capacity of ministry officers, (b) institutional capacity of African educational agencies, and (c) country capacity to enter and influence the policy dialogue in education. 3.24 ADEA has been active in providing technical capacity building in the region, especially in fields covered by its most active working groups (e.g., those on Finance and 29. The WG on Education Statistics was never oriented towards research productivity but more towards model and capacity building, and thus, strictly speaking, should not be compared with the others. 22 Education; Communications for Education and Development; and Education Statistics). Training and technical assistance have helped many ministries to strengthen relationships with ministries of finance (increasing the priority that the latter give to education); to build national plans and/or offices for communications about education; and improve the capacity of education system/managers to produce and effectively use more timely and reliable school data. One of the ministers interviewed for this review was particularly appreciative of what her staff gained from an ADEA exchange with a neighboring country and from sub-regional workshops; another cited ADEA as having helped his country create a national policy on early childhood education. Nevertheless, the reach of ADEA's technical capacity building has been limited, mainly available just to those directly involved in ADEA WGs or attending regional/sub-regional meetings (see 2005 external evaluation). This is likely what led all but one of Bank TTLs to conclude that capacity building by ADEA was of little or no benefit in the countries where they work (Table 2). 3.25 Institutional capacity building became an important theme in ADEA during the mid to late 1990s with its emphasis on anchoring and institutionalization of WG activities in African institutions, and this has been gaining ground. By 2006 seven of eleven active working groups were coordinated from institutions based in Africa.30 This has in most cases led to institutional strengthening and new continental focal points for relevant research and policy dialogue. In addition, as a follow-up to the 2003 bienniale in Mauritius the Ad Hoc Working Group on Quality of Education set up inter-country quality "nodes" to encourage deepening of research and exchange of experiences on various themes. Some of these started to take off in 2004­05 but activities were put on hold in 2006, given the need for financial austerity, and it is not clear how much new support will be forthcoming, given the shift to the activities defined by the new strategic plan. 3.26 The effort to anchor ADEA in Africa has also meant relying more heavily on the Caucus of Ministers to set its agenda. This meant a stronger representation of African Ministers on the ADEA Steering Committee and proactive efforts (e.g., peer review activities at Ministries) to assure that country needs are driving ADEA programming. In addition, an increasing amount of ADEA analytical work is being done by African researchers, and where research skills are not fully developed capacity building efforts are made. One limitation in such capacity building is the high turn-over among ministers. Typically, by the time a minister becomes accustomed to ADEA decision-making processes and has developed "a voice," the minister is replaced and the induction process has to start again. (For more about this issue see the section on Governance and Management.) PROGRAM: PROMOTING REGIONAL AND SUB-REGIONAL NETWORKING 3.27 ADEA is considered by its current and past executive secretaries to be a "network of networks." Mission interviews revealed this having come about in two ways: (a) through the creation of working groups which have grown into networks, (b) through partnering with existing networks, and (c) through sponsoring or co-sponsoring regional or sub-regional exchanges. The best example of network creation is the working group on Female Participation, which merged with FAWE, and has become one of the continent's most 30. This includes the working groups on Finance and Education, Communication for Education and Development, Higher Education, Distance Education and Open Learning, Mathematics and Science Education, Education Statistics, and Books and Learning Materials. 23 influential networks (and continues to be an ADEA associate). Two other working groups, those on Finance and Education and Educational Statistics, both anchored in Africa and mentioned for "graduation," have created acknowledged networks of specialist and animators in educational finance and statistics. (There does appear to be a positive relationship between a WG's being anchored in an African institution and becoming a recognized and enduring regional network.) The WGs on Higher Education and Communications for Education and Development, both now anchored in Africa, also have the capacity to spawn regional networks. If the "quality nodes" grow and mature over the next few years, they also have the capacity to do so in their specialty areas. One irony connected with the highly acclaimed ad hoc working groups is the following: Given that they are set up for specific tasks with a limited life span, and are mostly coordinated by the ADEA Secretariat, they may not have the right make-up for spawning sustainable networks. In sum, considering the fact that ADEA has only spawned one self-reliant network so far (FAWE), its success in networking is not as strong as it could be. 3.28 ADEA's second approach to networking networks has been its partnering with existing networks. The clearest example of this is in the growing ties (even a possible merger) between the ADEA Caucus of Ministers and COMEDAF, which is promoting the second Decade of Education for Africa (2006­15). Other regional and global networks to which ADEA has contributed include the New Partnership for Africa's Development (NEPAD), the various regional/sub-regional economic communities (the Economic Community Of West African States (ECOWAS), the Economic Community of Central African States (ECCAS), and the Southern African Development Community (SADC)), the Education Research Network in Eastern and Southern Africa (ERNESA), the Education Research Network for West and Central Africa (ERNWCA), and the EFA Fast Track Initiative (FTI). 3.29 The third approach has been networking in the form of exchanges among educators in the region. As mentioned under "promoting policy dialogue," this takes place every time ADEA sponsors or co-sponsors a regional or sub-regional meeting. The simplest form of exchanges are professional visits between neighboring countries, which ADEA has financed over the years -- they are a line item in the ADEA budget -- but given recent budget tightening, could not take place in 2005 and 2006. Closely related are "peer reviews" (ADEA- sponsored professionals critically reviewing country educational plans, coordinated by the WG on Education Sector Analysis), which have recently piloted reviews in Mauritius, Gabon and Nigeria. More significant are the growing number of sponsored and co-sponsored thematic and sub-regional meetings where professional networking takes place (mentioned in the section on policy dialogue and listed in Annex F.) CONCLUSION 3.30 For the strategic objective of improving educational policies, there is some anecdotal evidence that ADEA programs have had an impact at the country level, especially on girls' education and other areas where working groups have been especially active. However, since ADEA has not set up systems to evaluate the attainment of such outcomes, it is not possible to make a full judgment about the achievement of this objective. Similarly, for development agency coordination, there are indications of improvement but no clear evidence. Reinforcing a recommendation from the 2005 external evaluation, this review indicates the need for better strategic planning, including the specification of objectives, and the tracking of outcome indicators. 24 Efficiency 3.31 The 2005 external evaluation covered many aspects of ADEA efficiency. It did not undertake a cost-benefit analysis, given the difficulties in placing monetary values on ADEA-promoted outcomes, nor was it able to compute cost-effectiveness ratios, given the fact that outcomes have not been specified or measured. It did cover a set of issues that are still relevant today, namely, the efficiency of: (a) the modes of ADEA financing; (b) the operation of the working groups; (c) the management and oversight by the Steering Committee, and (d) management support by the Secretariat. (The latter two will be covered in the section on Governance and Management below). 3.32 As mentioned in section 1 above, agency funding for ADEA comes in two forms: unrestricted and restricted (which is mainly earmarked for the various working groups). Although ADEA prefers to receive unrestricted funds, a preference supported by recent evaluations, pledges of restricted funds have in recent years always exceeded those of unrestricted funds (by as much as 3½ times), and the gap between the two modes of funding has been widening (Figure 1). This has negatively affected ADEA's efficiency, since it restricts that capacity of the Association to allocate funds on the basis of performance. (Even though the absorptive capacity of many working groups is low, they still receive independent funding.) ADEA has continuously pleaded with agencies to increase their basic contribution (see footnote 16), and some agencies are now beginning to respond to the call.31 (There was an improvement in the proportion of unrestricted funding in 2006, but it is not clear if this will be sustained). Another sign of improvement was the 2007 resolution by the Bureau of Ministers to increase the membership fees for African ministries from $1,000 to $2,500 per annum. 3.33 Working groups receive about half of the ADEA budget each year, but it is not clear how accountable the WGs are for the funds they receive, an issue raised in the 2005 external evaluation. Whereas their annual plans and budget requests are reviewed by the Steering Committee each year, their performance, in terms of how well they have met their goals, has not been carefully monitored. Recently there has been a move to require annual WG reporting on performance indicators, but this has not received prominent attention in the latest (2006) ADEA annual activities reports. Governance and Management 3.34 At its creation in 1988 ADEA (then DAE) was an informal association of development agencies, coordinated by the World Bank, having the twin goals of discussing and improving educational policies and donor coordination. Initially, legitimacy was based on urgency and relevance of the policy paper messages that had stimulated DAE's formation, the World Bank's strong presence in Africa, and the limited participation of a few African governments. However, it was only after the mid-1990s, when the Association's name was changed to ADEA and the governance structure was formally revised to include a larger block of African Ministries of Education, that its legitimacy as a regional association became firmly established, both among agencies and ministries of education. Legitimacy was further strengthened in the 31. Encouragingly, there was some improvement in the proportion of funding being unrestricted in 2006, and three agencies announced their intentions to shift their support entirely to unrestricted funding in 2007 and 2008. 25 1990s by a conscious effort to anchor ADEA in Africa through holding all Biennial and most Steering Committee meetings in Africa and linking as many working groups as possible to African institutions. Finally, the 2005 external evaluation confirmed the reality that ADEA had become the forum for regional policy dialogue and education innovation in Africa. 3.35 ADEA's governance model has a mix of the shareholder and stakeholder elements. The Steering Committee comprises both shareholders (the development agencies), who are required to pay an annual subscription fee ($50,000 a year), and stakeholders (the African Ministries of Education) who are represented in their entirety though the Caucus of Ministers and the 10 Ministries that it selects to sit on the SC on a rotating basis.32 The representation of shareholders on the SC has enhanced legitimacy since virtually all of the major development agencies and foundations active in African education have been subscribers over the past ten years.33 The representation of stakeholders has also enhanced the legitimacy of ADEA, but has however raised some issues of efficiency, as discussed in the next paragraph. 3.36 The ADEA Steering Committee, its prime governing body, is a large body consisting of 21 contributing agency members and the ten member Bureau of Ministers, all providing their services on a voluntary basis. Given the growing reach and complexity of ADEA, this body has an enormous volume of work to do, all packed into three or so days of meetings every six months. The 2005 external evaluation noted that it was not possible for the SC to cover all items on the agenda with the level of detail that is required, "forcing members to focus on procedural obligations such as the review and approval of budgets and workplans at the expense of more in-depth discussions of the actual effectiveness and relevance of activities undertaken by ADEA" (page 26). In addition, given the Steering Committee's consensus approach to decision making, it is often not able to make difficult decisions quickly, if at all. Many important decisions, such as what to do with underperforming working groups, or how to streamline the SC itself, have been deferred for months or years.34 3.37 The balance of power between the development agencies and the ministries of education on the Steering Committee has also raised the issue of equity. In terms of sheer numbers, the agencies have twice as many seats on the Steering Committee as the ministry representatives. In addition to this, there is a reported tendency for development agencies to dominate the discussions often due to structural inequalities -- ministries being reluctant to rock the boat since they are generally the recipients of financial support from the agencies. Others contend that ministries contribute less than they might because of frequent turnover of ministers and because so many members are inexperienced in ADEA. There appears to be no 32. Legitimacy would presumably be strengthened even further if ADEA follows through on a serious proposal that its Caucus of Ministers be replaced by the African Union's Council of Ministers of Education (COMEDAF). A possible downside of this could be an increase in formality and political correctness among Ministers and a decrease in openness and frankness. 33. One recent exception, presenting a slight challenge to legitimacy, is the Swedish International Development Agency, which surprisingly withdrew from ADEA in 2006, contending that COMEDAF (of the African Union) was a more legitimate regional education body. It is not clear what SIDA will do if ADEA replaces its Caucus of Ministers with COMEDAF (see previous footnote). 34. Ironically, one of the recommendations of the 2005 external evaluation was that an "executive board" be formed to assist the SC in covering some of the more routine tasks, leaving the SC to focus on oversight, quality assurance, and strategic direction; but the decision to create such a board, its membership and function is itself taking time. 26 simple formula for rebalancing: doubling the ministry members would make the SC even more unwieldy, but halving the number of agency members would put legitimacy in jeopardy. Also, the 2005 external evaluation found that in recent years the African ministries have been setting the Association's priorities, suggesting that the need for rebalancing may not be crucial. What is crucial is finding ways to deal with the endemic problem of turnover among ministry delegates.35 One informant suggested bringing in as non-voting SC members or associate members long-serving permanent secretaries or directors general, who have a long history of policy reform experience, or prominent ministers "emeritus" (no more than one per country), who could add some depth to the discussions. 3.38 The day-to-day management of ADEA lies with the Secretariat, led by an Executive Secretary. As mentioned by the 2005 external evaluation, the Association has been blessed with dedicated, visionary, and highly respected Executive Secretaries, including the incumbent; many staff members have also given extraordinary service. Similar to the SC, the Secretariat's volume of work has expanded faster than the resources available (funds, time, space and skills) and some of the more substantive tasks (management support to working groups, synthesizing and disseminating cutting-edge messages, or gathering and analyzing monitoring and evaluation data, have taken a back seat to administrative ones. The creation a new post of deputy executive secretary or manager has been proposed as a way of increasing office efficiency. Another way would be to elevate the stature of current administrative and financial team.36 The 2005 evaluation also found that increasingly tense relations between the ADEA Secretariat and its host institution, IIEP, were putting a drag on institutional efficiency.37 The next few months will be crucial in ADEA's efforts to improve managerial efficiency: (a) it will need to recruit a new Executive Secretary, given the retirement of the incumbent in 2008; (b) it will realign its budget and its secretariat functions -- perhaps involving changes in job descriptions and personnel -- to the new medium term strategic framework (2008­2012); and (c) it will move its Secretariat to Africa, shifting its management partner from IIEP to AfDB. All of these changes can breathe new life into the organization, but can also put it at some risk of losing momentum. 3.39 According to this review, the main shortcomings in ADEA governance and management, and areas for future improvement, are: (a) the rapid turnover among SC members and its influence on quality of discussions and decision making; (b) the predominance of restricted funds within ADEA's budgeting system; (c) the continuation of unproductive working groups and the slow pace of "graduating" productive ones; (d) the shortcomings in making ADEA processes and products known beyond those directly involved (outreach); (e) the under-emphasis on evidence-based self-evaluation, and (f) the overloading of both the Steering Committee and the Secretariat. 35. This also appears to be a problem with some development agency delegations. Solutions for them would be quite different from those for ministries. 36. In the past three years the proportion of budget allocated to the Secretariat has been stable at 16-17 percent, a share in line with that of other global programs. Thus, it may not be that increasing budget is the answer, but perhaps reorganizing tasks and streamlining procedures. The impending move to Africa (the AfDB in Tunis) will present a chance for reorganization. 37. Tensions have arisen over some budget issues (see paragraph 1.18), contractual terms for Secretariat staff, and space available for ADEA staff and interns at the IIEP premises. 27 4. World Bank's Performance as a Partner 4.1 The World Bank has played numerous roles in ADEA: producer of the policy paper on African education that led to founding of ADEA; co-founder of the Association; initial host organization; a lead financial supporter; holder of a seat on Steering Committee; frequent source of leadership for the Secretariat and (at least once) the Steering Committee; source of professionals for many technical assistance tasks; agency coordinator for Working Groups (Higher Education and Post-Primary Education); and managing the Donor Information System (which eventually became a function of the Secretariat)). These crucial roles, often interacting in complex ways, have made the World Bank an indispensable partner in ADEA. This section will cover the following four features of Bank's performance as a partner which are generally covered in Global and Regional Program Reviews: (a) comparative advantage at the global and country levels; (b) exercising effective and independent oversight; (c) identifying and managing risks; and (d) disengagement strategy. 4.2 As mentioned earlier, a major WB policy paper documenting the mounting crisis in African education in the late 1980s provided the foundation and rationale for a regional program on African education, which became ADEA. After launching the paper, the Bank also convened the first gathering of development agencies active in the sector in Africa along with African educational officials to consider new ways of stimulating policy reform and inter-agency cooperation. Thus the Bank exercised its comparative advantage as a convener and as the largest provider of external funding to education in Africa to play a leading role in the creation of this new regional initiative. The Bank also provided seed money and staff time for backup support once the agencies involved decided to create a backstopping body. The Bank's leadership and recognized stature was helpful in leveraging support from other development agencies -- in the early years at a ratio of 1:5 (one part Bank funding to five parts by other agencies), moving to about 1:15 in the middle of the current decade, and back to about 1:7 after 2006. 4.3 ADEA has also added value to the country programs supported by the Bank. Many current and former Bank officials interviewed for this report pointed out how the ADEA's convening power brought legitimacy and weight to Bank-supported reforms at the country level. Championing such ideas by itself could have left the impression of the Bank's trying to impose its will, but there is less suspicion when the policy positions are taken over and furthered by ADEA (although still some risk, see paragraph 4.5). This has allowed many Bank- supported programs to bear fruit at the country level, such as evidence-based policy adjustment, the need for increased national political commitment as a prerequisite for reaching EFA goals, the need for a new "culture of quality" to enhance basic education outcomes, a firm commitment to further girls participation in education and girl-friendly schooling, and closer government cooperation with NGOs, teachers' unions and parent associations in reforming educational policy and practice. Changes in these areas have been found in many African countries, as a result of country involvement in ADEA. Instances of such value added are not universal, however. Instead they are patchy and concentrated in those countries where a Minister and/or the Bank team have become deeply and personally engaged with ADEA (e.g., during the time of ADEA bienniales). Of the nine TTLs of education sector projects interviewed for this review, three indicated that ADEA products and services were used in their country's operational work a substantial amount but four indicated that these were used "a 28 negligible amount" (Table 2). The value added of ADEA still seems to vary according to the level of engagement of the government and the Bank's country team. 4.4 When the ADEA Secretariat was created in Paris in 1992, Bank involvement shifted to the oversight mode. Over the 18 year period from 1990 to 2007 this oversight was handled by five Bank managers -- Peter Moock, Avril Van Adams, Adriaan Verspoor, Birger Fredriksen, and Dzingai Mutumbuka. All have been recognized for their professionalism and vision. Additional quality assurance support has come in the form of technical assistance from two prominent, retired Bank education task managers who helped to create background papers for the two recent bienniales on education quality. Bank oversight has also included financial support for the external evaluations that were conducted in 1994, 2000 and 2005. All three are considered to have been objective and constructive. In 2005, in a break with precedent, the SC elected a World Bank Sector Manager as the co-chair of the Steering Committee.38 The World Bank Education Advisor for the Africa Region took up the IDA seat on the Committee. In late 2007, on the Sector Manager's retirement from the Bank, the co-chair position rotated to someone outside the Bank. 4.5 The unprecedented situation in 2005­2007, when a Bank Sector Manager for education in Africa was also the co-chair of ADEA, put the Bank in a possible conflict of interest position, for example, when he was called upon both to judge ADEAs' performance (as Sector Manager) and advocate for continued funding for ADEA (as co-chair). Such conflict of interest situations are becoming increasingly prevalent where the Bank is an active participant in global and regional programs. In this case, it should have been acknowledged (and if it happens in the future, should be acknowledged) so that steps to address it can be taken.39 Another risk is that the Bank could be seen as having an undue influence over ADEA's agenda. In the early years of the DAE, the agenda did spring mainly from the Bank's policy paper and its conclusions. Over the years, however, the Association has transformed itself into a forum for enacting an African-owned agenda, at the regional and country level. Still, with the chair of the SC being a Bank official and the Executive Secretary being a former Bank employee, there is the possibility of decisions being biased in the direction of the Bank's goals and preferences. While this review came across no concerns about this risk, either in partner agencies or among Ministers of Education, the Bank could get a neutral party to sound this out periodically in order to mitigate this risk.40 38. There had been an unwritten rule that the leadership of the SC should be given to a representative of one of the "smaller agencies." In 2005 no candidate from these agencies was available for the position, given the fact that most were now providing mostly program support (e.g., through sector-wide programs) and thus had a reduced number of education experts on staff. 39. One way to mitigate the such a risk would be for the Sector Manager to recuse himself from Sector Board discussions of ADEA performance and budget decisions. 40. World Bank Management has acknowledged the existence of such conflicts of interest in the Bank's oversight and management of global and regional programs and, in its formal response to IEG's Phase 2 Report on the Bank's involvement in global programs, made a commitment to develop TORs for Bank staff serving on the governing bodies of global programs. That such Bank-wide guidelines have not yet been developed is not the fault of the Education Sector Manager or ADEA. Rather it is to their credit that they appear to have managed these potential conflicts of interest in a pragmatic way in the absence of any Bank-wide guidelines for this purpose. However, the need to develop such guidelines still remains. 29 4.6 There has been no serious discussion of a World Bank disengagement strategy from ADEA, despite its having sponsored the Association for almost two decades. The Bank's financial support through the Development Grant Facility is classified as "window 1," which signals a long-term engagement, at least through 2010. However, task managers have argued the need for an even longer engagement, given that the Millennium Development Goals in education, a key ADEA focal point, will culminate in 2015, and more than half of SSA countries are still off-track for reaching them. In addition, the next few years are seen as a crucial transition period for ADEA, given its new medium-term strategy (2008­2012), and the move of its Secretariat to an African country, starting in 2008, and the need to replace the current Executive Secretary when he retires in 2008. Bank financial and professional support are considered as crucial for keeping up momentum during these transitions and leveraging continued support by partner agencies. This is not to deny the importance of improved exit strategies with respect to the ADEA working groups. Only one group, FAWE, has graduated so far, whereas others have been on the brink of this for years or are almost dormant (already mentioned in the 2000 external evaluation), suggesting that some WGs are being carried for too long. It is crucial that this issue be taken up as the new strategic plan is put into place. 5. Lessons Learned 5.1 Over the course of almost two decades, ADEA has grown from a small consortium of donor agencies and a few ministries of education to become Africa's premier forum for educational policy development and agency cooperation. Its influence is largely a consequence of its growing legitimacy (it now includes as partners all African ministries of education and all major development agencies active in African education) and its relevance (the focus of its work has evolved along with the region's educational issues). However, such a high level of legitimacy has had costs in terms of efficiency, as decisions have not been arrived at quickly and insufficient attention has been given to strategic planning and performance assessment. These concerns are now being addressed with the formulation of a new strategic plan for 2008­2012 and the establishment of an Executive Board, which will manage the operation leaving the Steering Committee to focus on more strategic and substantive matters. ADEA's core strength has been its convening power, which allows it to promote policy dialogue through a variety of forums such as biennial meetings, technical gatherings, and the activities of its 15 working groups. It also promotes country and regional analytical work on educational problems, constraints and solutions; the strengthening of local capacity for educational planning and related skills; and the creation of regional and sub- regional professional networks. In each of these areas ADEA has shown both strengths and weaknesses, making it possible to identify a number of important lessons. Some of the following lessons are previously identified lessons which have not yet been heeded, some are in the process of being acted upon, some are modifications of previously identified lessons, and some are new. · Strategic planning for ADEA's future, which has focused mostly on desired processes and programs, also needs to specify organizational objectives in terms of desired outcomes (e.g., EFA outcomes, such as improved access or learning outcomes). This is a main point in the 2005 external evaluation which has not yet been heeded. In addition to desired outcomes, there is a need for measurable performance indicators. Even some of the so called "intangibles" that are prized in ADEA could be tracked in some way to 30 show how ADEA is measuring up to its ideals. This calls for an agreement among ADEA partners as to the type of results-oriented monitoring and evaluation framework that would be reasonable and credible, given ADEA's current and emerging objectives, resources, and organizational structure. · ADEA working groups need to be realigned with the Association's new strategic plan and subject to a new set of evaluation criteria which the Association needs to apply with firmness (a work already in progress). This would presumably lead to phasing out or graduation of some current working groups. The new pattern of forming ad hoc working groups could also be reviewed and modified as appropriate, given that its acknowledged benefits are accompanied by certain disadvantages, such as the groups not being anchored in Africa and their placing an administrative burden on the Secretariat. · Analytical work supported by ADEA is growing in volume but is of uneven quality. As ADEA moves into new realms of applied research it needs to set higher quality standards, and to concentrate as much on the quality (usefulness) of the research as on quantity (output). This recognizes the important capacity and ownership-building benefits of drawing new, local researchers into the process as co-investigators with more senior researchers. For research used as background documents in biennial and regional meetings and those published by ADEA, it would be desirable to establish a formal policy and procedures for document review. · Creative solutions to chronic overburdening of the ADEA Steering Committee must be found so that the Committee has time to exercise more substantive leadership, quality oversight, and identification of cutting edge issues. The formation of a smaller, constituency-based Executive Board would be a step in this direction. · Rapid turnover of Steering Committee members, particularly those from the Caucus of Ministers, is undermining the strength, confidence, continuity and institutional knowledge of their inputs into policy dialogue and decision making. Ways need to be considered to bring more permanent ministry officials into ADEA decision making circles and to find roles for emeritus committee members who are still prominent and influential in education circles. · ADEA programs and products are having a significant impact on those in the ADEA inner circle, but do not reach very far beyond this. Ways need to be found to extend the reach of ADEA programs and outputs, using country focal points or other innovative vehicles. 31 Annex A Annex A. Evaluation Framework for Global Program Reviews Note: This evaluation framework is a general framework that has been designed to cover the wide range of such programs in which the World Bank is involved, encompassing policy and knowledge networks, technical assistance programs, and investment programs. It is not expected that every global program review will cover every question in this table in detail. Annex Table 1. Assessing the Independence and Quality of the Evaluation Evaluation Questions 1. Evaluation process To what extent was the GRPP evaluation independent of the management of the program, according to the following criteria: · Organizational independence? · Behavioral independence and protection from interference? · Avoidance of conflicts of interest? Factors to take into account in answering these questions include: · Who commissioned and managed the evaluation? · Who approved the terms of reference and selected the evaluation team? · To whom the evaluation team reported, and how the evaluation was reviewed? · Any other factors that hindered the independence of the evaluation such as an inadequate budget, or restrictions on access to information, travel, sampling, etc.? 2. Monitoring and evaluation framework of the program To what extent was the evaluation based on an effective M&E framework of the program with: · Clear and coherent objectives and strategies that give focus and direction to the program? · An expected results chain or logical framework? · Measurable indicators that meet the monitoring and reporting needs of the governing body and management of the program? · Systematic and regular processes for collecting and managing data? 3. Evaluation approach and scope To what extent was the evaluation objectives-based and evidence-based? To what extent did the evaluation use a results-based framework -- constructed either by the program or by the evaluators? To what extent did the evaluation address: · Relevance · Governance and management · Efficacy · Resource mobilization and financial management · Efficiency or cost-effectiveness · Sustainability, risk, and strategy for devolution or exit 4. Evaluation instruments To what extent did the evaluation utilize the following instruments: · Desk and document review · Consultations/interviews and with whom · Literature review · Structured surveys and of whom · Site visits and for what purpose: for interviewing implementers/beneficiaries, or for observing activities being implemented or completed · Case studies · Other Annex A 32 Evaluation Questions 5. Evaluation feedback To what extent have the findings of the evaluation been reflected in: · The objectives, strategies, design, or scale of the program? · The governance, management, and financing of the program? · The monitoring and evaluation framework of the program? Annex Table 2. Providing an Independent Opinion on the Effectiveness of the Program Every review is expected to cover the first four criteria in the following table: (a) relevance, (b) efficacy, (c) efficiency, and (d) governance and management. A review may also cover (e) resource mobilization and financial management and (f) sustainability, risk, and strategies for devolution or exit if the latter are important issues for the program at the time of GPR, and if there is sufficient information available on which to base an independent opinion. Evaluation Criteria and Questions Relevance: The extent to which the objectives and design of the program are consistent with (a) current global/regional challenges and concerns in a particular development sector and (b) the needs and priorities of beneficiary countries and groups. 1. Supply-side relevance -- the existence of an international consensus that global/regional collective action is required. To what extent does the program reflect an international consensus on the need for action, on the definition of the problem being addressed, on priorities, and on strategies for action? Is the original consensus that led to the creation of the program still present? Is the program still needed to address specific global/regional public concerns? Take into account the origin of the program in answering these questions: · Is the program formally responsible for implementing an international convention? · Did the program arise out of an international conference? · Is the program facilitating the implementation of formal standards and approaches? · Is the program primarily donor-driven? Did donors establish the program with little consultation with developing countries? · Is the program primarily Bank-driven? Did the World Bank found the program and then seek other partners? 2. Demand-side relevance -- alignment with beneficiary needs, priorities, and strategies. To what extent are the objectives consistent with the needs, priorities, and strategies of beneficiary countries as articulated in the countries' own PRSPs, and in donors' strategies such as the World Bank CASs, and the UN Development Assistance Frameworks? To what extent has the voice of developing and transition countries been expressed in the international consensus underlying the program? 3. Vertical relevance -- consistency with the subsidiarity principle. To what extent are the activities of the program being carried out at the most appropriate level -- global, regional, national, or local -- in terms of efficiency and responsiveness to the needs of beneficiaries? To what extent are the activities of the program competing with or substituting for activities that individual donors or countries could do more efficiently by themselves? Pay particular attention to those programs that, on the face of it, are primarily supporting the provision of national or local public goods. 33 Annex A Evaluation Criteria and Questions 4. Horizontal relevance -- the absence of alternative sources of supply. What is the comparative advantage, value added, or core competency of the program relative to other GRPPs with similar or complementary objectives? To what extent is the program providing additional funding, advocacy, or technical capacity that is otherwise unavailable to meet the program's objectives? To what extent are the good and services being provided by the program in the nature of public goods? Are there alternative ways of providing these goods and services, such as by the private sector under regular market conditions? 5. Relevance of the design of the program To what extent are the strategies and priority activities of the program appropriate for achieving its objectives? What are the major activities of the program: · Policy and knowledge networking? · Financing country and local-level technical assistance? · Financing investments to deliver national, regional, or global public goods? Has the program articulated an expected results chain or logical framework, along with assumptions that relate the progress of activities with the achievement of the objectives? Does the results chain identify the extent to which the achievement of the objectives depends on the effective functioning of bureaucracies, markets, or collectivities? If so, to what extent are these assumptions valid? For programs providing global or regional public goods, is the design of the program consistent with the way in which the individual efforts of the partners contribute to the collective outcome for the program as a whole -- whether "best shot", "summation", or "weakest link?" Efficacy: The extent to which the program has achieved, or is expected to achieve, its objectives, taking into account their relative importance. 6. Achievement of objectives To what extent have the stated objectives of the program been achieved, or has satisfactory progress been made towards achieving these objectives? To what extent are there implicit objectives that are well understood and agreed upon by the partners and to which the program should also be held accountable? To what extent are there any positive, unintended outcomes of the program that have been convincingly document? To what extent have these assessments by the program or the evaluation been evidence-based? 7. Progress of activities, outputs, and outcomes. To what extent has the program or the evaluation measured the progress of activities, outputs, and outcomes? How did the program or the evaluation aggregate its outputs and outcomes at all levels -- global, regional, national, and local -- to provide an overall summary of its results? To what extent have factors such as changes in the location of the program, its legal structure, or governance processes affected the outputs and outcomes of the program? To what extent have there been outcomes that can be uniquely attributed to the partnership itself -- such as the scale of or joint activities made possible by its organizational setup as a GRPP, or its institutional linkages to a host organization? 8. Linkages to country or local-level activities. To what extent has the program established effective operational linkages with country-level activities, taking into account that: · The desired nature of these linkages will vary according to the objectives, design, and implementation of each program? · Positive outcomes at the country or local level are generally a joint product of both global/regional and county- level activities? Annex A 34 Evaluation Criteria and Questions Efficiency or cost-effectiveness: Efficiency -- the extent to which the program has converted or is expected to convert its resources/inputs (such as funds, expertise, time, etc.) economically into results. Cost-effectiveness -- the extent to which the program has achieved or is expected to achieve its results at a lower cost compared with alternatives. 9. Efficiency To what extent is it possible to place a monetary value on the benefits arising from the activities of the program? To what extent has the program or the evaluation conducted impact evaluations of representative program activities? To what extent has the program or the evaluation analyzed the program's costs in broad categories (such as overhead vs. activity costs), and categorized the program's activities and associated benefits, even if these cannot be valued in monetary terms? 10. Cost-effectiveness To what extent is the program measuring up against its own business plans: · Has the program cost more or less than planned? How did it measure up against its own costing schedule? · Have there been any obvious cases of inefficiency or wasted resources? To what extent is the program delivering its activities cost-effectively in comparison with alternatives: · How do actual costs compare with benchmarks from similar programs or activities? · Are the overhead costs of governing and managing the program reasonable and appropriate in relation to the objectives and activities of the program? How does the program compare with traditional development assistance programs: · For beneficiary countries, has receiving the development assistance through the GRPP increased the transactions costs compared with traditional development assistance programs? · For donors, has delivering the development assistance through the GRPP reduced donor costs by harmonizing efforts among donors or by reducing overlapping work (such as through joint supervision, monitoring and evaluation)? Governance and management: Governance -- the structures, functions, processes, and organizational traditions that have been put in place within the context of a program's authorizing environment to ensure that the program is run in such a way that it achieves its objectives in an effective and transparent manner. Management -- the day-to-day operation of the program within the context of the strategies, policies, processes, and procedures that have been established by the governing body. Whereas governance is concerned with "doing the right thing," management is concerned with "doing things right." 11. Compliance with generally accepted principles of good governance. To what extent are the governance and management structures and processes well articulated and working well to bring about legitimate and effective governance and management? To what extent do governance and management practices comply with the following seven principles: · Legitimacy -- the way in which governmental and managerial authority is exercised in relation to those with a legitimate interest in the program -- including shareholders, other stakeholders, implementers, beneficiaries, and the community at large? · Accountability -- the extent to which accountability is defined, accepted, and exercised along the chain of command and control within a program, starting with the annual general meeting of the members or parties at the top and going down to the executive board, the chief executive officer, task team leaders, implementers, and in some cases, to the beneficiaries of the program? · Responsibility -- the extent to which the program accepts and exercises responsibility to stakeholders who are not directly involved in the governance of the program and who are not part of the direct chain of accountability in the implementation of the program? · Fairness -- the extent to which partners and participants, similarly situated, have equal opportunity to influence 35 Annex A Evaluation Criteria and Questions the program and to receive benefits from the program? · Transparency -- the extent to which a program's decision making, reporting, and evaluation processes are open and freely available to the general public? · Efficiency -- the extent to which the governance and management structures enhance efficiency or cost- effectiveness in the allocation and use of the program's resources? · Probity -- the adherence by all persons in leadership positions to high standards of ethics and professional conduct over and above compliance with the rules and regulations governing the operation of the program? 12. Partnerships and participation To what extent has the program identified a complete list of stakeholders, or "stakeholder map", including the agreed- upon or perceived roles and responsibilities of the categories of stakeholders identified? To what extent is this a routine programmatic function, updated regularly, and transparently available? Has the program adopted primarily a shareholder model of governance (in which membership on the governing body is limited to financial and other contributors), or a stakeholder model (in which membership also includes non- contributors)? To what extent, if any, is the program's legitimacy being sacrificed in order to achieve greater efficiency, or vice-versa? 13. Programs located in host organizations To what extent is the location of the program in the Bank or other partner organization adversely affecting the governance, management, or other aspects of the program, such as compliance with the principles of transparency and fairness? For which functions is the program manager accountable to the host organization and the governing body of the program, respectively? Are conflicts of interest being managed appropriately? To what extent does the host organization play such a dominant role in the program, thereby reducing the incentives of other partners to participate effectively, or reducing the ability of the host organization to look at the weaknesses of the program objectively? Resource mobilization and financial management: Resource mobilization -- the processes by which resources are solicited by a program and provided by donors and partners. Financial management -- the processes that govern the recording and use of funds, including allocation processes, crediting and debiting of accounts, controls that restrict use, accounting, and periodic financial reporting systems. In cases where funds accumulate over time, this would also include the management of the cash and investment portfolio. 14. Resource mobilization To what extent has the program succeeded in raising financial resources commensurate with its objectives? And from what sources -- the Bank, bilateral donors, foundations, etc.? To what extent has the program succeeded in diversifying its funding beyond a small number of donors? To what extent are the sources of funding for the program (including donor restrictions on the use of resources) affecting, positively or negatively: · The strategic focus of the program? · The outputs and outcomes of the program? · The governance and management of the program? · The sustainability of the program? 15. Financial management Are there any issues that have emerged during the course of the review in relation to: · The quality of financial management and accounting? · The methods, criteria, and processes for allocating funds among different activities of the program? · Financial management during the early stages of the program? Annex A 36 Evaluation Criteria and Questions Sustainability, risk, and strategy for devolution or exit: Sustainability -- When applied to the activities of a program, the extent to which the benefits arising from these activities are likely to continue after the activities have been completed. When applied to a program itself, the extent to which the organization or program is likely to continue its operational activities over time. Devolution or exit strategy -- a proactive strategy to change the design of a program, to devolve some of its implementation responsibilities, to reduce dependency on external funding, or to phase out the program on the grounds that it has achieved its objectives or that its current design is no longer the best way to sustain the results which the program has achieved. 16. Sustainability of the benefits of the program's activities What is the risk, at the time of evaluation, that the development outcomes (or expected outcomes) of the program will not be maintained (or realized)? This depends on (a) the likelihood that some changes may occur that are detrimental to maintaining or realizing the expected outcomes, and (b) the affect on the expected outcomes if some or all of these changes actually materialize? 17. Sustainability of the program This will depend on a number of factors, such as the continued legitimacy of the program, its financial stability, its continuity of effective management, and its ability to withstand changing market or other conditions. To what extent is there still a sufficient convergence or accommodation of interests among the major partners to sustain the program financially? To what extent has the program developed institutional capacity such as performance- based management, personnel policies, learning programs, and knowledge management that help to sustain a program? In what areas could the program improve in order to enhance its sustainability, such as better marketing of the program's achievements in order to sustain its reputation? 18. Prospects for continuation and strategies for devolution or exit To what extent should the program be sustained? Is the continuation of the program the best way of sustaining the results achieved? Should the design of the program be modified as a result of changed circumstances, either positive or negative? What other alternatives should be considered to sustain the program's results more cost-effectively, in the light of the previous evaluation findings with respect to relevance, efficacy, efficiency, and sustainability: · Reinventing the program with the same governance? · Phasing out the program? · Continuing country or local-level activities with or without devolution of implementation? · Seeking alternative financing arrangements, such as revenue-generation, or self-financing to reduce dependency on external sources? · "Spinning off" from the host organization? 37 Annex A Annex Table 3. Assessing the Bank's Performance as a Partner in the Program Evaluation Questions 1. Comparative advantage at the global/regional level. To what extent is the Bank playing up to its comparative advantages at the global/regional level -- its global mandate and reach and convening power? To what extent is the Bank's presence as a partner in the program catalyzing other resources and partners for the program? 2. Comparative advantage at the country level. To what extent is the Bank contributing multi-sector capacity, analytical expertise, and country-level knowledge to the program? To what extent has the Bank's country operations established linkages to the GRPP, where appropriate, to enhance the effectiveness of both? 3. Oversight. To what extent is the Bank exercising effective and independent oversight of its involvement in the program, as appropriate, whether the program is housed in the Bank or externally managed? To what extent is the Bank's oversight independent of the management of the program? To what extent does the Bank's representative on the governing body have a clear terms of reference? 4. Risks and risk management. To what extent have the risks associated with the program been identified and are being effectively managed? For example, IEG identified the following risks in its global review: · Bank bears a disproportionate share of responsibility for governing and managing in-house programs? · Confusion at the country level between global program activities, Bank activities, and Borrower activities? · Representation of NGOs and the commercial private sector on program governing bodies? · Unclear role and application of Bank's safeguards? · Trust-funded consultants and seconded staff representing the Bank on some program governing bodies? 5. Disengagement strategy. To what extent is the Bank engaged at the appropriate level in relation to the Bank's new strategic framework: · Watching brief? · Research and knowledge exchange? · Policy or advocacy network? · Operational platform? To what extent is the Bank facilitating an effective, flexible, and transparent disengagement strategy for the program, in relation to the Bank's objectives for its involvement in the program: · The program declares "mission accomplished" and closes? · The program continues and the Bank withdraws from all aspects of its participation? · The program continues and the Bank remains engaged, but the degree of the Bank's engagement in some or all aspects (such as financing) declines over time? Annex B 38 Annex B. Key Dates in ADEA History 1988 · Donors to African Education (DAE) is founded. 1992 · DAE moves from the World Bank (Washington) to the IIEP (Paris). · The Steering Committee expands to include a Bureau of Ministers. · The Association evolves into a partnership between development agencies and ministries of education. 1995 · DAE changes its name to ADEA (Association for the Development of Education in Africa). 1997 · The Bureau of Ministers expands to ten ministers. It is decided hereafter to hold all Biennial Meetings in Africa. 2007 · The Steering Committee decides to move the ADEA Secretariat to Tunis, Tunisia in Association with the African Development Bank. · It is decided that ADEA will cover all of Africa and not just sub-Saharan Africa. · A new Strategic Plan for ADEA is drafted by the Secretariat (yet to be fully ratified by the Steering Committee. 39 Annex C Annex C. ADEA Working Groups Working Group Agency Regional Coordinator Year Year closed Coordinator established or graduated School Examinations HEDCO ­ Ireland 1988 Closed Vocational Education ILO 1988 Dissolved and Training 1995 Book Sector (Books ODA (DFID) 1988 & Learning Materials) Higher Education World Bank Association of African 1988 Universities Accra, Ghana Education Statistics SIDA UNESCO 1988 BREDA Teaching Profession Commonwealth W. Africa: Togo 1988 Secretariat E. Africa: Uganda Indian Ocean: Seychelles Southern Africa: Zambia Mobilization of French 1988 ?? Resources Government/ World Bank Donors Information World Bank 1988 Became System function of ADEA Secretariat Action Group on UNESCO 1988 Exchange of Sector Studies Educational IDRC 1989 Dissolved Research and Policy 2000 Analysis Female Participation Rockefeller Forum for African Women 1989 Graduated Foundation Educationists (FAWE) Evaluation of French 1990 Closed Education Systems Government Finance & Education CIDA CODESRIA 1994 Dakar, Senegal Non-formal Swiss Agency for 1996 Education Development and Cooperation; UNESCO (UIE) Early Childhood Netherlands UNICEF Ghana 1997 Development Government Distance Education UNESCO BREDA Mauritius Ministry of Ed & 1997 and Open Learning Research Communication for NORAD Wanad Center Benin ?? Education and Development Annex C 40 Working Group Agency Regional Coordinator Year Year closed Coordinator established or graduated Mathematics and Center for Math, Science 2004 Science Education & Technology in Africa, Nairobi, Kenya Ad Hoc WG on HIV/ ADEA Secretariat 2001 AIDS and Education Ad Hoc WG on ADEA Secretariat 2002 Quality Education Ad Hoc WG on ADEA Secretariat 2003 Policy Dialogue Ad Hoc WG on Post- World Bank 2004 Primary Education 41 Annex D Annex D. ADEA Biennial Meetings, 1991­2006 Date Location Theme Ministers Present October 28 ­ Manchester, United Textbook Provision and 30 ministers or other November 1, Kingdom Library Development in Africa Senior officials 1991 October 22­24, Angers, France Improving the Implementation Over 30 ministers 1993 of Educational Projects in Africa through Ownership October 18­22, Tours, France Formulating Educational Policy 1995 in Sub-Saharan Africa October 14­18, Dakar, Senegal Partnership for Capacity 1997 Building and Quality Improvements in Education in Africa December 5­9, Johannesburg, S. What Works and What's New 1999 Africa in Education: Africa Speaks! October 7­11, Arusha, Tanzania Reaching Out, Reaching All: 2001 Sustaining Effective Policies and Practices for Education in Africa December 3­6, Grand Baie, The Quest for Quality: 2003 Mauritius Learning from the African Experience March 27­31, Libreville, Gabon More and Better Education: 26 countries 2006 What makes Effective Learning represented by in Schools and in Literacy ministers of education Programs and Early Childhood Development Programs? Annex E 42 Annex E. ADEA Organizational Chart STEERING COMMITTEE Chair: Mr. Dzingai Mutumbuka (World Bank); Alternate Chair: Hon. Rosalie Kama-Nyamayou (Congo); Vice-Chair (ex-officio): Mark Bray (IIEP), all as of 2006* Development Agency Members Bureau of Ministers (as of 2006) African Development Bank West Africa : Austria Senegal, Burkina Faso, Mali Carnegie Corporation CIDA (Canada) Southern Africa: Denmark Angola, Mozambique DfID (United Kingdom) European Commission Central Africa: Finland Central African Republic, Congo France GTZ (Germany) Eastern Africa: Gulbenkian Foundation Tanzania, Rwanda Ireland JICA (Japan) Indian Ocean: Netherlands Mauritius Norway Rockefeller Foundation SDC (Switzerland) UNESCO UNICEF USAID (United States) World Bank SECRETARIAT Executive Secretary: Mamadou Ndoye Senior Program Specialist: Hamidou D. Boukary Publications/Communications Officer: Thanh Hoa Desruelles Program Specialist: Maud Seghers Administrative and Finance Officer: Sabine Ayeh Senior IT and Database Specialist: Beedeeanun Conhye *These officers plus the Executive Secretary make up what is referred to as the Bureau of ADEA. 43 Annex F Annex F. Technical and Sub-Regional Meetings, 2003­2007 Topic Location Date Effective Responses to the HIV/AIDS Pandemic in the Libreville, Gabon May 27­29, Education Sector: From Analysis to Action (co-sponsored 2003 by UNAIDS) Education for Lusophone and Hispanic Countries (co- Luanda, Angola Nov 29 ­ sponsored by IBE and the World Bank Institute) Dec 4, 2004 Teaching in a World with AIDS (co-sponsored by Nov 29­Dec 1, UNESCO) 2004 Education in Countries in Crisis or Post-Conflict Mombasa, Kenya June 2­4, 2004 Integration of ICT in Education: Issues, Challenges, and Abuja, Nigeria July 26­30, Infrastructure in Africa 2004 Employment and Use of Contractual Teachers in Primary Bamako, Mali Nov 21­23, Education (co-sponsored by the World Bank, Education 2004 International, and the Government of Mali) 3rd African Conference on Early Childhood Development Accra, Ghana May 30 ­ (co-sponsored by the Government of Ghana, World Bank, June 3, 2005 UNESCO, the Consultative Group on Early Childhood Development, and the Virtual University) Bilingual Education and the Use of Local Languages in Windhoek, Namibia August 3­5, Educational Systems (co-sponsored by GTZ, UNESCO 2005 Institute for Education, and the Agence Intergouvere- mental de la Francophonie) Assessment and Certification of Competences for the Grand Baie, Mauritius May 2­4, 2005 Indian Ocean Countries (co-sponsored by the MOES Mauritius and IIEP) Education for Rural People in Africa (co-sponsored by Addis Ababa, Ethiopia Sept 7­9, FAO, the French Ministries of Foreign Affairs and 2005 Agriculture and the Italian Ministry of Foreign Affairs) Financing of Literacy Programs Morocco Dec 19-20, 2006 Secondary Education in Africa (co-sponsored by the · Kampala, Uganda Jn 9­13, 2004 World Bank) · Dakar, Senegal Jn 6­9, 2004 · Accra, Ghana Apr 1­4, 2007 Conference on School Fee Abolition (co-sponsored by Bamako, Mali June 19-22, UNICEF and the World Bank) 2007 Annex G 44 Annex G. Uses of ADEA Funds, 2001­2006 2001 2002 2003 2004 2005 2006 PROGRAM Audited Audited Audited Audited Audited Estimated Policy Dialogue 748,033 257,641 817,788 512,276 593,721 811,196 Regional and Sub- 0 47,985 204,383 460,862 317,948 87,517 regional Cooperation Ad hoc Working Groups 0 0 861,563 171,380 184,646 168,629 Support to Working 1,464,775 1,930,388 1,998,653 2,322,774 3,109,975 3,034,236 Groups Complementary Programs 282,492 504,677 merged with other programs Coordination, Facilitation & Knowledge 138,986 120,980 76,981 205,789 182,307 247,826 Management Publications & 179,960 189,634 408,006 651,481 381,572 347,472 Communications Secretariat 0 0 0 923,415 977,967 1,064,370 Overhead and Reserve 625,070 674,064 829,787 0 295,351 831,963 TOTAL COSTS 3,439,316 3,725,369 5,197,161 5,247,977 6,043,487 6,593,209 45 Annex H Annex H. IEG Assessment of the Quality of ADEA's Analytical Work Report Number #7 #10 #15 #22 #24 #35 #36 #43 #47 #50 Dimensions 1. Task Objectives a. Clearly stated? 2 3 3 1 1 1 2 2 1 3 b. Relevant to ADEA mission 1 2 1 1 1 1 3 1 1 2 2. Quality of analysis a. Methods appropriate 2 3 3 1 2 2 3 3 1 4 b. Quality of the evidence 3 3 2 1 1 2 3 3 2 3 3. Quality of conclusions (appropriately specific and consistent with analysis 3 3 2 1 1 2 3 3 1 3 4. Quality of recommendations a. Clearly stated? 2 4 1 1 1 3 1 2 1 2 b. Flow well from conclusions 2 4 1 1 1 3 4 1 1 3 c. Likely to have an impact (feasible)? 3 4 1 1 2 2 1 3 1 2 Note: Dimensions rated on a four point scale with 1 being the highest Annex J 46 Annex I. Persons Consulted Members of the ADEA Secretariat Mamadou Ndoye, Executive Secretary Hamidou Boukary, Senior Program Specialist Thanh-Hoa Desruelles, Publications and Communications Officer Maud Seghers, Program Specialist Beedeanun Conhye, ADEA Database and IT Specialist Richard Sack, former Executive Secretary Partner Agency Representative Alice Hamer, African Development Bank Cream Wright, UNICEF Ann-Therese Ndong-Jatta, UNESCO, Director Pre-Primary to Secondary Education Joyce Moock, Rockefeller Foundation (retired) Mark Bray, Director, IIEP Robert Prouty, Deputy Secretary, EFA-Fast Track Initiative World Bank Staff Members (active and retired) Dzingai Mutumbuka, Education Sector Manager, and Co-Chair ADEA Steering Committee Jee-Peng Tan, Education Advisor, Africa Region Birger Fredriksen, former Senior Education Advisor for Africa (2001­04) and Sector Director for Human Development (Africa) (1997­2001) Peter Moock, First and founding ADEA Executive Secretary and former, Adriaan Verspoor, consultant AFTHD, former Lead Education Specialist, Keiko Inoue, Operations Officer, AFTHD World Bank TTLs in Africa: Olatunde Adetoyoyese Adekola, TTL for Nigeria Jacob Bregman, TTL for Cameroon and Nigeria Rest Barbabas Lasway, TTL for Tanzania Xiaoyan Liang, TTL for Mozambique Mmantetsa Marope, TTL for Malawi Susan Opper, TTL for Rwanda Mercy Tembon, TTL for Mauritania Gary Theisen, TTL for Ethiopia Academics and NGO Representatives Martial Dembele, Associate Professor, University of Montreal Penelope Bender, Plan USA 47 Annex I Minister and Ex-Ministers of Education Rosalie Kama Niamayoua, Minister of Education, Republic of Congo Brazzaville, and Chair, ADEA Bureau of Ministers Margareth S. Sitta, Minister of Education and Vocational Training, Tanzania Louis Steven Obeegadoo, Ex-Minister of Education, Mauritius (and ex-Chair of the ADEA Bureau of Minsters) Ann-Therese Ndong-Jatta, Ex-Minister of Education, The Gambia Amadou Wade Diagne, Director, Cabinet d'appui à l'Education, Senegal Annex J 48 Annex J. Official Response of ADEA to IEG's Global Program Review January 29, 2008 ADEA welcomes the World Bank IEG review of ADEA. The timing of the report is particularly opportune as ADEA is completing its first strategic planning exercise following a recommendation stemming from the external evaluation of ADEA conducted in 2005. Comments on Overall Lessons Learned As a result of the strategic planning exercise and ADEA's new Medium-Term Plan (2008- 2012), some of the recommendations spelled out in section 5 listing "lessons learned" are already being acted upon. In particular, we are pleased to inform that ADEA has addressed issues raised in the bulleted lessons learned N°1, N°2, N°4 and N°6 (page 28 of review): · ADEA has re-formulated its strategic objectives (there are five) ­ now officially approved by the Steering Committee ­ and has developed a Performance Evaluation Framework (PEF)for monitoring and evaluating ADEA's progress in achieving the 5 strategic objectives. The PEF includes expected outcomes, outputs and performance indicators. · The process of realigning ADEA's Working Groups to the overall Strategic Plan has started and the ADEA Steering Committee has proposed the merging and graduation of some WGs. As of today, the merging of three long standing Working Groups (Education Statistics, Education Sector Analysis and Finance and Education) is underway, with the view of having one unified Working Group that will bring its expertise to the setting up of the African Union's Observatory for the Second Decade of Education for Africa. · ADEA has decided to create an Executive Committee in charge of operational and management oversight that will relieve the Steering Committee (SC) in order to enable it to focus on more substantive policy, organizational and strategic matters. · ADEA's new Strategic Plan also addresses lesson learned N°6 (the need to extend the reach of ADEA programs and outputs) by putting greater emphasis on outreach activities towards key partners and stakeholders in education in Africa. In response to lesson learned N°3 related to analytical work, it should be stressed that ADEA ­ which considers itself as a learning organization committed to social transformation ­ is first and foremost engaged in fostering a culture of analysis. For the Biennales for example, ADEA tries to get as many countries as possible to engage in a learning process through the analysis of their policies and practices, the learning from their experiences and the sharing of lessons learned with other countries. The ultimate objective sought is the qualitative transformation of policies and practices. For ADEA, the process activated is more important than the actual result and the "quality" of what is produced is not the only element that counts. The process continues after the Biennale as countries are invited to participate in "thematic inter-country quality nodes" where countries interact with other countries and experts/organizations having an edge in specific thematic areas. It should be added that in addition to studies conducted by countries, ADEA also commissions background papers and syntheses from experienced researchers and experts. 49 Annex J Effectiveness of the Program Other comments on the review findings are the following: · Networking (paras. 3.25 and 3.26) goes beyond what is described in the Review. It is an essential element of ADEA's strategy and activities and ADEA defines itself as a network of networks for a host of reasons: as the Review mentions, ADEA WGs have contributed to building networks of African specialists in their specific thematic area (networks of education statistics and information systems specialists, education finance specialists, specialized education journalists and communicators, etc.), ADEA maintains strong links with African research networks such as ERNWACA (Education Research Network for Western and Central Africa) and ERNESA (Education Research Network for Eastern and Southern Africa) and it is developing relations with the Regional Economic Communities (RECs) and engaging them to support the implementation of AU's Second Decade of Education for Africa. In carrying out activities, ADEA also partners with a number of organizations with which it has common areas of interest: in 2007 for example, the Conferences on School Fee Abolition, Secondary Education and Education for Rural People and the High Level Meeting of RECs were organized in collaboration with many partners (UNICEF, the World Bank, FAO, the French Ministries of Foreign Affairs and Agriculture and the Italian Ministry of Foreign Affairs, AU, UNESCO and ADB). · Several mechanisms exist within ADEA to ensure that ADEA's program and activities are country-driven (para. 8):The Bureau of Ministers composed of 10 Ministers of education representing the five sub-regions of Africa voices its concerns and guides ADEA in defining its priorities and activities; the Caucus of Ministers, meeting every two years during the ADEA Biennale, also ensures that ADEA is responsive to country needs and priorities. For instance, the last meeting of the Caucus of Ministers recommended that ADEA facilitate education peer review exercises, which it has done through its Working Group on Education Sector Analysis. Working Groups also make sure that African stakeholders are represented on their Steering Committees and they are increasingly anchored in Africa and coordinated by African organizations, thereby ensuring that their agendas are African-driven and that synergies are developed with African bodies (the WG on Higher Education is coordinated by the AAU based in Accra; the WG on Communication for Education and Development by the West African News Media Organization (WANAD) in Cotonou, the WG on Maths and Science Education by SMASSE-WECSA in Nairobi and the Working Group on Distance Education and Open Learning by the Mauritius Ministry of Education and BREDA). The Review could also mention that ADEA has developed a questionnaire addressed to Ministries of Education to assess their needs on a regular basis and to make sure that ADEA programs and activities respond to these needs. · Regarding development agency coordination (para. 3.10) while there is no systematic outcomes information specifically related to ADEA, there is no doubt today that there is greater coordination among agencies than there was when ADEA was founded and that coordination is increasingly sought. A number of mechanisms specific to the education sector have developed in the last twenty years and are evidence of this: the EFA Movement, the FTI, Sector wide approaches, the Rome Declaration on Harmonization, the EPDF funding mechanism. Although improved coordination cannot be solely attributed to ADEA, it can be stated that ADEA was the Annex J 50 first organization to set this prominently on the agenda of educational cooperation and therefore one of the main instigators for greater coordination among agencies. The existence of PRISME should be mentioned in the Review as a database of externally funded projects and programs on education in Africa, specifically developed by ADEA as a tool for enhanced coordination. · Within ADEA, which defines itself as a learning organization, the notion of "capacity- building" (paras. 3.21 ­ 3.24) is broad and directly linked to the central importance of learning in ADEA's methodological approaches and strategic objectives. Capacity- building goes beyond the technical capacity-building and shifting of WGs in African institutions described under "Strengthening capacities in African countries" (para1.5). Capacity-building within ADEA operates as a learning process building on the exchange of experiences, the sharing of knowledge, the pooling of expertise, strategic partnerships with expert institutions, networking, and, most importantly learning by doing (the "praxis" approach). Capacity-building occurs throughout ADEA's activities e.g. in preparation of ADEA's Biennales when countries are asked to document and to analyze country initiatives, within ADEA's inter-country quality nodes (networks of countries and expert organizations set up to ensure that recommendations coming out of the ADEA Biennales and other major meetings are translated into concrete actions in African countries), within ADEA Working Group activities and through study tours organized for African Ministers and senior Ministry representatives. · Regarding communication, publications and dissemination (para 1.11), many efforts are being made to extend ADEA's reach to education stakeholders. The ADEA Newsletter reaches some 12 000 addresses, in Africa and other regions and readership includes professionals in Ministries of Education, development agencies and organizations, universities, civil society organizations, NGOs and other African and non-African organizations involved in education. ADEA also has a web site containing comprehensive information on ADEA's programs and activities, a Members Only Area to facilitate exchanges between ADEA members and a number of ADEA publications on-line. Media relations ­ which the review does not mention ­ have also developed considerably over the past 6-7 years. ADEA now systematically invites the media to all major ADEA events and meetings with two main objectives in mind: extending its outreach beyond the education community of practitioners and specialists; and contributing to the development of a network of African journalists specialized in education reporting. In 2007 for example, journalists were invited to the following ADEA meetings: the Regional Conference on Secondary Education in Accra; the International Conference on School Fee Abolition on Bamako; the Workshop on Education for Rural People in Africa in Rome and a fringe meeting organized during the ADEA Steering Committee in Brazzaville. · Regarding support to ADEA from African countries (Para 1.15), the phrase "ADEA is funded almost entirely through its 20-22 development agency partners, with marginal support coming from the African countries" should be modulated. Indeed, countries provide significant contributions when meetings are organized in their countries (they appear as in-kind contributions in ADEA reports). Amounts going towards the organization of Biennial Meetings and conferences have been particularly significant, averaging roughly 200 000 US Dollars for the Biennales and 100 000 US Dollars for major conferences. 51 Annex J · Concerning ADEA's financial and budget management (para 1.20), the paragraph should be re-worded and the footnote deleted. In 2005, ADEA's financial situation deteriorated unexpectedly, when IIEP claimed a significant amount of payment arrears from ADEA, based on its 2002­04 accounting. An independent audit found that the claim was unjustified as it was established that ADEA had been debited twice for the payment arrears claimed. The amount claimed has since been returned to ADEA, which thereby recovered its healthy financial condition. · On the impact of ADEA in countries (paras. 3.5 ­ 3.9), while ADEA takes note of the TTL views and the need to track policy impacts more systematically, one needs to be cautious and to take into account two factors: (i) the fact that the impact of ADEA's efforts are long-term and outcomes are often observed years later; and (ii) the difficulty of linking outcomes/results directly to ADEA. A case in point is the impact of the study tour organized in 1999, which invited African Ministers of Education to visit Guatemala and El Salvador. Years later, we are observing the direct impact ADEA has had on country policies and programs: Senegal is implementing a students' government scheme in secondary schools; Niger has implemented a rural education program inspired by Guatemala's Escuela Nueva; Madagascar has put in place school/community contracts. In some cases, outcomes are only observed several years after, as is the case for Senegal, which will only start to apply the student's government scheme as of 2008. · Finally, regarding the balance between restricted and unrestricted funding (para 3.29), since the Review was drafted several agencies have either moved towards unrestricted funding (CIDA, Netherlands and Irish Aid) or have indicated that they would be contributing unrestricted funding starting 2008 (SDC). Overall, we find the IEG Evaluation pertinent and a useful perspective as ADEA is refining its strategies as well as its governance and management instruments and tools. We thank the evaluation team for the review which will be circulated to all ADEA constituencies. Yours Sincerely, Mamadou Ndoye ADEA Executive Secretary The Global Program Review Series The following reviews are available from IEG. Volume #1, Issue #1: ProVention Consortium Issue #2: Medicines for Malaria Venture Issue #3: Development Gateway Foundation Issue #4: Cities Alliance Volume #2, Issue #1: Critical Ecosystem Partnership Fund Issue #2: Association for the Development of Education in Africa Issue #3: Population and Reproductive Health Capacity Building Program Issue #4: International Land Coalition Volume #3, Issue #1: Consultative Group to Assist the Poor RPR Over the course of almost two decades, the Association for the Development of Education in Africa (ADEA) has become the premier forum for educational policy develop- ment and agency cooperation in Africa. Its influence is largely a consequence of its growing legitimacy (it now includes as partners all African ministries of education and all major development agencies active in African education) and its relevance (the focus of its work has evolved along with the region's educational issues). However, such a high level of legiti- macy has had costs in terms of efficiency. These concerns are now being addressed with the formulation of a new strategic plan for 2008­12 and the establishment of an executive board to oversee the operation, thereby allowing its steering committee to focus on more strategic and substantive matters. ADEA's convening power has enabled it to promote policy dialogue and analytical work on African educational problems through a variety of forums, such as at biennial meetings and technical gatherings and in 15 working groups.