Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00005744 IMPLEMENTATION COMPLETION AND RESULTS REPORT Credit No. 5606-ML and Grant No Grant No. D043-ML ON A CREDIT IN THE AMOUNT OF SDR 10.0 MILLION (US$14.0 MILLION EQUIVALENT) AND A GRANT IN THE AMOUNT OF SDR 13.5 MILLION (US$19 MILLION EQUIVALENT) TO THE REPUBLIC OF MALI FOR A HIGHER EDUCATION SUPPORT PROJECT December 30, 2021 Education Global Practice Africa West Region CURRENCY EQUIVALENTS Exchange Rate Effective December 7, 2021 Currency Unit = FCFA FCFA 552.77= US$1.00 US$ 1.39 = SDR 1.00 FISCAL YEAR July 1 - June 30 Regional Vice President: Ousmane Diagana Country Director: Clara Ana Coutinho de Sousa Regional Director: Dena Ringold Practice Manager: Meskerem Mulatu Task Team Leader(s): Jeffrey Waite, Amina Debissa Denboba ICR Main Contributors: Sandra Beemer, Shyam Srinivasan, Setou Mamadou Diarra ABBREVIATIONS AND ACRONYMS ACE Africa Higher Education Centers of Excellence Project AMAQ-SUP Malian Agency for Quality Assurance of Higher Education and Scientific Research AWPs Annual Work Plans BLR Local Radio Loop (Boucle Locale Radio) CA Board of Trustees (Conseil d’Administration) CAMES African Higher Education Council (Conseil Africain et Malgache de l’Enseignement Supérieur) CIAQs Internal Quality Assurance Units COVID-19 Coronavirus Disease CP Council for Improvement (Conseil de Perfectionnement) CPF Country Partnership Framework CPS Educational and Scientific Council (Conseil Pédagogique et Scientifique) DGESRS National Directorate of Higher Education and Scientific Research (Direction Nationale de l’Enseignement Supérieur et de la Recherche Scientifique) DLI Disbursement-Linked Indicator DLR Disbursement-Linked Results ECOWAS Economic Community of West African States ENETP Bamako School for Technical and Vocational and Technical Education ( Ecole Normale d’Enseignement Technique et Professionnel de Bamako) ENI Abderhamane Baba Touré National School of Engineering ( Ecole Nationale d’Ingénieurs) ENSUP Bamako School for Higher Education (Ecole Normale d’Enseignement Supérieur de Bamako) ESIA Environmental and Social Impact Assessment ESMF Environmental Social Management Framework ESMP Environmental and Social Management Plans FA Financing Agreement FM Financial Management GDP Gross Domestic Product GER Gross Enrollment Rate GRM Grievance Redress Mechanism HEI Higher Education Institution HESP Higher Education Support Project ICA Investment Climate Assessments and Surveys ICR Implementation Completion Results Report ICT Information and Computer Technology IDA International Development Agency IDP Institutional Development Plan IPF Investment Project Financing IPRK Institute Polytechnic Rural of Katibougou (Institut Polytechnique Rural de Katibougou) IPR/IFRA Rural Polytechnic Institute for Training and Applied Research ( Institut Polytechnique Rural de Formation et de Recherche Appliquée) ISN Interim Strategy Note i ISR Implementation Status Reports IRI Intermediate Results Indicator IRR Internal Rates of Return MALIREN Mali Research and Educational Network MDG Millennium Development Goals M&E Monitoring and Evaluation MESRS Ministry of Higher Education and Scientific Research MIS Management Information System NER Net Enrollment Rate NICHE Netherlands Post-Secondary Education Capacity Building Project NOC Network Operations Center NPV Net Present Value ONEF National Observatory for Employment and Training (Observatoire National de L’Emploi et de la Formation) PAD Project Appraisal Document PBC Performance-Based Contract PDO Project Development Objective PIU Project Implementation Unit PNES National Higher Education Policy 2021-2030 POM Project Operations Manual PPP Public Private Partnership PPU Planning and Prospective Unit PRODEC National Policy and Strategy for Higher Education Development ( Programme décennal de l’éducation) PSC Project Steering Committee PTC Project Technical Committee QA Quality Assurance RPF Resettlement Policy Framework S&T Science and Technology SCD Systematic Country Diagnostic SIGES Higher Education Management Information System (Système d’information et de gestion de l’enseignement supérieur) SSA Sub-Sahara Africa STEP Systemic Tracking of Exchanges in Procurement TVET Technical and Vocational Education and Training ULSHB Bamako University for Arts and Human Sciences (Université des Lettres et de Sciences Humaines de Bamako) USJPB Bamako University for Legal and Political Sciences (Université des Sciences Juridiques et Politiques de Bamako) USO University of Segou (Université de Ségou) USSGB Bamako University for Social and Management Sciences (Université des Sciences Sociales et de Gestion de Bamako) USTTB Bamako University for Science and Technology (Université des Sciences, Techniques et Technologies de Bamako) WACREN West and Central African Research and Education Network WDR World Development Report ii TABLE OF CONTENTS DATA SHEET ....................................................................... ERROR! BOOKMARK NOT DEFINED. I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 1 A. CONTEXT AT APPRAISAL.................................................................................................................1 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) ............................................. 10 II. OUTCOME .................................................................................................................... 11 A. RELEVANCE OF PDOs .................................................................................................................... 11 B. ACHIEVEMENT OF PDOs (EFFICACY) .............................................................................................. 12 C. EFFICIENCY ................................................................................................................................... 18 D. JUSTIFICATION OF OVERALL OUTCOME RATING ........................................................................... 19 E. OTHER OUTCOMES AND IMPACTS (IF ANY) ................................................................................... 20 Not Applicable. ................................................................................................................................ 21 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 22 A. KEY FACTORS DURING PREPARATION ........................................................................................... 22 B. KEY FACTORS DURING IMPLEMENTATION .................................................................................... 24 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 24 A. QUALITY OF MONITORING AND EVALUATION (M&E) .................................................................... 24 B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE ............................................................ 26 C. BANK PERFORMANCE................................................................................................................... 27 D. RISK TO DEVELOPMENT OUTCOME .............................................................................................. 28 V. LESSONS AND RECOMMENDATIONS ............................................................................. 29 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 31 ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................... 46 ANNEX 3. PROJECT COST BY COMPONENT ........................................................................... 49 ANNEX 4. EFFICIENCY ANALYSIS ........................................................................................... 50 ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ... 61 ANNEX 6. DISBURSEMENT LINKED INDICATORS ................................................................................ 62 ANNEX 7. SUPPORTING DOCUMENTS .................................................................................. 68 iii The World Bank Mali - Higher Education Support Project (P151318) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name P151318 Mali - Higher Education Support Project Country Financing Instrument Mali Investment Project Financing Original EA Category Revised EA Category Partial Assessment (B) Partial Assessment (B) Organizations Borrower Implementing Agency Ministry of Economy and Finances Ministry of Higher Education and Scientific Research Project Development Objective (PDO) Original PDO The development objective of the proposed Mali Higher Education Support Project (HESP) is to improve the relevance of selected higher education programs and the stewardship of the higher education system in Mali.   iv The World Bank Mali - Higher Education Support Project (P151318) FINANCING Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Financing 19,000,000 19,000,000 18,633,814 IDA-D0430 14,000,000 14,000,000 12,797,519 IDA-56060 Total 33,000,000 33,000,000 31,431,333 Non-World Bank Financing 0 0 0 Borrower/Recipient 0 0 0 Total 0 0 0 Total Project Cost 33,000,000 33,000,000 31,431,332 KEY DATES FIN_TABLE_DAT Approval Effectiveness MTR Review Original Closing Actual Closing 07-May-2015 23-Sep-2015 09-Apr-2018 31-Dec-2020 30-Jun-2021 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 02-Dec-2020 28.20 Change in Loan Closing Date(s) KEY RATINGS Outcome Bank Performance M&E Quality Satisfactory Satisfactory Substantial v The World Bank Mali - Higher Education Support Project (P151318) RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 20-Oct-2015 Satisfactory Satisfactory 0 02 28-Apr-2016 Satisfactory Satisfactory 3.60 03 16-Nov-2016 Satisfactory Satisfactory 7.82 04 28-Apr-2017 Satisfactory Satisfactory 8.29 05 28-Dec-2017 Satisfactory Moderately Satisfactory 12.08 06 27-Jun-2018 Satisfactory Moderately Satisfactory 16.82 07 31-Dec-2018 Satisfactory Moderately Satisfactory 18.14 08 19-Jun-2019 Satisfactory Moderately Satisfactory 23.64 09 23-Dec-2019 Satisfactory Moderately Satisfactory 24.80 10 09-May-2020 Satisfactory Moderately Satisfactory 26.29 11 01-Feb-2021 Satisfactory Moderately Satisfactory 30.19 12 27-Apr-2021 Satisfactory Moderately Satisfactory 30.54 SECTORS AND THEMES Sectors Major Sector/Sector (%) Education 100 Public Administration - Education 21 Tertiary Education 60 Workforce Development and Vocational Education 19 Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) vi The World Bank Mali - Higher Education Support Project (P151318) Human Development and Gender 100 Education 100 Access to Education 25 Science and Technology 25 Teachers 25 Standards, Curriculum and Textbooks 25 ADM STAFF Role At Approval At ICR Regional Vice President: Makhtar Diop Ousmane Diagana Country Director: Paul Noumba Um Clara Ana Coutinho De Sousa Director: Claudia Maria Costin Dena Ringold Practice Manager: Peter Nicolas Materu Meskerem Mulatu Jeffrey Waite, Amina Debissa Task Team Leader(s): Pierre Joseph Kamano Denboba ICR Contributing Author: Sandra F. Beemer vii The World Bank Mali - Higher Education Support Project (P151318) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES A. CONTEXT AT APPRAISAL Context 1. Country Context. At the time of appraisal, Mali was in the process of stabilization following the 2012 political and security crisis. Insecurity in the north of Mali led to a rebel attack in December 2011, whereby the three main regions of the north fell under rebel control. This was followed by a coup d’état in March 2012 and the establishment of a de facto military government. These developments led to the suspension of support from major donors to the country, with negative consequences for the economy. In 2013, with support from the international community, the country was able to return to relative normalcy with a new President and National Assembly democratically elected. In May 2013, the international community also provided unprecedented funding support for the Programme de Relance Economique et de Développement amounting to Euro 3.2 billion. Despite strong support from the international community, the security context was challenging with regions in the north remaining highly unsafe, particularly in rural areas. Peace negotiations resumed in February 2015 and an agreement was signed on March 1, 2015. However, the agreement only included the Government and some of the rebel groups. At this time, there was a significant need for sustained support to promote capacity-building and strengthening of the national institutions given the fragile country context. Providing more economic opportunities for young Malians and greater institutional capacity building remained urgent. 2. In 2015, Mali as a landlocked country, had a fast-growing population of approximately 14.9 million people. While the country experienced sustained economic growth between 2006 and 2011 (averaging five percent per annum), Mali’s gross domestic product (GDP) declined during the crisis. GDP rebounded to the point that by 2013 GDP per capita had been restored to its 2011 levels (US$667). Mali’s economic growth was heavily reliant on the country’s natural resource endowment base and a number of key economic sectors. The sectors that contributed the most to Mali’s GDP were: agriculture (21.9 percent); trade – notably in the informal sector (16.6 percent); livestock (9.7 percent); and mining (largely gold—5.7 percent). These sectors employed roughly 80 percent of the labor force, with the skilled workforce coming predominantly from neighboring countries. The most important sectors in terms of their contribution to GDP (i.e., agriculture and livestock) were primarily concentrated in rural areas where the large majority of the population was poor and vulnerable. 3. While the political and security crisis severely impacted the economy, there was also a substantial deterioration in the delivery of social services, and a contingent increase in the country’s vulnerability to exogenous shocks (security crisis, climate change, pests, and commodity prices). Mali’s economic fragility was further exacerbated by structurally entrenched obstacles to growth, including a poor investment climate, limited and poorly maintained infrastructure, and limited and inadequate skills, particularly among young workers. A critical challenge entailed addressing the chronically low qualifications of the Malian labor force in the country’s emerging formal labor market. The majority of Malians (65 percent) had no education and the average years of schooling amongst adults was 2.4 years. The status quo did not adequately address the needs of existing employers in the formal sector who historically complained of persistent difficulties in recruiting appropriately skilled workers, and this in turn, also undermined the county’s potential for growth and economic diversification, while also limiting poverty reduction. The skills shortage was demonstrated by an increase in the demand for specialized human capital from rapid growth sectors (extractive industries, energy, water, infrastructure, and in service sectors) and the concurrent growth in vacancies. Low levels of labor productivity also undermined the economic performance of the informal sector. The mismatch between the level of skills demanded by Page 1 of 68 The World Bank Mali - Higher Education Support Project (P151318) the labor market and the skills of workers seeking employment was further exacerbated by the crisis, which in turn, led many skilled workers (who could compete in the sub-regional market) to leave the country. 4. The recognition of the skills gap led the Government to conclude that without an ambitious skills development program that imparts entrepreneurial and labor skills commensurate with the requirements of a dynamic economy, it would not be able to implement an economic diversification strategy, compete in a globalized regional and world economy or escape the vicious cycle of drought, famine, insecurity, and population displacement which led to poverty and inequity. The Government’s 2009 National Policy and Strategy for Higher Education Development, and the 2011 National Policy on Technical and Vocational Training, were developed in response to these concerns. It was on this basis that the Government was implementing the World Bank-supported Skills Development and Youth Employment Project (P145861) which aimed at improving youth employability through education and training by strengthening public and private secondary technical and vocational education and training (TVET) institutions in key sectors. Similarly, the Government felt efforts needed to be undertaken at the tertiary level to complete the skills value-chain necessary for the upgrading of the economy. 5. Sector Context. Between 2004 and 2010, Mali had made significant progress in basic education, increasing the primary gross enrollment rate (GER) from 72 in to 82 percent, primarily through reforms initiated to fulfill the Millennium Development Goals (MDGs) agenda. Access to secondary and tertiary education also increased from 12 to 34 percent and from 4 to 5 percent, respectively, over the same period. Further progress was hampered by the negative effects of the crisis for all social sectors. In primary education, the GER declined to 69 percent in 2013, while the primary completion rate decreased from 58 percent in 2010 to 48 percent in 2013. Equity in access to secondary and tertiary education remained a challenge, with the upper income quintiles of the population being the largest beneficiaries of secondary and tertiary education. There was also low quality and relevance of education throughout the system. This was demonstrated by the results of a 2011 national student assessment which found that less than 20 percent of students who had completed six grades of primary school had the requisite skills to actively participate in the labor market. The assessment also highlighted poor labor market outcomes for upper secondary and post-secondary education graduates, among whom unemployment rates were close to 30 percent. Moreover, Investment Climate Assessments and Surveys (ICAs) had consistently found that formal sector employers were unable to recruit workers with the appropriate mix of skills, further buttressing the view of a mismatch between education and training outcomes and the needs of the formal sector and the economy at large. 6. These challenges were, in large part, the consequence of poor teaching practices, and inadequate learning environments. A 2014 Report on the Future of Higher Education in Mali found that these challenges were particularly acute in the tertiary education. This led to the failure of the sub-sector to fulfill its role as a source of high-level skills, with negative knock-on effects that limited the potential for economic diversification and improved productivity and growth. The study found that only 53 percent of 25 to 34-year-old workers with a tertiary education were employed due to low levels of knowledge and poor skills. The share of science and engineering students as a proportion of tertiary enrollment was only 3.8 percent in 2011, the lowest among 60 countries surveyed in Sub-Saharan Africa (SSA), Middle East and North Africa, East Asia Pacific, South Asia, and Latin America and the Caribbean. Despite the important contribution of the agricultural sector to GDP, Mali ranked 11th lowest for enrollment in tertiary qualifications pertaining to agriculture. In contrast, at almost 80 percent, Mali ranked highest in the share of students enrolled in humanities and social sciences programs. 7. The higher education system faced many challenges which were interconnected and required urgent action. The tertiary education system lacked sufficient institutional diversity, and a mix of program and degree specialization. Five Page 2 of 68 The World Bank Mali - Higher Education Support Project (P151318) public universities (four in Bamako and one in Segou – University of Segou (UOS)) and four technical institutes and grandes écoles, accounted for more than 95 percent of total enrollment at the tertiary level with the overwhelming majority of students concentrated in Bamako. More than two-thirds of enrollment was concentrated in the law and humanities faculties. There were insufficient opportunities for graduates to obtain technical and professional qualifications in a short period of time. Increasing the breadth and scope of programs and degree offerings, with a particular focus on Science and Technology (S&T) qualifications, while concurrently encouraging the expansion and capacitation of private higher education institutions (HEIs), would enable the higher and tertiary education sub-sector to more effectively respond to challenges related to relevance and quality, and more effectively meet the needs of the economy. 8. The tertiary education system’s institutional and program licensing process constrained quality and relevance of courses. The HEI accreditation process, which used external resources such as the African Higher Education Council (Conseil Africain et Malgache de l’Enseignement Supérieur – CAMES) to deliver accreditation of institutions which were valid for few years and renewable, was essentially not utilized. Moreover, the licensing system established by a 2006 Decree1 needed to be revamped to ensure a more independent evaluation and improve the quality of targeted academic programs. Moreover, pedagogical inputs and innovation were limited. There was inadequate emphasis on pedagogical inputs such as books and scientific documentation, information, and computer technology (ICT), laboratory equipment and supplies, research activities, staff recruitment and development, and investment in infrastructure. The situation was compounded by uncontrolled admissions which, adding to the high repetition rates during the two first years of higher education, resulted in the overcrowding of some universities. The student-teacher ratio was, on average, very high and there was no comprehensive staff development plan to meet the teaching needs and quality requirements. 9. The higher education sector was undermined by systemic and institutional gaps in governance. In 2013, the bulk of the sector’s budget was allocated to students’ welfare and extra teaching hours with the scholarship system accounting for 56 percent (FCFA18.5 billion, equivalent to US$40 million) of the total higher education budget. Improving the structure of financing for higher education would enable HEIs to more efficiently allocate their own budgets in support of improved pedagogical inputs and innovation. Moreover, the sector was negatively impacted by an absence of performance standards for HEIs, low levels of accountability for results, and limited institutional autonomy although reforms to rectify this were underway. Recruitment of teaching staff was conducted with limited coordination, with the Ministry of Civil Service organizing recruitment, while the Ministry of Higher Education and Scientific Research (MESRS) determined the academic schedule and admissions criteria for students without taking into account availability of teaching staff and resource limitations. This situation resulted in recurrent and long-lasting teacher and student strikes, reducing the academic year and negatively affecting learning outcomes. Improving institutional autonomy while increasing accountability would enable HEIs to professionalize their management and create the conditions for the introduction of alternative sources of revenue (including, for example, introducing student fees). The higher education sub-sector system also suffered from the absence of a comprehensive regulatory and financing framework to support the development of private HEIs and university-industry linkages. 1The 2006 Decree created an independent national body – comité d’habilitation – under the coordination of the National Directorate of Higher Education and Scientific Research ( Direction Nationale de l’Enseignement Supérieur et de la Recherche Scientifique-DGESRS). The comité d’habilitation members were appointed by a ministerial decree signed by the Minister of Higher Education which analyzed the information provided by the requesting HEI and then made a recommendation to the Minister. Page 3 of 68 The World Bank Mali - Higher Education Support Project (P151318) Theory of Change (Results Chain) 10. The Project’s results chain is presented in Figure 1. It illustrates the key inputs, activities, outputs, and outcomes by project component. The Mali Higher Education Support Project (HESP) supported the realization that the Government needed assistance in its efforts to stabilize the fragile political, economic, and social situation following the 2013 political and security crisis in the country by creating opportunities for vulnerable youth and supporting tertiary level education to complete the skills value chain needed for upgrading the economy. The long-term goal of HESP was to contribute to the achievement of transformational growth through economic diversification by increasing the ability of the tertiary education sector to supply high-quality graduates with skills matched to the needs of labor market. To eventually realize this long-term goal, the first step was to address the poor supply and quality of skilled labor limiting growth opportunities in areas that required high levels of technical expertise and education (e.g., mining, agriculture, livestock management and ICT). By addressing skills-related constraints in agriculture, livestock, and mining (in predominantly rural activities) the Project aimed to positively impact labor productivity and enhance the incomes of the poor, with positive implications for poverty reduction more generally. The project development objective (PDO) and supporting project activities were based on the 2011 World Development Report (WDR)2 and internationally recognized research which confirmed the critical role of value chain programs particularly related to skills development. 2 World Bank, 2011, “World Development Report 2011: Conflict, Security and Development”, World Bank, Washington, DC. Page 4 of 68 The World Bank Mali - Higher Education Support Project (P151318) Figure 1: Results Chain T0 Component Activities/Inputs Outputs Outcomes Sub-component 1.1 Support to selected  Increased number of Improve relevance of existing public HEIs national students selected higher  Support for four (4) institutions to enrolled in licensed education programs in implement Institutional Development programs at Mali Plans (IDPs) and annual work plans participating HEIs Measured as: (AWPs) (DLIs (disaggregated by  Employer Sub-component 1.2 Support to the number of females) satisfaction rate (in University of Segou’s (UOS) livestock  Increased share of the three priority department (first Mali public regional budget in participating sectors) university) HEIs allocated to  Increased number of Component 1:  Support for development of a new pedagogical inputs programs in Support to livestock and animal health department  Increase in amount of participating HEIs Selected at UOS through civil works, equipment cash generated that are proposed Public HEIs goods and technical assistance to internally at and developed with establish a modern facility. participating HEIs private sector  Provision of transitory job opportunities.  Annual report on DLIs participation Sub-component 1.3 Support to the produced on time  Tracer studies (for PDO: to improve development of the new Bamako Africa  Total number of graduates from the the relevance of School of Mines students benefiting selected HEIs) selected higher  Support for development of a business from direct conducted education model for Bamako Africa School of interventions to  DLI - Improved programs and Mines to incorporate a public private enhance learning Relevance of the stewardship partnership (PPP) framework. (disaggregated by Selected Programs of the higher females) education Support for government to Ministry of  Increased number of system in Mali. Improve stewardship Higher Education (MoHE) improve public HEIs with a of higher education stewardship of higher education system management system in Mali through: information system Measured as:  Conducting tracer studies with operational  Number of  Conducting employer satisfactory resource management institutional surveys  Increased number of development (IDPs) Component 2:  Implementation of a Bachelor-Masters- quality assurance unit plans approved at Higher Doctorate (License-Masters-Doctorat- established and public HEIs Education LMD) system operational, with semi-  Number of public System  Support for a higher education autonomous status HEIs with a Support information system (SIGES) that  Increased number of functional Conseil included a virtual library, on-line courses users accessing digital d’Administration or  Support for studies on development of library system Conseil d’Université the HEI private sector  MIS statistical reports (CA) (Board of  Support for a quality assurance system published annually with Trustees) relevant data from #  DLI - Improved HEIs (Public & Private) Institutional Governance  Support for project management  Number of public HEIs Component 3: Project  technical assistance for all project with proposal(s) for components acquiring support from Management  Exchange programs with international Africa Centers of and Technical HEIs Excellence in West and Assistance Central Africa. Page 5 of 68 The World Bank Mali - Higher Education Support Project (P151318) Project Development Objectives (PDOs) 11. The Project’s PDO was to improve the relevance of selected higher education programs and the stewardship of the higher education system in Mail. The PDO statement in the Project Appraisal Document (PAD) and Financing Agreement (FA) were consistent. Key Expected Outcomes and Outcome Indicators 12. The six PDO-level indicators were directly linked to the overarching objectives and were supplemented by 11 intermediate results indicators (IRIs). There were also two Disbursement Linked Indicators (DLIs) and seven Disbursement Linked Results (DLRs). PDO-level indicators 1-4 corresponded directly to Outcome 1 (relevance of selected higher education programs) and PDO level indicators 5-6 corresponded directly to Outcome 2 (stewardships of the higher education system in Mali). • Outcome 1 sought to increase employer satisfaction as measured by an employer satisfaction survey in priority sectors in participating HEIs (in the three priority sectors3; the number of programs in participating HEIs4 that are proposed and developed with private sector participation5; tracer studies (for graduates from the selected HEIs) conducted and number of direct project beneficiaries6. DIL 2— Improved relevance of selected programs was related to Outcome 1. • Outcome 2 was to measure stewardships of the higher education system in Mali through the number of institutional development plans (IDPs) approved at public HEIs; and the number of public HEIs with a functional7 Conseil d’Administration (CA) (Board of Trustees). DLI 1—improved institutional governance was associated with Outcome 2. 3 The three priority sectors were agriculture, livestock, and mining. 4 The participating HEIs were: The Institute Polytechnic Rural of Katibougou ( Institut Polytechnique Rural de Katibougou (IPRK) focused on agriculture, research and outreach; the University of Segou (Université de Ségou-UOS) focused on livestock, and agriculture programs with a component on water management and irrigation; the Bamako University of Science and Technology (Université des Sciences, Techniques et Technologies de Bamako -USTTB) provided training in applied sciences to support other sectors; and the Abderhamane Baba Touré National School of Engineering (Ecole Nationale d’Ingénieurs-ENI) dealt with engineering programs including mining. 5 These programs, defined as “Licensed programs”, refer to programs approved by the Licensing Commission or Quality Assurance unit of the MESRS following a request of habilitation by the HEI. They had to be previously approved by the Board of Trustees of the HEI based on the Conseil de Perfectionnement (CP) recommendation which comprised 50 percent of members of the private sector specializing in the area of the program. The licensing process was strengthened under Component 2 of the Project. Licensed programs included both existing programs to be newly licensed and new programs. 6 Direct project beneficiaries were defined as the number of students benefitting from the project in the four selected HEIs and other HEIs also benefitting from the Project. 7 A functional Board of Trustees was defined as having at least 30 percent of non-civil servants from sectors relevant to programs developed at the HEI, which met at least twice a year to deliberate on the institution key aspects, such as but not limited to the budget, appointment of key staff, as well as the rules and regulations of the institution. Page 6 of 68 The World Bank Mali - Higher Education Support Project (P151318) 13. As indicated, the Project also had results areas with a set of DLIs and DLRs, which included a balanced mix of activities, outputs, and outcomes. The DLIs and DLRs were only implemented within the four participating HEIs (see footnote 4). (See Annex 6 for full details on DLI disbursement rules, DLI achievement measures and DLI verification process.) The results areas were: (i) DLI 1: Improved institutional governance (in the participating HEIs) • DLR 1: Functional Board of Trustees (or University Council) of HEIs in place. • DLR 2: Adoption and Implementation of management tools and procedures of HEIs. • DLR 3: Approval of HEIs IDPs and yearly operational budget with indicators. (ii) DLI 2: Improved relevance of selected programs • DLR 4: Number of training programs licensed as per agreed guidelines. • DLR 5: Number of new students enrolled in licensed programs developed with the private sector and approved by the Board of Trustees. • DLR 6: Number of teachers trained in the licensed courses approved by the Board of Trustees. • DLR 7: Percentage increase in financial resources generated internally. Components 14. Component 1: Support to Selected Public HEIs (original allocation-US$22 million equivalent; actual US$19.58 million equivalent). Component 1 sought to improve the relevance of selected higher education programs, while also contributing to improving stewardship by supporting governance improvements and quality related interventions in selected existing HEIs which had demonstrated a strong commitment to reform, and which supported new course offering programs in agriculture, livestock, mining and technology. It was comprised of three sub-components: Component 1.1: Support to selected existing public HEIs; Component 1.2: Support to the University of Segou’s new livestock and animal health department; and Component 1.3: Support to the development of the new Bamako Africa School of Mines. 15. Sub-Component 1.1: Support to selected existing public HEIs. This sub-component supported four selected existing institutions: (i) the Institute Polytechnic Rural of Katibougou (Institut Polytechnique Rural de Katibougou-IPRK) that focused on agriculture, research and outreach; (ii) the University of Segou ( Université de Ségou-UOS) that focused on livestock, and agriculture programs with a component on water management and irrigation; (iii) the University of Science and Technology (Université des Sciences, Techniques et Technologies de Bamako-USTTB) which provided training in applied sciences to support other sectors; and (iv) the National School of Engineering (Ecole Nationale d’Ingénieurs- ENI) which focused on engineering programs including mining. The Project supported the implementation of each of the selected HEI’s five-year (2015-2020) Institutional Development Plans (IDPs)8 and Annual Work Plans (AWPs) to be approved by each HEI’s Board of Trustees (Conseil d’Administration – CA) prior to project effectiveness. The AWP served as a basis for contracts to be signed by the rector of the HEI, the Minister of Higher Education and the Minister of the Ministry of Economy and Finance (MEF), outlining agreed upon annual objectives and indicators for each HEI. The AWP 8 The IDPs were to be focused on achieving the HEI’s development objectives of improving the quality and relevance of programs, and the governance and management of the institution. The IDPs could include, but were not limited to, provision of books and scientific documentation, ICT, laboratory equipment and supplies, research activities, staff recruitment and development, and investment in infrastructures. Page 7 of 68 The World Bank Mali - Higher Education Support Project (P151318) included all activities and funds (including Government and external funds allocated to the institution), with clear roles and responsibilities, and mechanisms for monitoring and evaluation (M&E) to assess the HEI’s yearly performance. The annual report on the AWP and annual audit report were to be used to complement the report of the Quality Assurance (QA) unit9 on HEI performance, which in turn was used for the monitoring of DLIs. There were two DLIs with seven DLRs, related to governance, relevance, and quality dimensions, with relevant targets for each selected HEI. These DLIs were linked to the performance-based contracts that were signed between the four project-supported HEIs that included the DLRs to be achieved. 16. Sub-Component 1.2: Support to the University of Segou’s livestock department . Unlike the three other HEIs that benefitted from the HESP (which were already established and had facilities that will require upgrading), the UOS, the first public regional university in Mali, was still in the process of development and required greater support for the delivery of programs. Therefore, this sub-component specifically supported the development of a new livestock and animal health department at UOS through the construction and equipping of new teaching facilities. The Project was to finance the civil works, equipment, goods, and TA required to set up a modern functional facility. 17. Sub-Component 1.3: Support to the development of the new Bamako Africa School of Mines . The Government had established the Bamako -Africa School of Mines to more effectively address the intermediate and higher skills gaps inhibiting the further development of the country’s mining sector. Bamako was also exploring opportunities for the participation of the private sector in the establishment of the school and had already had some interest from mining companies in supporting the school. Based on this, the Government requested Bank support in the development of a business model for the new school incorporating a Public Private Partnership (PPP) framework. To achieve this objective, this sub-component supported a feasibility study which covered financing options, architectural plans, curriculum and program development, and equipment specifications for the new proposed school. 18. Component 2: Higher Education System Support (original allocation-US$8 million equivalent; actual US$8.67 million equivalent). Component 2 aimed to improve the stewardship of the higher education system in Mali by strengthening the Government’s capacity to articulate and implement reforms; play an overall oversight and regulatory role, including for the private sector; and manage information. To achieve this the goal, this component supported:  Upgrading of the Directorate-General for Higher Education and Scientific Research ( Direction Générale de l’Énseignement supérieur et de la Recherche Scientifique—DGESRS). The Project financed the building of new premises, and the procurement of new office furniture, IT equipment and the provision of staff training. The Project was to directly contract with the ENI and the International School of Architecture (Ecole Supérieure Internationale d’Architecture et d’Urbanisme) to design the DGESRS new building and the new animal health faculty in UOS, as well as for oversight of the civil work associated with UOS construction.  Support to the Planning and Prospective Unit (PPU) and the development of the sub-sector plan . The Project financed the procurement of equipment, furniture, relevant consultants, and the training of staff from the PPU and HEIs to enable the production of annual statistics indicators with regard to enrollment, graduates, staff, facilities, etc. in coordination with the management information system (MIS), as well as the tracer studies. The PPU was expected to assist HEIs not participating under Component 1 in the development and monitoring of their respective five-year IDPs. The Netherlands Post-Secondary Education Capacity Building Project (NICHE) project was to finance the development of a template and manual of 9 The QA unit was under the National Directorate of Higher Education and Scientific Research (Direction Nationale de l’Enseignement Supérieur et de la Recherche Scientifique—DGESRS). Page 8 of 68 The World Bank Mali - Higher Education Support Project (P151318) procedures for HEIs, business plans, prepare an implementation plan and train focal points in each HEI. The HESP was to finance additional TA as needed, background and tracer studies, and consultant services and workshops for the preparation of the sector medium term development plan and annual reports.  Support to the Quality Assurance Unit. The Project financed consultant services and workshops to promote and facilitate the implementation of QA in HEIs, assist selected academic units to undertake self- assessments, and guide the implementation of the Bachelor - Masters - Doctorate System (3-5-7) (LMD) system. The Project also supported the procurement of equipment and furniture for the QA Unit.  Management and Information System. The Project financed activities in relation to the Higher Education MIS system (Système d’information et de gestion de l’enseignement supérieur—SIGES) as an integral part of the West and Central African Research and Education Network (WACREN): The Project complemented existing interventions through the financing of TA premised on enhancing the design of the MIS and upgrading software, based on the specifications outlined in a 2011 preliminary study (updated to take into account needs identified by NICHE). This included: (i) the development of a standard MIS for HEIs (accounting, budgeting, academic and students’ management, facilities management, etc.) and a central database at the MESRS level; (ii) the implementation of the SIGES at institutional level which included the procurement of IT equipment, consultancy services and training of staff for the deployment of the system in HEIs under the leadership of the MESRS; and (iii) the implementation of the SIGES at the central level included the financing of IT equipment, consultant services and training for the implementation of the SIGES at the MESRS level.  Virtual library and upgrading libraries. The UNESCO ICT Support Project (Projet d'Appui à la mise en place de Technologies d'Information et de Communication—PACTICE) financed the virtual library. However, the HESP financed the acquisition of eBooks and complementary equipment for HEIs. The Project also financed the development of three videoconferencing rooms (two in the center which hosts the Resource Center (Centre de Resources), and one in the new DGESRS).  Development of Private Sector Providers. The Project also financed several studies and related consultancy services to support the development and expansion of private HEIs. This included, inter alia: a study of factors constraining private HEI development; an analysis of the status and challenges to the provision of private higher education in Mali; recommendations to enhance the contribution of the private sector; and an investigation of financing alternatives for private HEIs with local banks (in collaboration with the IFC), as well as a study on student loans. The unit responsible for private HEIs within the Ministry were to be strengthened through TA and the upgrading of equipment to better articulate the Government strategy for higher education and private HEIs. 19. Component 3: Project Management and Technical Assistance (original allocation-US$3.0 million equivalent; actual US$3.11 million equivalent). Component 3 funded activities with project management, and associated TA, with regard to fiduciary issues, as well as studies and analytical work. The studies financed under this component were to contribute to the improvement of M&E in the higher education sector as well the design of a new sector plan integrating a financing system that incentivized a more competitive higher education system. Financed studies included: background studies towards the development of the sector program (financed under Component 2); annual higher education reports; a study of potential financing mechanisms; and an assessment of the implementation of new programs, including those developed with the support of Economic Community of West African States (ECOWAS). Component 3 also financed exchange programs for representatives of public HEIs with internationally renowned institutions of higher learning, visiting teaching missions from the Malian diaspora, and innovation to promote learning outcomes, academic excellence and responsiveness to market demand. The TA was used to support the provision of higher education services Page 9 of 68 The World Bank Mali - Higher Education Support Project (P151318) to institutions not supported in Component 1 from the emerging 19 African Centers of Excellence (ACE) in West and Central Africa. These Centers of Excellence, focused on science, technology, engineering, mathematics, health, and agricultural sciences are beneficiaries of a World Bank-supported regional project. 10 Malian institutions will define their needs and competitively purchase support packages that may include short-term specialized training, longer-term faculty development, the sponsoring of visiting faculty, and curriculum design and development. The PIU will be responsible for the daily management, implementation, administration, project coordination, and M&E of the project. B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) Revised PDOs and Outcome Targets 20. The Project’s PDOs were not revised during the project period. Revised PDO Indicators 21. The Project’s PDO indicators were not revised during the project period. Revised Components 22. The Project’s components were not revised during the project period. Other Changes 23. Changes made at the time of the November 2020 Level 2 restructuring. A Level 2 restructuring modified the closing date of the Credit and Grant from December 31, 2020 to June 30, 2021. The closing date was extended due to delays caused by the COVID-19 pandemic and the August 18, 2020 coup d’état. The World Bank’s rules obliged it to cease project financing beyond minimal levels to allow project units to function while it evaluated the de facto Government’s position relative to prior commitments and its international and domestic legitimacy. Following the completion of the process, disbursements resumed on November 9, 2020. This pause in disbursements slowed the Project’s overall implementation progress. As a result of implementation delays resulting from the COVID-19 pandemic and the country's recent regime change, the Government requested—and the World Bank approved—a six-month extension of the Project's closing date from December 31, 2020 to June 30, 2021. 10The Africa Higher Education Centers of Excellence (ACE) Project (P126974) seeks to meet labor market demands for skills in specific fields. This multi-phased program has provided significant financial and technical support to strengthen higher education in Africa, focusing on improved quality and relevance of education and training programs, improved teaching, and learning, and expanded post-graduate access in key priority fields. Its first phase, ACE I, introduced the first-ever regional higher education competition, which selected universities in West and Central Africa based on their potential for education and research excellence, as well as strong governance and management. Page 10 of 68 The World Bank Mali - Higher Education Support Project (P151318) Rationale for Changes and Their Implication on the Original Theory of Change 24. The modifications did not affect the original theory of change, but rather allowed for the completion of a small number of activities that were already programmed but could not be completed due to the COVID-19 pandemic and political events as indicated. II. OUTCOME A. RELEVANCE OF PDOs Assessment of Relevance of PDOs and Rating 25. The PDO was Highly relevant to the country’s sectoral needs when the Project was developed. The objectives fit into the Government’s 2009 National Policy and Strategy for Higher Education Development, its 2003-2013 ten-year plan (Programme décennal de l’éducation – PRODEC) and an Interim Program for 2014-2016—which focused on laying the foundations for improving education sector quality, including higher education. The National Policy on Higher Education and Scientific Research in 2010, which was informed by a 2008 National Forum on Education, identified the need for systemic change in higher education. A 2011 national student assessment highlighted poor labor market outcomes, and similarly, ICAs highlighted mismatches between education and training outcomes and labor market needs. The Project was also consistent with the World Bank’s May 2013 Interim Strategy Note (ISN) for Mali which focused on long term economic growth, building human capital and resilience to shocks, and highlighted the need to improve governance, and boost economic competitiveness by addressing skills gaps. The Project was also consistent with the 2018 Systematic Country Diagnostic (SCD) (P159672), which was still being written, given the latter’s focus on rural areas where most of the poor lived, and highlighted major skills gaps in agriculture. There was an urgent need to address unemployment through the development of relevant skills among youth. Investments in education were also expected to help build confidence in the education system. A 2014 Report on the Future of Higher Education in Mali found acute challenges in the tertiary education sub-sector including: (i) a lack of diversity and low relevance of degree programs (low share of science, engineering, and agriculture graduates); (ii) poor teaching practices; (iii) inadequate learning environments; (iv) pervasive governance gaps (low accountability and limited institutional autonomy); and (v) inefficient resource utilization. The PDO and project design were developed based on these analyses and aimed to address shortcomings in the sector, by strengthening governance of the sector, elevating the quality and relevance of programs offered by selected HEIs, improving teaching quality by financing a series of critical inputs such as infrastructure and teaching and learning materials. The Project was relevant to the World Bank’s FY 2016-2019 Country Partnership Framework (CPF) 11 (P151414) by supporting two of the focus areas of improving governance at the central and local levels and building resilience, by developing human capital. Finally, the Project was consistent with the World Bank’s twin goals of eliminating extreme poverty and promoting shared prosperity. 26. The development objectives continue to be Highly relevant to the Government’s strategic objectives and plans for the country. The Project continues to be highly relevant—and strongly aligned with—the objectives outlined in the new National Higher Education Policy (PNES) 2021-2030. Specifically, the new PNES aims to build on the foundation laid by the HESP by continuing to focus on: (i) improving the quality of programs to meet the labor market needs; (ii) improving the quality of the faculty and/or teaching staff; (iii) increasing the number of teachers and faculty to address the shortage of 11 World Bank. 2015. Mali - CPF for the period FY16-19. Report Number - 94005. Page 11 of 68 The World Bank Mali - Higher Education Support Project (P151318) teaching staff; (iv) increasing the enrollment of students; (v) strengthening the integration of higher education and scientific research; and (v) improving governance within the system. The Project also fits into the Strategic Plan for Higher Education (2021-2026)12 which is the five-year plan to operationalize the PNES over the next five years. B. ACHIEVEMENT OF PDOs (EFFICACY) Assessment of Achievement of Each Objective/Outcome 27. The overall efficacy of the Project is rated Substantial. As indicated, the Level 2 restructuring only modified the closing date of the Credit and Grant, therefore a split evaluation is not warranted for this Project. Six of the six PDO indicators were met or exceeded and the 10 of the 11 intermediate results indicators (IRIs) were met or exceeded. The Project also included two DLIs with associated DLRs which were all achieved and fully disbursed. The following assessment of the PDOs is organized around two key outcomes: (i) improved relevance of selected higher education programs; and (ii) improved stewardship of the higher education system in Mali. 28. Outcome 1: Improve the relevance of selected higher education programs. Achievement of this envisaged outcome is judged to be Substantial and was achieved through a combination of interventions that supported HEIs in Mali to produce high quality graduates with skills aligned to the labor market. Outcome 1 was measured by the following indicators: (i) increase in employer satisfaction as measured by employer satisfaction survey in priority sectors in participating HEIs (in the three priority sectors (PDO level indicator); (ii) increase in the number of programs in participating HEIs that are proposed and developed with private sector participation (PDO level indicator); (iii) tracer studies (for graduates from the selected HEIs) conducted (PDO level indicator); and (iv) direct project beneficiaries (PDO level indicator). The intermediate level indicators (IRIs) were: (a) number of national students enrolled in licensed programs at participating HEIs (disaggregated by number of females); (b) number of teachers trained in the licensed programs at participating HEIs; (c) share of budget in participating HEIs allocated to pedagogical inputs; (d) increase in the amount of cash generate internally at participating HEIs; and (e) number of public HEIs with proposal(s) for acquiring support from Africa Centers of Excellence in West Africa and Central Africa Project -P126974).13 29. Disbursement Linked Indicators. As indicated, DLI 2: Improved relevance of selected programs and the four associated DLRs were achieved in the four participating HEIs-- IPRK; UOS, USTTB and ENI. DLR 5, DLR6 and DLR7 were the same as the IRIs reported below. DLR 4—number of training programs licensed as per agreed guidelines—increased from a total of 45 programs in year 0 to 197 by the end of June 2020, thus exceeding the original total target of 135. As indicated, all four participating universities met their annual targets, and the funds were disbursed for each DLR. (See below for other achievements.) 30. The results of the PDO indicator—employer satisfaction as measured by employer satisfaction survey in priority sectors in participating HEIs (in the three priority sectors) increased from the 2015 baseline of 0 to 87.3 percent, thus exceeding the 2021 target of 40 percent by 47.3 percentage points. The employer satisfaction survey was conducted in 12 This strategic plan has five strategic axes: (i) strengthening the quality, effectiveness and efficiency of governance; (ii) optimizing access to higher education in an equitable and inclusive manner; (iii) improving the quality of training and learning environments; (iv) increasing the employability of graduates; and (v) deploying the ITSP by strengthening links with IESR. 13 This IRI is related to improving the relevance of higher education programs because HEIs in Mali could receive HESP funds to support technical assistance for the development of collaborative projects with the ACE Project—P126974 in West and Central Africa. Page 12 of 68 The World Bank Mali - Higher Education Support Project (P151318) 2019 as part of the project tracer study14 since the first cohort of beneficiary students completed their Bachelors ( License) at the end of academic year 2017-2018. The survey showed a high employer satisfaction with the graduates from the Project supported HEIs in the three priority sectors. The results for the PDO indicator— number of programs in participating HEIs that are proposed and developed with private sector participation—increased from the 2015 baseline of 45 to 197, thus exceeding the 2021 target of 135 (a 46 percent increase over the target). These programs referred to ‘licensed’ programs that were approved by the QA unit within the MESRS based on a request of the HEI to the QA unit to review. It should be noted that before the programs were reviewed, they were first recommended by the universities; approved by a council in-charge of reviewing programs on offer (Conseil de Perfectionnement), which included a mix of professors and private sector representatives from sectors targeted by the university; and/or the committee responsible for reviewing the content from scientific and pedagogical perspectives (Conseil Pédagogique et Scientifique). The programs were then approved by the Board of Trustees (Conseil d’Administration) which again comprised private sector members (30 percent) from relevant sectors. This organizational structure was a factor in ensuring strong private sector involvement in the development of relevant ‘licensed’ programs. In addition, the HEIs have signed partnership agreements with industry associations and private sector companies, to complement the training of their students through apprenticeships or internships. This also allows the HEIs to tap on the private sector for refining their programs and improving instruction. 31. The PDO indicator—tracer studies (for graduates from the selected HEIs) conducted (yes/no) was met. During the project period there were three tracer studies15 conducted for graduates from the selected HEIs, thereby meeting the target for the indicator. The results of the 2019 tracer study 16 made it possible to identify opinions of teaching staff, locate the graduates’ of HEIs and identify the companies in which the graduates were employed. As indicated above, the survey showed a high employer satisfaction with the graduates from the three priority sectors. In addition, the survey showed that there were some formal partnerships between companies and HEIs but that they were limited and still in an embryonic stage. A summary of findings specifically related to the HESP HEIs are that: (i) administrators in the HESP HEIs were satisfied with the quality of the professionally licensed teaching staff upon completion of their training; (ii) the staff of the HESP universities would like to see a greater focus on practical skills with 60 percent of the volume of hours devoted to practical skills; (iii) the large majority of graduates who benefitted from internships at the time of training were enrolled in HESP supported universities and that the internships were offered largely based on personal relationships; (iv) the insertion rate of HESP institution graduates was 6 out of 10 graduates (about the same as non-HESP HEIs) and the percentage of HESP graduates working in their primary sector is higher than non-HESP graduates; and (v) 6 out of 10 jobs of HESP graduates are formal jobs. It should be noted that the overall study results for both HESP HEIs and non-HESP graduates showed that employability was problematic with 30 percent of graduates persistently unemployed, 28 percent in paid employment, 4 percent on a paid internship, 15 percent self-employed, and 23 percent continuing training. The overall results of the survey have been used by the MESRS in the preparation of the Strategic Plan for Higher Education 14 Expanded Study on Vocation Integration and Employability of Graduates of Public Higher Education Institutions of Mali (Cohort 2015-2018), Observatoire National de L’Emploi et de la Formation (ONEF), 2019. 15 The first study was on the professional integration of higher education graduates in Mali—Promotion 2011-2015, March 2017; the second study was the professional integration of graduates of institutions support by HESP—Qualitative component—survey of employers, teachers and administrative staff of the HEIs, July 2017; and the third was the 2019 study conducted by ONEF. 16 The tracer study was conducted by ONEF in close collaboration with HESP. HESP provided data on 12,505 students from seven public HEIs (not just HESP HEIs) enrolled in bachelor’s degree programs during the period 2015-2018. A random sample was selected based on academic field and institution. A final sample of 2,415 graduates were selected to be interviewed which took place between January 2019 and September 2020. Page 13 of 68 The World Bank Mali - Higher Education Support Project (P151318) (2021-2026) and the PNES which will build on the progress made under HESP and continue the focus on improving the overall relevance of the training programs. 32. The PDO indicator—direct project beneficiaries (Corporate core-indicator)—increased from the 2015 baseline value of 0 to 37,795, thus exceeding the 2020 target of 18,900 by 17,492 (92 percent increase). The percent females increased from 0 in 2015 to 35.01 percent, thereby reaching the 2020 target of 35 percent. It should be noted that one of the IRIs of the project “students benefitting from direct interventions to enhance learning (number female)” is equivalent to this PDO indicator with the exception of reporting on the actual numbers of female students. The achievement of the IRI was the numbers for total students benefiting from direct intervention are 37,795, thus exceeding the 2020 target of 18,900 and the number of females increased from the 2015 baseline value of 2,500 to 12,774 in 2021, thus exceeding the target of 5,000 by 7,774. Based on the similar nature of these two indicators, the IRI achievement is provided in conjunction with the PDO level indicator. 33. The first IRI that supported the achievement of the first objective was number of students enrolled in licensed programs at participating HEIs (disaggregated by number of females). The total number of students in licensed programs increased from a baseline of 10,000 in 2015 to 36,392 in 2021, thus exceeding the target of 18,900 (92.5 percent increase over the target). The female numbers increased from a baseline of 2,500 in 2015 to 12,774, exceeding the 2021 target of 5,000 (155 percent increase over the target). The increase can be attributed to outreach efforts to improve the flow of information to potential students on program offerings, affirmative action in favor of girls in competitive selection processes, scholarships, and financial incentives to reduce cost barriers to participation. Other initiatives undertaken by universities using project funds also contributed to this increase. For instance, IPRK converted part of a building constructed with support from the Project into a childcare center, in a bid to facilitate girls’ participation in university education. The second IRI was number of teachers trained in the licensed programs at participating HEIs increased from the baseline of 120 in 2015 to 1,403 in 2021, thus exceeding the target of 780 (79 percent increase over the target). As indicated above, the tracer study showed that administrators in the HESP HEIs were satisfied with the quality of the professionally licensed teaching staff upon completion of their training. The third IRI was share of budget in participating HEIs allocated to pedagogical inputs which increased from the 2015 baseline value of 10 percent to 15.28 percent but fell short of the 2021 target of 30 percent by 14.72 percentage points. This indicator was to measure the share of the project funding going to HESP HEIs allocated to pedagogical inputs. At the design stage, the task team aimed at incentivizing the targeted HEIs to have adequate focus and prioritize the provision of pedagogical resources beyond the planned physical improvements of the learning environment to improve teaching and learning. Although the target was missed, partly due to the definition of the indicator and the focus only on the HESP HEIs, the Project has achieved a substantial systemwide improvement of the quality of teaching and learning through the provision of pedagogical resources nationwide. In retrospect, the target may have been ambitious. The fourth IRI was increase in amount of cash generated internally at participating HEIs (percentage) which increased from the 2015 baseline value of 100 percent to 216.21 precent, thus exceeding the 2021 target of 160 precent by 56.21 percentage points. This increase is based on the financial resources generated by the university from a combination of fees, donations, and services rendered using the specialized equipment provided through the Project. 34. Finally, the fifth IRI number of public HEIs with proposal(s) for acquiring support from the Africa Centers of Excellence in the West Africa and Central Africa Project increased from the baseline value of 0 in 2015 to 6 in 2021, thus meeting the target of 6. These proposals were competitively selected by a committee of national and international experts with the aim of supporting public HEIs, other than the four-project selected HEIs, in implementing innovate proposals for human resource development and scientific research with ACE institutions. This was also the first time Malian HEIs had had an Page 14 of 68 The World Bank Mali - Higher Education Support Project (P151318) opportunity to participate in a regional higher education competition. The six-participating public HEI had a total of 15 research proposals that support research and training for 13 Master’s degree students, 31 Ph.D. students, and 15 post doctorates.17 Moreover, the programs have been important for maintaining professional connections and research collaboration with other regional higher education institutions. 35. In line with Outcome 1, the Project also supported the priority sectors of livestock and mining through the creation of the Department of Livestock and Animal Health at the University of Ségou and the development of a new African School of Mines to increase the quality graduates with skills aligned to these priority sectors. The Project supported construction of new facilities18, equipment, development of curriculum, new training programs, and training of faculty to ensure that these programs could train students in the priority sectors. These activities were all completed by project closing and will make it possible to more systematically train students for these priority sectors. 36. Outcome 2: Improved stewardship of the higher education system in Mali . Achievement of this outcome is judged to be Substantial and was achieved through a combination of interventions related improving the capacity of the MESRS in: (i) planning; (ii) oversight/regulatory role for both public and private HEIs; (ii) quality assurance; and (iii) data collection, reporting and utilization. Outcome 2 was measured by the following indicators: (i) number of institutional development plans approved at public HEIs (PDO level indicator); and (ii) number of public HEIs with a functional Board of Trustees (Conseil d’Administration—CA) (PDO level indicator). The IRIs associated with Outcome 2 were: (i) annual reports on DLIs produced on time (before end of March) (yes/no); (ii) number of public HEIs with a management information system with operations resource management; (iii) quality assurance unit established and operational, with semi-autonomous status (yes/no); (iv) number of users accessing digital library system; and (v) MIS statistical report published annually with relevant data from # of HEIs (yes/no). 37. Disbursement Linked Indicators. As indicated, DLI 2: Improved institutional governance (stewardship) of the system and the three associated DLRs were achieved in the four participating HEIs. DLR 1 ensured that the CA were established in all four institutions by March 2015 (Year 0 in the DLIs conditions) and ensured that they met twice a year before funds would be disbursed. DLR2 ensured that management tools and procedures were in place within the four HEIs and were continuously used before funds were disbursed. Finally, DL3 ensured that the institutional development plans were reviewed and approved each year and that the development plans indicators were on track to be achieved. Success of these DLRs was directly linked to the achievement of the PDO and intermediate indicators because they were a precondition for the achievement of this objective’s indicators. 38. The results of the PDO indicator—number of institutional development plans approved at public HEIs increased from the 2015 baseline of 0 to 9, thus meeting the 2021 target of 9.19 The results for the PDO indicator—number of public 17 Examples of the types of projects funded were related to development of a technology platform; research on heliotropium indicum, research on antioxidant activities of moringa leaves; training on acute poisoning in hospitals, and training on pedagogical requirements for fruit crops etc. 18 A total of 1,891.24 square meters of facilities were constructed, comprising a block for administration and pedagogy, a block of laboratories, and a classroom block with 10 classrooms for masters’ students, as well as a videoconference room. 19 The nine institutions are all public. Together, these institutions cover the full range of disciplines offered at the bachelors, masters and doctorate levels, and account for 97 percent of students enrolled in public higher education institutions in Mali. The list of institutions is as follows: l’Université des Lettres et de Sciences Humaines de Bamako (ULSHB) , l’Université de Ségou, l’Université des Sciences Juridiques et Politiques de Bamako (USJPB), l’Université des Sciences Sociales et de Gestion de Bamako (USSGB), l’Université des Sciences Techniques et des Technologiques de Bamako (USTTB), l’Ecole Nationale d’Ingénieurs Abderrhamane Baba TOURE, l’Ecole Normale d’Enseignement Supérieur de Bamako (ENSUP), l’Ecole Normale d’Enseignement Technique et Professionnel de Bamako (ENETP), l’Institut Polytechnique Rural de Formation et de Recherche Appliquée (IPR/IFRA). Page 15 of 68 The World Bank Mali - Higher Education Support Project (P151318) HEIs with a functional Board of Trustees (Conseil d’Administration) increased from the 2015 baseline value of 0 to 9, thus meeting the 2021 target of 9. The establishment of the CA as a part of the governance structure of HEIs has been important for the management of the HEIs. The established CAs now hold meetings twice a year during which they deliberate on the management and budgets of the institutions. The CAs responsibilities included, but were not limited to: (i) developing and revising the HEIs strategic plans; (ii) developing HEI procedures, management tool and audit reports; (iii) ensuring data systems are in place and statistical data is made available; (iv) ensuring advisory bodies (i.e., pedagogical and scientific council) report regularly on activities; and (v) ensuring HEI quality assurance is maintained in conjunction with the Malian Agency for Quality Assurance of Higher Education and Scientific Research (AMAQ-SUP). 39. The IRI—annual reports on DLIs produced on time (before end of March) (yes/no) was met. The Project Implementation Unit (PIU) was able to submit the DLI annual reports with the DLI achievements in a timely manner which indicated that the Project’s DLIs were achieved and all the DLIs were fully disbursed. This was a considerable achievement for MESRS given that this was the first higher education project to introduce results financing. 40. The IRI—number of public HEIs with a management information system with operations resource management increased from the 2015 baseline value of 0 to 920 in 2012, thereby meeting the target of 9. While the target for this IRS was met, the overall achievement related to connectivity and information systems is far more expansive than the IRI would indicate. As indicated, the Project helped expand the system to nine universities so that SIGES is now able to provide accurate statistics, on-line registration, human resource management, financial information etc., that was used to publish the statistical report (see below). In addition, the Project supported the Mali Research and Educational Network (MALIREN) which allowed for the interconnectivity of Malian HEIs as well as with other African HEIs. This support consisted of: (i) payment of MALIREN’s membership to the Research and Education Network of West and Central Africa (WACREN); (ii) payment of MALIREN’s participation in the Africa Connect 2 Project 21; (iii) establishment of the Boucle Locale Radio (BLR) for the interconnection of MALIREN member sites which includes HEIs and research institutes; 22 (iv) installation and implementation of MALIREN’s local network operations center (NOC); and (v) acquisition, installation and configuration of computer equipment and networks. Additionally, the Project helped establish videoconferencing centers in seven sites23 which has allowed for virtual meetings with other institutions related to teaching, research. This overall connectivity also allowed the MESRS to launch online courses when the COVID-19 pandemic forced schools to close in 2020. The Project supported the establishment of an online platform (Moodle platform 24) and trained teachers and researchers on the preparation and delivery of courses. Overall, 625 videos were prepared, 126 higher education teachers were trained in online course preparation and delivery, and 123 online courses developed by teachers and researchers. Going forward, the MESRS envisions using the Moodle platform for the creation of a future virtual university in Mali. 20 See footnote 20 for the list of the nine institutions. 21 Africa Connect 2 is a project funded by the European Union which aims to support the development of high capacity national and regional research and education networks across Africa. The Project also aims to strengthen African institutions’ linkages with the pan-European GÉANT network, thus boosting global collaboration opportunities for African research and education communities. WACREN is the implementing partner for the project in West and Central Africa. The project is currently in its third phase (Africa Connect 3). Mali is a member of WACREN through the MALIREN association (MALI Research and Education Network). 22 BLR represents a local radio loop or a wireless local loop. The Project supported the establishment of the MALIREN BLR to connect MALIREN member institutions with each other through radio. Specific support included the acquisition of the Network Operational Center and equipment for the deployment of the BLR. 23 The seven sites include USJPB (with the central point being in the Rectorate), ULSHB, USTTB, USSGB, ENI-ABT, IPF/IFRA and the University of Ségou. 24 Moodle is a learning platform designed to provide educators, administrators, and learners with a single robust, secure and integrated system to create personalized learning environments. Page 16 of 68 The World Bank Mali - Higher Education Support Project (P151318) 41. The IRI—MIS statistical report published annually with relevant data from # of HEIs (yes/no) was met. The MIS statistical report was published as planned. As indicated, the support provided for the SIGES system has allowed for the collection of standardized data that can be used to compare data as well as publish the statistical yearbook. The IRI— number of users accessing digital library system—increased from the 2015 baseline value of 0 to 84,341, thus exceeding the 2021 target of 10,000 by 74,341 users. The virtual library is fully established, and access was given to all student and teachers of HEIs registered in the MIS. With support of HESP, a contract was entered into with CAIRN.INFO 25 and provided more than 10,000 online resources to all professors and students in public HEIs, identified through SIGES. 42. Finally, the IRI—quality assurance unit established and operational, with semi-autonomous status (yes/no) was met with the Malian quality assurance unit established, operational with a semi-autonomous status. The Project supported the establishment of the AMAQ-SUP in June 2018 and the decree formalizing the organizational structure and modalities of operation in September 2018. The Project also supported the development of the governance and organizational structures for internal quality assurance units (CIAQs) within the HEIs.26 Once the CIAQs were established the HEIs conducted self-assessments to improve their programs based on evaluation standards. These accomplishments were achieved through support for: (i) four study tours27 that provided an opportunity for sharing best practices related to quality assurance mechanisms; (ii) development of evaluation standards; (iii) development of the administrative and financial operating procedures for the AMAQ-SUP; and (iv) support for awareness raising within HEIs on the importance of quality assurance. 43. Additional achievements related to improved stewardship of the higher education system in Mali has been the increased capacity of the DGESRS to fulfill their roles and responsibilities. The DGESRS is now able to carry out studies, collect and analyze data provided by SIGES, support planning with the HEIs, provide statistical reports, monitor the performance-based conditions (PBCs) of the HEIs, and provide support and training related to quality assurance. Moreover, the MESRS has given the DGESRS the overall responsibility of developing a common vision for HEIs as well the broader sub-sector program to improve the quality and relevance of higher education. 44. Results Based Financing. In addition, to the achievement of the PDO indicators and IRIs, the Project introduced DLIs with the goal of developing the four project HEIs into new models of institutional and pedagogical governance. The four project HEIs signed individual annual PBCs related to their institutional development plans that included individual DLR targets with the MESRS in conjunction with the MEF. As indicated, all of the targets for the HEIs were achieved. More importantly is the fact that the four HEIs have embedded the Project supported governance and operational models into their institutions and they now serve as governance model for the MESRS and other HEIs in Mali. In addition, the MEF, based on the experience and good practices of the PBC approach in the HESP, has established a performance-based resource allocation in its operational model and has entered into PBCs with other HEIs which the DGESRS is responsible for monitoring. Although not part of the original project design, the government has also drafted a new regulatory 25 Cairn.info is a French-language web portal, founded in 2005, containing scholarly materials in the humanities and social sciences. Much of the collection is in French, but it also includes an English-language international interface to facilitate use by non-francophone countries. 26 To date, the four selected HEIs as well as the vast majority of public HEIs have a CIAQ. This is also the case in most private higher education institutions. 27 The study tours were related to: (i) the quality assurance system in the United States; (ii) attendance in a quality assurance conference in France related to distance learning; (iii) reviewing the institutional and regulatory frameworks for the private sector and learning from Cote d’Ivoire; and (iv) reviewing the institutional and regulatory frameworks for the private sector and learning from Senegal. Page 17 of 68 The World Bank Mali - Higher Education Support Project (P151318) framework conferring autonomy to HEIs based on the HESP model universities. The approval of the framework is still pending while the draft continues the review and consultation process. Justification of Overall Efficacy Rating 45. The overall efficacy rating of Substantial is justified by the points mentioned above. All of the PDO indicators were met or exceeded and the DLIs were met. The Project played a significant role in laying the foundation for quality and relevance of the higher education system in Mali. The four-project supported HEIs are now demonstration models for a new governance and operational structure and the Government is now using the good practice of the PBC approach to establish a performance-based resource allocation in its operational model. The Project helped establish a robust management information system which has allowed for improved data collection, reporting, and analysis of data. The national and international connectivity has allowed for: (i) researchers to collaborate; (ii) students, teachers, and researchers to access digital resources through the virtual library; and (iii) delivery of online courses which could lead to a virtual university in Mali. In addition, a quality assurance system has been established which has led to an increase in licensed programs which are more relevant to the labor market needs, teachers trained in these programs and students attending the programs. Moreover, the Project met 10 out of 11 of the intermediate indicator targets and completed the creation of livestock and mining institutions in support of these priority economic sectors. C. EFFICIENCY Assessment of Efficiency and Rating 46. The overall efficiency under the Project is rated Substantial. This section summarizes the economic and financial analysis of the HESP by presenting: (i) the economic rationale for investment in higher education in Mali; and (ii) the results of the project cost-benefit analysis at the project implementation completion stage. The detailed economic and financial analysis is presented in Annex 4. 47. The economic analysis undertaken at appraisal provided the rationale for investing in higher education in Mali . First, the quality and relevance of higher education programs in the country were judged to be low, which led to relatively high unemployment rates, particularly among new graduates. While tertiary graduates generally earned higher wages than individuals with lower education levels in the long run, this was not the case for younger cohorts, perhaps because of the positive effect on wages of experience, and again an indication that tertiary graduates were not bringing the right skills to the marketplace. The share of science and engineering students in Mali’s HEIs was less than 4 percent in 2011, which could explain the skills mismatch, given that the priority sectors of agriculture, livestock, and mining, which accounted for nearly 40 percent of GDP. Only 4 percent of the tertiary educated worked in these sectors, reducing the likelihood of adoption of new technologies and innovation which could translate to expanded economic opportunities. Higher education is not only associated with higher wage earnings and employment opportunities, but also with an increased likelihood of wage employment and employment in sectors with higher returns and employment contracts, which offer greater job stability. Urgent reforms to uplift program quality and relevance in these sectors were therefore expected to improve employment and boost salaries among project beneficiaries. Lastly, higher education is not only beneficial for the individual, but also for families and society at large, due to strong spillover effects. 48. The cost-benefit analysis at project completion confirms that the Project is likely to have generated substantial benefits and is also likely to have had reasonably high rates of return. Both the internal rates of return (IRR) and net Page 18 of 68 The World Bank Mali - Higher Education Support Project (P151318) present value (NPV) estimates show that the Project is economically viable at the end of the project implementation based on actual project beneficiaries. The Project had a lower bound estimate for the NPV of benefits of US$228.9 million, and a lower bound estimate for the IRR of 12.5 percent. Although some benefits cannot be fully quantifiable, the NPV from the quantifiable benefits are larger than the NPV costs, and this strongly supports both the viability and efficiency of investments undertaken under the Project. Table 1: Net Present Value in millions of USD and Internal Rate of Return (base case) Baseline: Employment rate increase by 6.3 percentage points due to the project, from 81% to 87.3% Sub-Components 1.1, 1.2 Total IRR 12.5% 11.9% Discounted cost (present value of costs) $273.2 $283.5 Project cost $15.1 $25.5 Forgone earning $258.1 $258.1 Present value of incremental benefits $502.1 $502.1 NPV (with 2015 as the base year) $228.9 $218.5 Benefit/cost ratio 1.8 1.8 Source: World Bank’s estimation based on assumptions as detailed above, and the Implementation Completion Report for Higher Education Support Project (Borrower’s Final Project Report, 2021). Ministère de l’Enseignement Supérieure et de la Recherche Scientifique. 49. A few factors negatively impacted the overall implementation of the Project and for this reason efficiency is rated Substantial rather than High. The ISR in December 2017 noted some delays in the implementation of quality assurance activities/regulations at the central level, which negatively impacted the ability of the National School of Engineers (ENI) and the Rural Polytechnic Institute for Training and Applied Research (IPR/IFRA) to attain their DLI targets, but this was subsequently resolved. The closing date was extended by six months from December 31, 2020 to June 30, 2021, as described earlier. Despite the significant challenges posed by the COVID-19 pandemic and the challenging political situation in Mali, the HESP largely achieved all its intended outcomes in a substantially efficient manner. D. JUSTIFICATION OF OVERALL OUTCOME RATING 50. In terms of relevance of PDO, the Project’s development objectives were Highly relevant at appraisal and continue to remain so today, directly responding to the current key issues and challenges facing Mali youth employment and skills development on: (i) new PNES 2021-2030; (ii) Strategic Plan for Higher Education (2021-2026); and (iii) World Bank’s FY 2016-2019 CPF that focused on improving governance and building resilience. In terms of efficacy, the Project is rated Substantial having met all of the PDO indicators and 10 of the 11 IRIs as well as laying the foundation for systemic change in higher education. Efficiency is rated Substantial given the substantial benefits and reasonably high rates of return. Page 19 of 68 The World Bank Mali - Higher Education Support Project (P151318) 51. The overall outcome is rated as Satisfactory based on the High relevance of the PDO, the Substantial rating of the Project’s efficacy, and the Substantial rating of efficiency. E. OTHER OUTCOMES AND IMPACTS (IF ANY) Gender 52. As already indicated, women’s enrollment in licensed programs exceeded the original target 5,000 by 7,774 women. Again, as previously noted, the increase can be attributed to outreach efforts to improve the flow of information to potential students on program offerings, affirmative action, scholarships, and financial incentives to reduce cost barriers to participation and provision of childcare for women entering the programs. All-in-all this is considerable progress and support for women to be able to access higher education programs. While this was a substantial achievement for the Project, there is still a need to focus on closing the gender gap both within HEIs and the labor market. The tracer study showed positive results for HESP graduates such as more HESP graduates working in the primary sectors of the economy and that the HESP graduates earned an average of 100,000 CFA francs monthly while a quarter of the non-HESP graduates earned less than 100,000 CFA. However, overall, for HESP and non-HESP graduates, the study found that male graduates are more likely to have access to sustainable employment than women and that the average time to find employment for men was five months and for women was eight months. Going forward, the new Strategic Plan for Higher Education (2012-2026) focuses specifically on reviewing the scholarship program with specific focus on prioritizing women’s applications. Moreover, the MESRS aims to increase the number of women in fields of technological science, medicine, pharmacy, agronomy, and animal medicine. Institutional Strengthening 53. Governance Structure for Higher Education in Mali. After five years the governance structure of the higher education system had completed the needed reforms to become a viable system. In order to achieve this ambitious goal, the Government: (i) established the Board of Trustees; (ii) established the AMAQ-SUP and the CIAQs and developed the standards and evaluation criteria for the institutions; and (iii) developed the capacity of DGESRS. The system had moved from limited capacity to one that could oversee the management of institutions through the Boards of Trustees, DGESRS, and the quality-of-service delivery through the AMAQ-SUP an CIAQs. 54. Directorate General of Higher Education and Scientific Research. As indicated, the capacity of the DGESRS has been increased which has improved stewardship of the higher education system in Mali. The Project also helped to transform the DGESRS to a directorate that now has overall responsibility of developing a common vision for HEIs as well the broader sub-sector program to improve the quality and relevance of higher education. In support of this role, the DGESRS helped carry out research on: (i) the financing of higher education; (ii) the three project-supported tracer studies; and (iii) an institutional and organizational diagnosis of the current higher education and scientific research which were all used in the formulation of the new Strategic Plan for Higher Education. Moreover, the DGESRS is fully staffed and able to collect and analyze data provided by SIGES, support planning with the HEIs, provide statistical reports, monitor the PBCs of the HEIs, and provide support and training related to quality assurance. DGESRS has been fully institutionalized which is a strong achievement of the Project. Page 20 of 68 The World Bank Mali - Higher Education Support Project (P151318) 55. Malian Agency for Quality Assurance of Higher Education and Scientific Research. The higher education quality assurance process has been institutionalized in Mail. The policies, mechanism and ongoing processes for quality assurance have been established with support from the Project. The necessary ingredients of institutional self- assessment based on collection of qualitative and quantitative data by key players is functioning and the external independent quality assurance body has been fully established. AMAQ-SUP is also plugged into international quality assurance networks and has a partnership with national assessment agencies.28 There are also CIAQs established within most HEIs that support the ongoing quality assurance processes at the institutional level. 56. Performance Based Contracts. The PBCs were introduced through the Project and have now become institutionalized and are an important driver of change. HEIs have a sense of ownership and autonomy because of the PBC. The PBCs have clearly defined objectives, activities, and expected results. As indicated above, the MEF has stated that performance-based contracts and budgets will be instituted across the higher education system to continue the efforts started under the Project to improve accountability. The MEF has developed a model contract for PBCs that institutions can adapt to their individual and specific needs. Going forward, the DGHESR will continue to be responsible for monitoring the implementation of HEIs PBC as well as evaluating the success of PBC based on agreed annual indicators. Mobilizing Private Sector Financing Not Applicable. Poverty Reduction and Shared Prosperity 57. The Project supported the World Bank’s twin goals of eradicating extreme poverty and increasing shared prosperity through investment in human capital. The HESP contributed to expanded access to learning and facilitated the acquisition of skills for young people in Mali. The HESP contributed to poverty reduction and shared prosperity by providing a level of quality education to the beneficiaries which improved their employment opportunities as evidenced by the tracer study. However, unemployment and underemployment are still a concern for the graduates of HEIs in Mali and are cited by studies as factors facilitating the radicalization and violent extremism of individuals. 29 To mitigate these risks for HEI graduates, as well as its large youth population, Mali must continue to improve the relevance of its HEI programs, continue to connect with the private sector and encourage more productive employment throughout the economy. Other Unintended Outcomes and Impacts 58. Although not part of the PDO, digitalization of the higher education sub-sector was a critical part of the Project design and implementation. Overall, the HESP has resulted in significant digitalization efforts, which in the aftermath of the COVID-19 pandemic, formed a key building block for the national dialogue on how to ensure continuity of learning with the use of technology. The Project’s key digital achievements, including the establishment of a robust management information system in the higher-education sub-sector (Progiciel de Gestion Integree, PGI) and its key features (institutionalization of a unique identifier linked to a digitized student card, access portal, digital working place, online 28 This includesthe FrAQ-SUP network which comprises quality assurance agencies from several francophone countries in Europe and Africa, and the RAFANAQ which comprises quality assurance agencies in Francophone Africa. The national agencies include AEQES (Belgium), ANAQ (Guinea), ANAQ (Senegal), ANEAQ (Morocco), and HCERES (France). 29 West Africa Network for Peacebuilding. 2018. Current dynamics and challenges of violent extremism in West Africa. Page 21 of 68 The World Bank Mali - Higher Education Support Project (P151318) learning, online registration, integrated management of payments etc.), as well as the establishment of a distance learning platform rapidly informed the Government’s policy decision to introduce the ‘ Approche par Canal’. This approach supported the use of alternative education delivery channels during the COVID-19 pandemic with a focus on the use of radio for primary and lower secondary education provision, the use of television for upper secondary, and the use of distant learning (internet) for higher education. Subsequently, the HESP team took the lead in facilitating continuity of learning during the pandemic and achieved, as indicated, the following results: 625 instructional videos were developed for national education; 126 teachers were trained on distant learning provision; and 123 online courses were delivered. The HESP has also allowed students, teachers, and researchers to have open access to digital resources through the virtual library and researchers to collaborate remotely. These digitalization efforts will likely lead to partnerships with universities at the global level to provide high quality tertiary education moving forward. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION 59. Evidence Based Design. The Project’s design drew on several studies that pointed to major gaps in the higher education system and was aligned with the Government’s strategic goals and envisaged outcomes for the higher education sub-sector. Government strategy documents, including its 2003-2013 ten-year plan (Programme décennal de l’éducation – PRODEC) and an Interim Program for 2014-2016, focused on laying the foundations for improving education sector quality, including higher education. The National Policy on Higher Education and Scientific Research in 2010, which was informed by a 2008 National Forum on Education, identified the need for systemic change in higher education. A 2011 national student assessment highlighted poor labor market outcomes, and similarly, ICAs highlighted mismatches between education and training outcomes and labor market needs. A 2014 Report on the Future of Higher Education in Mali found acute challenges in the tertiary education sub-sector including: (i) a lack of diversity and low relevance of degree programs (low share of science, engineering and agriculture graduates); (ii) poor teaching practices; (iii) inadequate learning environments; (iv) pervasive governance gaps (low accountability and limited institutional autonomy); and (v) inefficient resource utilization. The Project was designed to holistically address shortcomings in the sector, by strengthening governance of the system, elevating the quality and relevance of programs offered by selected HEIs, and improving the overall quality of the learning and learning environment in the sector by financing critical inputs such as infrastructure and teaching materials. 60. The theory of change behind the Project was sound and the selection of key targets was appropriate . The PDO was focused and appropriate given the need to improve relevance, quality, oversight and accountability of HE institutions. The objective was outcome-oriented and was appropriately pitched for the Government capacity and development status of higher education in Mali. The Results Framework was developed to measure all the PDO areas using measurable indicators available at the time of project design. The DLIs were also developed using measurable indicators for the four project supported HEIs. The technical design of the Project drew from globally recognized approaches for developing effective HE interventions that were, and continue to be, well-aligned with international best practice. The various project-supported activities were comprehensive and selected in close collaboration with the Government in order to provide the means at the national and institutional levels to bring about maximum improvements in the areas of project focus—relevance, quality, accountability and oversight. The design and preparation activities provided a sound foundation for the Project. The Government also used the Project to shape its quality assurance mechanism system. The Project’s design included a comprehensive M&E system set-up and provided timely and reliable data on the various outputs and outcomes achieved under the operation. (See additional information Page 22 of 68 The World Bank Mali - Higher Education Support Project (P151318) under M&E section.) The targeted project beneficiaries for capacity development were project supported HEI education managers, faculty, researchers, and accreditation staff. 61. Project Financing. The Project was financed by an International Development Association (IDA) grant in the amount of US$19 million equivalent and an IDA Credit in the amount of US$14 million equivalent, for a total Project envelope of US$33 million equivalent. Approximately US$3.2 million of the Project’s total financial resources were allocated to DLIs. The instrument used was Investment Project Financing (IPF) with two disbursement modalities of items-based disbursements and DLIs. Using the mix modality approach provided an opportunity to address challenges at the HEI level and demonstrate that improvements could occur using an incentives-based approach. The IPF allowed for provision of the needed technical assistance and institutional strengthening schemes necessary to build the capacity within the Government to implement youth employment schemes and skills development. The DLIs were used to improve HEIs governance structures and leverage HEIs capacity to be responsive to labor market demands and to develop relevant programs. 62. Implementation Arrangements. The implementation arrangements gave the Council of Cabinet ( Conseil de Cabinet) of the MESRS and served as the Project Steering Committee (PSC) with overall responsibility for sector and policy coordination. The PSC was also the decision-making body for the Project. The minister of MESRS was the chair of the PSC which also consisted of the rectors and directors of the Grandes Ecoles30 of universities, and directors of the central units of MESRS, including the Director General of Higher Education, the Secretary General of the MESRS and the project coordinator who represented the overall Project. The Project Technical Committee (PTC) was chaired by the DGESRS which met bi-monthly to review annual workplans and monitor project progress. The PTC included the project coordinator, heads of planning, QA, private sector units, representatives from the Ministry of Mines, and other members of the Project Implementation Unit (PIU). The DGESRS was also responsible for ensuring that the HEI PBCs were being implemented and that the overall results of DLIs were being achieved. The PIU was located in the MESRS and was responsible for overall project coordination and implementation which included procurement, financial management (FM) preparation of annual workplans, results reports, consolidating HEI DLI reports, monitoring the achievement of the DLIs through the appointment of a consultant within the QA unit, and ensuring compliance with safeguards. The PIU also included a representative from the Ministry of Mines who provided support for the development of the new mining school. The HEIs were to be in charge of the implementation of their respective PBCs—which included both the DLIs and DLRs. 63. Risks and Mitigation Measures. Key risks and appropriate mitigation measures were identified during preparation. The overall risk was rated Substantial because of the consensus within the country on the need to reform the higher education sector to meet the skills challenges facing Mali particularly in agriculture, livestock and mining which accounted for more than 40 precent of Mail’s GDP at the time of Project preparation. The Project’s activities were directly related to risk mitigation through the: (i) inclusion of technical assistance and development of a procedures to increase capacity to implement PBCs; (ii) clearly defining the roles and responsibilities of the HEIs and the MESRS so as to not lose the advantages of autonomy while improving coordination and accountability; (iii) inclusion of FM procedures in the project operations manual (POM), customization of accounting software for the Project, and inclusion of FM arrangements of the HEIs in performance-based contracts; and (iv) recruitment of a procurement specialists, inclusion of detailed procurement guidelines and instruction in the POM, and capacity building through extensive training of all staff involved in procurement. 30 Grandes écoles are elite academic institutions with highly competitive admission requirements. Page 23 of 68 The World Bank Mali - Higher Education Support Project (P151318) B. KEY FACTORS DURING IMPLEMENTATION 64. Project implementation progress was rated moderately satisfactory and satisfactory throughout implementation. There were various factors that played a role in the challenges and successes of the Project. 65. Factors subject to Government and/or implementing entities control . Overall, the success and achievements of the Project were strongly linked to the work of the PIU, DGESRS and the HEIs implementing the PBCs. Initially, there was resistance within MESRS and HEIs to the reforms supported under the Project. However, through constant technical assistance support, the reforms in governance have been institutionalized. In addition, there have been significant changes within the targeted HEIs based on the investments made in building and equipping classrooms and laboratories that brought them to international standards, and training teachers/faculty and administrative staff of the HEIs. The challenges the Government faced with implementation were largely capacity constraints related to procurement. The challenges were addressed through continuous technical support and training by the World Bank’s procurement staff. 66. Factors subject to World Bank control. The Project benefitted from the fact that there was a consistent World Bank team that supported implementation, comprised of both local and international staff. This allowed the team to interact regularly with the PIU, DGESRS and other Government implementation units which facilitated the successful implementation of the Project. As indicate, the World Bank’s procurement staff provided on-going technical support which led to implementation of all project contracts. There was a formal mid-term reviews (MTR) of the Project in April 2018 that confirmed the PDO, project components, and DLIs remained relevant to the Mali context. The World Bank also supported the Government’s request to extend the Project by six months in order to complete a small number of project activities that were delayed due to the COVID-19 pandemic and the August 2020 coup d'état during which World Bank operations were suspended. 67. Exogenous factors. In March 2020, schools and Government offices were closed in Mali due to the COVID-19 pandemic which had an impact of the final months of implementation for HESP. All components were affected by the delays since all activities had to be delayed due to the virus. Moreover, in August 2020 there was a coup d’état which led to uncertainty whether World Bank-supported operations would continue to be effectively implemented and monitored by Mali’s new administration. Mediation efforts led by the ECOWAS paved the way for an 18-month transition period, with the appointment of a civilian President and Prime Minister in September 2020. These exogenous events led to Mali being designated a fragile state by the World Bank and led to the six-month request for the project extension to June 30, 2021. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E) M&E Design 68. The links between the inputs, outputs, and outcomes and the PDO were sound (as shown in Figure 1). The PDO was clearly specified at appraisal and remained the same throughout the life of the project. Six PDO-level indicators and 11 intermediate outcome indicators were defined at appraisal to measure the Project’s outcomes. Page 24 of 68 The World Bank Mali - Higher Education Support Project (P151318) 69. The M&E design was developed to monitor and report on progress toward meeting the targets of the PDO and intermediate indicators in the Results Framework (RF) through regular routine monitoring and reporting by the PIU and DGESRS. The design included support for the development of a computerized MIS that would integrate university subsystems into the national higher education system. Upon completion, the system was to capture the needed data for reporting on the various project indicators, a yearly statistical yearbook, and progress on HEIs development plans which included the introduction of licensed programs and teacher training, etc. The design team used an IPF instrument that utilized a results-based approach that included DLIs and DLRs for the four-project supported HEIs. The DLIs and DLRs paid specific attention to the introduction of new programs, delivery of teacher training, IT updates and monitoring and implementation of HEI development plans. These DLIs were also linked to the HEIs institutional development plans and the PBCs. This was highly appropriate given that the Project’s objective of supporting the Government’s overall HEI reform agenda. This was also the first higher education project in Mali to utilize DLIs. The design also included the establishment of a quality assurance process for institutions and HEI programs. Finally, the project M&E design included studies and evaluation which included studies on higher education financing, tracer studies and employer surveys. M&E Implementation 70. Reporting of data for the RF indicators was systematically done throughout the life of the Project. The DGESRS, PIU and universities analyzed and reported on results framework indicator achievement on a semiannual basis. The universities also provided progress reports on the achievement of the DLIs which were linked to the PBC which were all met. The PBCs were monitored by the PIU and DGESRS to ensure they were implemented and that the targets were being met. As indicated above, Malian Agency for Quality Assurance of Higher Education and Scientific Research has been fully institutionalized and performed (and continues to perform) monitoring and evaluation of a higher education quality assurance mechanism. Institutional self-assessments have been conducted and CIAQs established within institution. Moreover, as indicated, the Project helped establish of a robust management information system in the higher-education sub-sector. The DGESRS also conducted the studies31 with the support of technical assistance that were expected to be produces under the Project. In particular, the project conducted tracer studies and an employer satisfaction survey as well as a study on financing of higher education and feasibility of a student loan system. M&E was consistently rated satisfactory in the Implementation Status Reports (ISRs). M&E Utilization 71. Data was made available regularly for the results framework and was used to inform the analysis of the Implementation Completion Results Report (ICR). The MESRS has centralized all the data on a single platform and regularly monitors the workforce by university, faculty and sector. The data has also been used for the publication of an annual statistical yearbook thereby meeting the target of one of the intermediate indicators. The IT system implemented within each University has made it possible to collect data and systematically report on the implementation of performance contracts which helps MESRS and DGESRS monitor the PBCs and is used by universities to inform progress on their performance indicators. Again, CIAQ have been established within HEIs and self-evaluations 31 The project studies included the: (i) final report of the study on the challenges and constraints of private higher education in Mali; (ii) feasibility study for the African School of Mines—Mali; (iii) environmental and social impact study for the New School of Mines; and (iv) study on the curriculum and programs for the New School of Mines. Page 25 of 68 The World Bank Mali - Higher Education Support Project (P151318) have been conducted which will continue given the institutionalization of the Malian Agency for Quality Assurance of Higher Education and Scientific Research. The MEF has expanded the use of PBCs and the DGHESR will continue to monitor the PBCs and university budgets will be allocated based on achievement of specific university indicators. The higher education management information system institutionalized a unique identifier linked to a digitized student card, access portal, digital working place, online learning, online registration, integrated management of payments etc.). Finally, the studies were used to inform the new Strategic Plan for Higher Education (2012-2026). Justification of Overall Rating of Quality of M&E 72. The overall rating for M&E is Substantial. The well-developed design, implementation, and utilization of the system allowed for consistent reliable tracking of progress made under the Project, including utilizing data analysis to inform DGESRS oversight of universities, university decisions on developing their PBCs, and monitoring progress toward achieving the goals of the PBCs. As indicated, MEF has stated that PBCs will be instituted across the higher education system to continue the efforts started under the project to improve accountability and more outcome- based monitoring. Data is also now available for annual publication in a statistical yearbook that allows for greater transparency of data. In addition, capacity was built within the DGESRS to conduct studies and analyze data which played an important contributing role in the development of the new Strategic Plan for Higher Education (2012-2026). B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE 73. Safeguards. The original Project was rated as Category ‘B’ partial assessment given that none of the Project’s activities were expected to induce a negative impact on environmental resources or the quality of life in areas affective by implementation. OP/BP 4.01 “Environment Assessment” was triggered due to the potential production of solid waste associated with the refurbishment and construction of new facilities for the DGESRS and the University of Segou. OP/BP 4.12 “Involuntary Resettlement” was also triggered. An Environmental Social Management Framework (ESMF) and a Resettlement Policy Framework (RPF) from the Mali Emergency Education for All Project (P123503) were updated for the HESP to comply with OP/BP 4.12. The ESMF and RPF were disclosed in—country on March 30, 2015 and at the World Bank InfoShop on April 2, 2015. 74. Safeguards were monitored consistently during project implementation and ISRs were consistently rated moderately satisfactory and satisfactory. The main area of weakness identified related to the safeguards was that the PIU engaged the safeguards specialist remotely and they needed a local expert to be hired. However, in 2019, the PIU completed the required Environmental and Social Impact Assessment (ESIA) confirming that there would be limited impact and scale of the key construction activities implemented under the Project. In addition, the construction sites’ Environmental and Social Management Plans (ESMP) were prepared to mitigate the potential risks during the construction phase of the activities. In 2017, the Project was utilizing the grievance redress mechanism (GRM) in the MESRS that was not project specific. GRM information was being distributed to the public and grievances could be lodged through emails and normal mail. In 2019, the MESRS established a Project-specific GRM within the PIU for specific project complaints. Over the project period the GRM received, and satisfactorily processed two complaints32 related to planned construction in the area of Segou. 32One complaint was related to a basketball court that was set up by the National Center for University Works (CENOU) during the construction of the new livestock and animal health department of the University of Ségou. There was a complaint that the university students were using the court. The university resolved the complaint by allocating new space for use by students. The second complaint was related to teachers from l’Université des Sciences Juridiques et Politiques de Bamako (USJPB) who lodged Page 26 of 68 The World Bank Mali - Higher Education Support Project (P151318) 75. Financial management. FM performance rating was satisfactory throughout the project period except for the April 2016 rating which was moderately satisfactory (MS). The MS rating was due to FM staff changes in the first year of the Project, but the staffing situation was resolved and FM ratings remained satisfactory thereafter. During the project period there was a strong financial management team in place and the FM arrangements were in place that allowed for proper financial management. These included: (i) regular meetings of the steering committee to review annual work programs and validate internal audit tools; (ii) timely and satisfactory submission of interim financial reports (IFRs) of an acceptable quality on a regular basis; (iii) submission of the required audits to the World Bank in accordance with the Financing Agreement in which the auditors’ opinions were unqualified (clean); and (iv) effective implementation of recommendations that were made by the external audit and Bank supervision missions. The audit report covering the period January 1-December 31, 2020, issued an unqualified opinion on the financial statements of the Project. Moreover, the auditor conducted an assessment of the internal control system and noted that all FM recommendations had been effectively implemented. Annual financial statements were prepared by the PIU which complied with the Organization for the Harmonization of Business Law in Africa (OHADA) System (SYSCOHADA) which is a system used in West African Francophone countries. The final audit report is due on June 30, 2022, and will cover the period January 1-December 31, 2021. The total IDA project financing was US$33,000,000 equivalent with a total disbursement of approximately US$31.36 million equivalent. There was an undisbursed amount of US$1.46 million equivalent was returned to the World Bank on December 10, 2021 and approximately US$0.18 million in US$/SDR exchange rate losses. 76. Procurement. Procurement was consistently rated moderately satisfactory in the project ISRs. The sustained rating of MS was primarily due to delays in processing, approval and signing of procurement documents. During the project, World Bank procurement staff provided training and technical support to the PIU to ensure compliance with the procurement guidelines and to improve the efficiency of procurement activities. The World Bank team also conducted annual procurement reviews that provided recommendations on improvements to the processing of contracts. The PIU migration to the Bank’s Systematic Tracking of Exchanges in Procurement (STEP) took place in 2020 with procurement activities being included in the STEP platform. Procurement plans were updated regularly and submitted to the Bank and always of good quality. C. BANK PERFORMANCE Quality at Entry 77. The project preparation team ensured that the Project’s design was closely aligned with the 2003-2013 PRODEC, the World Bank’s May 2013 ISN for Mali which focused on long term economic growth and human capital development. The design was simple and focused, the PDO was precise, and key indicators and DLIs were appropriate for measuring progress toward achieving the PDO and the medium-term goals outlined in the results chain. The design took into consideration the recommendations from peer reviewers and lessons learned from the other education projects implemented in Mali and other World Bank-supported HE projects, particularly related to the design of PBC mechanisms. The use of results-based financing was a new approach for Mali that helped to improve HEI governance, and the quality and relevance of programs delivered within the selected HEIs. The preparation team identified the appropriate risks, incorporated design features to mitigate them, and included the relevant technical specialists to a complaint about the management of courses designed for its continuing education program. The HEI was advised to settle the matter internally since the complaint concerned the validation of courses before they went live. Page 27 of 68 The World Bank Mali - Higher Education Support Project (P151318) develop the Project. Moreover, preparation was a consultative process with various stakeholders in the sector such as faculty, staff, and students and parents at the HEIs as well as MESRS and MEF staff. Quality of Supervision 78. Supervision missions were semi-annually, staffed with higher education specialists as well as fiduciary and safeguards staff and consultants. The project team was actively engaged in supporting the Government in its efforts to implement the Project. The supervision teams consistently reported on FM, procurement and safeguards progress during supervision missions and worked with the DGESRS, PIU and other implementation agencies, to build their capacity in these areas. As indicated above, safeguards compliance was monitored regularly. The missions also systematically documented project progress in aide memoires, back-to-office reports, and ISRs, all of which kept the World Bank management informed of progress and provided the foundation for the ICR analysis. Whenever implementation challenges arose, the World Bank team worked with the Government to find appropriate solutions that would not compromise the integrity of the design. This was particularly true when the World Bank’s team responded to governments request for a six-month extension of the project closing date due to the 2020 COVID-19 school closures and coup d’etat. The World Bank’s willingness to respond to the Government’s request, led to the completion of the Project as designed and the institutionalization of many of the Project’s activities, in particular, PBCs, AMAQ-SUP and the establishment of Boards of Trustees. Justification of Overall Rating of Bank Performance 79. Based on the analysis above, overall World Bank performance is rated Satisfactory. D. RISK TO DEVELOPMENT OUTCOME 80. The risk to sustaining the development outcome is Moderate. This rating is based on the fact that project activities have been institutionalized by the DGESRS, the MESRS and the four-project supported HEIs which now serve as models for improved governance and program relevance. The original capacity risks identified were mitigated through the institutional strengthening at the central and institutional levels. Capacity as been developed to conduct tracer studies and employer surveys to better monitor progress of employment related interventions. This has fed into improvements in data systems and data being used to develop employment policies. The quality assurance capacity has been developed and the quality assurance mechanism has also been institutionalized, thereby contributing to the quality and relevance of programs. These all contributes to the likelihood of HESP interventions being sustained in project support areas. 81. However, the overarching risks to development outcomes are firstly the continuing political tension and conflicts within Mali. Over the last decade, relatively poor education outcomes combined with relatively high unemployment and poverty rates have made Mali’s large youth demographic particularly susceptible to ongoing political tensions. To mitigate these ongoing risks, Mali will need to continue to encourage more productive employment throughout the economy, including for the large number of self-employed, informal workers in the agriculture, livestock, and mining sectors. In addition, the longevity of equipment and construction activities financed by the Project will depend on regular maintenance and renewal of buildings and equipment, which the Government will need to provide going forward to ensure continued impact of the project. Finally, the project demonstrated that the use of performance-based contracts signed between selected HEIs and the Ministry of Higher Education and Page 28 of 68 The World Bank Mali - Higher Education Support Project (P151318) Scientific Research, requiring the HEIs to achieve pre-identified DLIs and DLRs prior to the disbursement of project funds, was a good model of boosting quality, institutional autonomy, and strengthening governance of the sector. The regulatory framework would extend this model sector-wide, for the disbursement of investment funds. Mali will need to adopt the draft regulatory framework conferring institutional autonomy on HEIs which is critical for ensuring the sustainability of the reforms adopted on a pilot basis in the Project’s selected HEIs as well as the scale-up of the reforms. V. LESSONS AND RECOMMENDATIONS 82. Lesson 1. Establishing a solid governance system is a critical component for ensuring implementation and institutionalization of reforms. In the case of Mali, the Project helped support the establishment of the DGESRS that is responsible for overseeing the development of the higher education sector, quality assurance, and the implementation of PBCs. Instrumental in achieving this was the commitment of the actors that facilitated the implementation of the reform which led to the autonomy of the Project-supported HEIs. These achievements provided a strong foundation for the governance system and are all important components for sustaining a strong higher education system. Recommendation. The Government now needs to adopt the draft higher education regulatory framework, thereby ensuring the institutionalization of autonomy for higher education institutions. 83. Lesson 2. The establishment of a M&E system should always be prioritized as a way to ensure that the achievements and challenges of a project are well-documented and monitored in order to allow governments and project teams to enhance their overall effectiveness and impact . The HESP project successfully established a sound M&E system and the numerous studies and evaluations which were undertaken during the life of the Project were essential for making policy decisions in the higher education sector. 84. Lesson 3. Effective accreditation mechanisms are essential for ensuring quality of HE programs and institutions. The Malian Agency for Quality Assurance of Higher Education and Scientific Research has evolved into an effective accreditation body. Moreover, institutional quality assurance units are essential to HEIs in the preparation of their self-evaluations. Recommendation. Going forward, the Malian Agency for Quality Assurance of Higher Education and Scientific Research team will need to increase their capacity to respond to the increased demand for accreditation of HEI programs and institutions. 85. Lesson 4. Performance based contracts are an important mechanism to improve dialogue, transparency, and monitoring of higher education institutions. The use of PBCs within HEIs proved to be an important mechanism for higher education institutions to negotiate annual budgets based on targets and results. The management culture with universities is now shifting from an inputs-based orientation to one focusing on results. As previously mentioned, the broader impact has been that PBCs are recognized by the MEF as best practice, and they are recommending the use the PBC framework to improve the oversight and efficiency of other HEIs. Key to the success of the Project was also the extent of buy-in from the HEIs for such a performance-based fund allocation mechanism. Senior management of the HEIs have been supporting the reforms enthusiastically and restructured internal processes in keeping with the spirit of these reforms. They have also been strong proponents of the continued use and expansion of the reform systemwide. 86. Lesson 5. Government strategic thinking and agility on the use of information technology can lead to a more expansive use of the technology than original designed or envisioned. The HESP investments in an HEI information Page 29 of 68 The World Bank Mali - Higher Education Support Project (P151318) system allowed for considerable connectivity with other institutions related to teaching and research as designed. However, because of the initial investments in information technology, the MESRS and universities were able to quickly adapt to the COVID-19 pandemic school closures by launching online courses. The delivery of on-line courses was not limited to HEIs but also primary and secondary school courses when and where possible. The flexibility and the agility of the MESRS to adapt to the COVID-19 pandemic, led to a much broader use of the information systems and technology than originally envisioned. As indicated, ongoing forward the MESRS envisions using the information . technology platform for the creation of a future virtual university. Page 30 of 68 The World Bank Mali - Higher Education Support Project (P151318) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: Improve the relevance of selected higher education programs Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Increase in Employer Percentage 0.00 40.00 87.30 satisfaction as measured by employer satisfaction survey 31-May-2015 31-Dec-2020 30-Jun-2021 of priority sector of participating HEIs Comments (achievements against targets): Target exceeded. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of programs in Number 45.00 135.00 197.00 participating HEIs that are proposed and developed with 30-Mar-2015 31-Dec-2020 30-Jun-2021 Page 31 of 68 The World Bank Mali - Higher Education Support Project (P151318) the private sector participation Comments (achievements against targets): Target exceeded. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Tracer Study (for gaduates Yes/No No Y Yes from the selected HEIs conducted) 31-Mar-2015 31-Dec-2020 30-Jun-2021 Comments (achievements against targets): Target achieved. Objective/Outcome: Improve the stewardship of the higher education system Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of institutional Number 0.00 9.00 9.00 development plans approved at public HEIs 31-May-2015 31-Dec-2020 30-Jun-2021 Page 32 of 68 The World Bank Mali - Higher Education Support Project (P151318) Comments (achievements against targets): Target achieved. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of Public HEIs with a Number 0.00 9.00 9.00 functional CA (Board of Trustees) 31-May-2015 31-Dec-2020 30-Jun-2021 Comments (achievements against targets): Target achieved. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Direct project beneficiaries Number 0.00 18900.00 37,795.00 31-May-2015 31-Dec-2020 30-Jun-2021 Female beneficiaries Percentage 0.00 35.00 35.01 Comments (achievements against targets): Page 33 of 68 The World Bank Mali - Higher Education Support Project (P151318) Target exceeded. A.2 Intermediate Results Indicators Component: Component 1: Support to Selected Public Higher Education Institutions (HEIs) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of national students Number 10,000.00 18900.00 36,392.00 enrolled in licensed programs at participating HEIs 30-Mar-2015 31-Dec-2020 30-Jun-2021 Number of female national Number 2,500.00 5000.00 12,774.00 students enrolled in licensed programs at 30-Mar-2015 31-Dec-2020 30-Jun-2021 participating HEIs Comments (achievements against targets): Target exceeded. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of teachers trained Number 120.00 780.00 1,403.00 Page 34 of 68 The World Bank Mali - Higher Education Support Project (P151318) in the licensed programs at 30-Mar-2015 31-Dec-2020 30-Jun-2021 participating HEIs Comments (achievements against targets): Target exceeded. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Share of budget in Percentage 10.00 30.00 15.28 participating HEIs allocated to pedagogical inputs 30-Mar-2015 31-Dec-2020 30-Jun-2021 Comments (achievements against targets): Target missed. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Increase in amount of cash Percentage 100.00 160.00 216.21 generated internally at participating HEIs 30-Mar-2015 31-Dec-2020 30-Jun-2021 Comments (achievements against targets): Page 35 of 68 The World Bank Mali - Higher Education Support Project (P151318) Target exceeded. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Annual Report on DLIs Yes/No No Y Yes produced on time (Before end of March) 31-May-2015 31-Dec-2020 30-Jun-2021 Comments (achievements against targets): Target achieved. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Students benefiting from Number 10,000.00 18900.00 37,795.00 direct interventions to enhance learning 30-Mar-2015 31-Dec-2020 30-Jun-2021 Students benefiting from Number 2500.00 5000.00 12,774.00 direct interventions to enhance learning - Female Comments (achievements against targets): Target exceeded. Page 36 of 68 The World Bank Mali - Higher Education Support Project (P151318) Component: Component 2: Higher Education System Support Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of public HEIs with a Number 0.00 9.00 9.00 management information system with operational 31-May-2015 31-Dec-2020 30-Jun-2021 resource management Comments (achievements against targets): Target achieved. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Quality-Assurance unit Yes/No No Y Yes established and operational, with semi-autonomous 31-May-2015 31-Dec-2020 30-Jun-2021 status Comments (achievements against targets): Target achieved. Page 37 of 68 The World Bank Mali - Higher Education Support Project (P151318) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of users accessing Number 0.00 10000.00 84,341.00 Digital library system 31-May-2015 31-Dec-2020 30-Jun-2021 Comments (achievements against targets): Target exceeded. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion MIS statistical report Yes/No No Y Yes published annually with relevant data from # HEIs 31-May-2015 31-Dec-2020 30-Jun-2021 (Public and Private) Comments (achievements against targets): Target achieved. Component: Component 3: Project Management and Technical Assistance Indicator Name Unit of Measure Baseline Original Target Formally Revised Actual Achieved at Page 38 of 68 The World Bank Mali - Higher Education Support Project (P151318) Target Completion Number of public HEIs with Number 0.00 6.00 6.00 proposal(s) for acquiring support from Africa Centers 31-May-2015 31-Dec-2020 30-Jun-2021 of Excellence in West and Central Africa Comments (achievements against targets): Target achieved. Page 39 of 68 The World Bank Mali - Higher Education Support Project (P151318) B. KEY OUTPUTS BY COMPONENT Objective/Outcome 1: Improve the relevance of selected higher education programs 1.Employer satisfaction as measured by employer satisfaction survey in priority sectors in participating HEIs Outcome Indicators 2. Increase in the number of programs in participating HEIs that are proposed and developed with private sector participation 3. Tracer Studies (for graduates of participating HEIs) conducted 1. Number of national students enrolled in licensed programs at participating HEIs (disaggregated by number of females) 2. Number of teachers trained in the licensed programs at participating HEIs Intermediate Results Indicators 3. Share of budget in participating HEIs allocated to pedagogical inputs 4. Increase in amount of cash generated internally at participating HEIs 5. Number of public HEIs with proposal(s) for acquiring support from Africa Centers of Excellence in West Africa and Central Africa Project Key Outputs by Component (linked to the achievement of the Objective/Outcome 1) Component 1: Support to Selected Public HEIs. In support of this envisaged outcome, the Project supported the priority sectors of livestock and mining through the creation of the Department of Livestock and Animal Health at the University of Ségou and the development of a new African School of Mines to increase the quality graduates with skills aligned to these priority sectors. The Project also supported construction of new facilities, equipment, development of curriculum, new training programs, and training of faculty to ensure that these programs could train students in the priority sectors. These activities were all completed by project closing and will make it possible to more systematically train students for these priority sectors. A total of 1,891.24 square meters of facilities were constructed, comprising a block for administration and pedagogy, a block of laboratories, and a classroom block with 10 classrooms for masters’ students, as well as a videoconference room. Page 40 of 68 The World Bank Mali - Higher Education Support Project (P151318) The first IRI that supported the achievement of the first outcome was number of students enrolled in licensed programs at participating HEIs (disaggregated by number of females). The total number of students in licensed programs increased from a baseline of 10,000 in 2015 to 36,392 in 2021, thus exceeding the target of 18,900 (92.5 percent increase over the target). The female numbers increased from a baseline of 2,500 in 2015 to 12,774, thus exceeding the 2021 target of 5,000 (155 percent increase over the target). The increase can be attributed to outreach efforts to improve the flow of information to potential students on program offerings, affirmative action in favor of girls in competitive selection processes, scholarships, and financial incentives to reduce cost barriers to participation. Other initiatives undertaken by universities using project funds also contributed to this increase. For instance, IPRK converted part of a building constructed with support from the Project into a childcare center, in a bid to facilitate girls’ participation in university education. The second IRI was number of teachers trained in the licensed programs at participating HEIs which increased from the baseline value of 120 in 2015 to 1,403 in 2021, thus exceeding the target of 780 (79 percent increase over the target). As indicated above, the tracer study showed that administrators in the HESP HEIs were satisfied with the quality of the professionally licensed teaching staff upon completion of their training. The third IRI was share of budget in participating HEIs allocated to pedagogical inputs which increased from the 2015 baseline value of 10 percent to 15.28 percent, but fell short of the 2021 target of 30 percent by 14.72 percentage points. The fourth IRI was increase in amount of cash generated internally at participating HEIs (percentage) which increased from the 2015 baseline value of 100 percent to 216.21 precent, thus exceeding the 2021 target of 160 precent by 56.21 percentage points. This increase in is based on the financial resources generated by the university from a combination of fees, donations, and services rendered using the specialized equipment provided through the Project. Component 2: Higher Education System Support. The results for the PDO indicator—number of programs in participating HEIs that are proposed and developed with private sector participation—increased from the 2015 baseline value of 45 to 197, thus exceeding the 2021 target of 135 (a 46 percent increase over the target). These programs referred to ‘licensed’ programs that were approved by the QA unit within the MESRS based on a request of the HEI to the QA unit to review. It should be noted that before the programs were reviewed, they were first recommended by the university’s approved by a council in-charge of reviewing programs on offer (Conseil de Perfectionnement), which included a mix of professors and private sector representatives from sectors targeted by the university, and/or the committee responsible for reviewing the content from scientific and pedagogical perspectives (Conseil Pédagogique et Scientifique). The programs were then approved by the Board of Trustees (Conseil d’Administration) which again comprised private sector members (30 percent) from relevant sectors. This organizational structure was a factor in ensuring strong private sector involvement in the development of relevant ‘licensed’ programs. In addition, the HEIs have signed partnership agreements with industry associations and private sector companies, to complement the training of their students through apprenticeships or internships. This also allows the HEIs to tap on the private sector for refining their programs and improving instruction. Page 41 of 68 The World Bank Mali - Higher Education Support Project (P151318) The PDO indicator—direct project beneficiaries (Corporate core-indicator)—increased from the 2015 baseline of 0 to 37,795, thus exceeding the 2020 target of 18,900 by 17,492 (92 percent increase). The percent females increased from 0 in 2015 to 35.01 percent, thereby reaching the 2020 target of 35 percent. It should be noted that, one of the IRIs of the project “students benefitting from direct interventions to enhance learning (number female)” is equivalent to this PDO indicator with the exception of reporting on the actual numbers of female students. The achievement of the IRI was the numbers for total students benefiting from direct intervention are 37,795, thus exceeding the 2020 target of 18,900 and the number of females increased from the 2015 baseline value of 2,500 to 12,774 in 2021, thus exceeding the target of 5,000 by 7,774. Based on the similar nature of these two indicators, the IRI achievement is provided in conjunction with the PDO level indicator. Component 3: Project Management and Technical Assistance. Employer satisfaction as measured by employer satisfaction survey in priority sectors in participating HEIs (in the three priority sectors) increased from the 2015 baseline value of 0 to 87.3 percent, thus exceeding the 2021 target of 40 percent by 47.3 percentage points. The employer satisfaction survey was conducted in 2019 as part of the project tracer study since the first cohort of beneficiary students completed their Bachelors (License) at the end of academic year 2017-2018. The survey showed a high employer satisfaction with the graduates from the project-supported HEIs in the three priority sectors. The number of public HEIs with proposal(s) for acquiring support from the Africa Centers of Excellence in the West Africa and Central Africa Project increased from 0 in 2015 to six in 2021, meeting the target of six. These proposals were competitively selected by a committee of national and international experts with the aim of supporting public HEIs, other than the four-project selected HEIs, in implementing innovate proposals for human resource development and scientific research with ACE institutions. The six-participating public HEI had a total of 15 research proposals that support research and training for 13 Master’s degree students, 31 Ph.D. students, and 15 post doctorates. Moreover, the programs have been important for maintaining professional connections and research collaboration with other regional higher education institutions. The PDO indicator—tracer studies (for graduates from the selected HEIs) conducted (yes/no)—was met. During the project period, there were three tracer studies conducted for graduates from the selected HEIs thereby meeting the target for the indicator. The results of the 2019 tracer study made it possible to identify opinions of teaching staff, locate the graduates of HEIs and identify the companies in which the graduates were employed. As indicated above, the survey showed a high employer satisfaction with the graduates from the three priority sectors. In addition, the survey showed that there were some formal partnerships between companies and HEIs but that they were limited and still in an embryonic stage. A summary of findings specifically related to the HESP HEIs are that: (i) administrators in the HESP HEIs were satisfied with the quality of the professionally licensed teaching staff upon completion of their training; (ii) the staff of the HESP universities would like to see a greater focus on practical skills with 60 percent of the volume of hours devoted to practical skills; (iii) the large majority of graduates who benefitted from internships at the time of training were enrolled in HESP supported universities and that the internships were offered largely based on personal relationships; (iv) the insertion rate of HESP institution graduates was six out of 10 graduates (about the same as non-HESP HEIs) and the percentage of HESP graduates working in their primary sector is higher than non-HESP graduates; and (v) six out of ten jobs of HESP Page 42 of 68 The World Bank Mali - Higher Education Support Project (P151318) graduates are formal jobs. It should be noted that the overall study results for both HESP HEIs and non-HESP graduates showed that employability was problematic with 30 percent of graduates persistently unemployed, 28 percent in paid employment, 4 percent on a paid internship, 15 percent self-employed, and 23 percent continuing training. The overall results of the survey have been used by the MESRS in the preparation of the Strategic Plan for Higher Education (2021-2026) and the PNES which will build on the progress made under HESP and continue the focus on improving the overall relevance of the training programs. Objective/Outcome 2: Improve stewardship of the higher education system 1. Number of institutional development plans approved at public HEIs Outcome Indicators 2. Number of public HEIs with a functional Board of Trustees (Conseil d’Administration) 1. Annual reports on DLIs produced on time (before end of March) (yes/no) 2. Number of public HEIs with a management information system with operations resource management Intermediate Results Indicators 3. Quality assurance unit established and operational, with semi- autonomous status (yes/no) 4. Number of users accessing digital library system 5. MIS statistical report published annually with relevant data from # of HEIs (yes/no) Key Outputs by Component (linked to the achievement of the Objective/Outcome 2) Component 1: Support to Selected Public HEIs. The results of the PDO indicator—number of institutional development plans approved at public HEIs—increased from the 2015 baseline value of 0 to 9, thus meeting the 2021 target of 9. The results for the PDO indicator—number of public HEIs with a functional Board of Trustees (Conseil d’Administration)—increased from the 2015 baseline value of 0 to 9, thus meeting the 2021 target of 9. The establishment of the CA as a part of the governance structure of HEIs has been important for the management of the HEIs. The established CAs now hold meetings twice a year during which they deliberate on the management and budgets of the institutions. The CAs responsibilities include but are not limited to: Page 43 of 68 The World Bank Mali - Higher Education Support Project (P151318) (i) developing and revising the HEIs strategic plans; (ii) HEI procedures, management tool and audit reports; (iii) ensuring data systems are in place and statistical data is made available; (iv) ensuring advisory bodies (i.e., pedagogical and scientific council) report regularly on activities; and (v) ensuring HEI quality assurance is maintained in conjunction with the AMAQ-SUP. The IRI—annual reports on DLIs produced on time (before end of March) (yes/no)—was met. The Project Implementation Unit (PIU) was able to submit the DLI annual reports with the DLI achievements in a timely manner which indicated that the project DLIs were achieved and all the DLIs were fully disbursed. This was a considerable achievement for MESRS given that this was the first higher education project to introduce results financing. Component 2: Higher Education System Support. The IRI—number of public HEIs with a management information system with operations resource management—increased from the 2015 baseline value of 0 to 9 in 2012, thereby meeting the target of 9. While the target for this IRS was met, the overall achievement related to connectivity and information systems is far more expansive than the IRI would indicate. As indicated, the Project helped expand the system to nine universities so that SIGES is now able to provide accurate statistics, online registration, human resource management, financial information etc. that was used to publish the statistical report (see below). In addition, the Project supported the MALIREN which allowed for the interconnectivity of Malian HEIs as well as with other African HEIs. This support consisted of: (i) payment of MALIREN’s membership to the WACREN; (ii) payment of MALIREN’s participation in the Africa Connect 2 Project; (iii) establishment of BLR for the interconnection of MALIREN member sites which includes HEIs and research institutes; (iv) installation and implementation of MALIREN’s local network operations center; and (v) acquisition, installation and configuration of computer equipment and networks. Additionally, the Project helped establish videoconferencing centers in seven sites which has allowed for virtual meetings with other institutions related to teaching, research, etc. This overall connectivity also allowed the MESRS to launch online courses when the COVID-19 pandemic forced schools to close in 2020. The Project supported the establishment of an online platform (Moodle platform) and trained teachers and researchers on the preparation and delivery of courses. Overall, 625 videos were prepared, 126 higher education teachers were trained in online course preparation and delivery, 123 online courses developed by teachers and researchers. Going forward, the MESRS envisions using the Moodle platform for the creation of a future virtual university in Mali. The IRI—MIS statistical report published annually with relevant data from # of HEIs (yes/no)—was met. The MIS statistical report was published as planned. As indicated, the support provided for the SIGES system has allowed for the collection of standardized data that can be used to compare data as well as publish the statistical yearbook. The IRI—number of users accessing digital library system—increased from the 2015 Page 44 of 68 The World Bank Mali - Higher Education Support Project (P151318) baseline value of 0 to 84,341, thus exceeding the 2021 target of 10,000 by 74,341 users. The virtual library is fully established, and access was given to all student and teachers of HEIs registered in the MIS. With support of HESP, a contract was entered into with CAIRN.INFO and provided more than 10,000 online resources to all professors and students in public HEIs, identified through SIGES. The access however ended on December 31, 2020 (the original project closing date) and has thus far not been renewed. Finally, the IRI—quality assurance unit established and operational, with semi-autonomous status (yes/no)—was met with the Malian quality assurance unit established, operational with a semi-autonomous status. The Project supported the establishment of the AMAQ-SUP in June 2018 and the decree formalizing the organizational structure and modalities of operation in September 2018. The Project also supported the development of the governance and organizational structures for internal quality assurance units within the HEIs. Once the CIAQs were established the HEIs conducted self-assessments to improve their programs based on evaluation standards. These accomplishments were achieved through support for: (i) four study tours that provided an opportunity for sharing best practices related to quality assurance mechanisms; (ii) development of evaluation standards; (iii) development of the administrative and financial operating procedures for the AMAQ-SUP; and (iv) support for awareness raising within HEIs on the importance of quality assurance. Additional achievements related to improved stewardship of the higher education system in Mali is the increased capacity of the DGESRS. The DGESRS now is able to carry out studies, collect and analyze data provided by SIGES, support planning with the HEIs, provide statistical reports, monitor the PBCs of the HEIs, and provide support and training related to quality assurance. Moreover, the MESRS has given the DGESRS the overall responsibility of developing a common vision for HEIs as well the broader sub-sector program to improve the quality and relevance of higher education. Page 45 of 68 The World Bank Mali - Higher Education Support Project (P151318) ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION A. TASK TEAM MEMBERS Name Role Preparation Pierre Joseph Kamano Task Team Leader(s) Mahamadou Bambo Sissoko Procurement Specialist(s) Celestin Adjalou Niamien Financial Management Specialist Alice Diarra Sangare Program Assistant Janet Omobolanle Adebo Team Member Ruxandra Costache Counsel Maman-Sani Issa Social Specialist Andreas Blom Lead Economist Yeyande Kasse Sangho Team Member Emmanuel Ngollo Education Consultant Emanuela Di Gropello Lead Education Specialist Bleoue Nicaise Ehoue Team Member Salamata Bal Social Specialist Supervision/ICR Jeffrey Waite, Amina Debissa Denboba Task Team Leader(s) Pierre Jospeh Camano Senior Education Specialist (Task Team Leader) Adama Ouedraogo Senior Education Specialist (Task Team Leader) Boubacar Diallo Procurement Specialist Mamadou Sangare Procurement Specialist Tahirou Kalam Financial Management Specialist Page 46 of 68 The World Bank Mali - Higher Education Support Project (P151318) Shyam Srinivasan ICR Contributor, Young Professional Angele Compaore Ouattara Procurement Team Marouan Maalouf Counsel Adama Diop Finance Officer Ghislaine Kouedi Nombi Finance Analyst Mahamadou Ahmadou Maiga Senior Social Development Specialist Simbo Toukara Operations Consultant Tolidji Blaise Donou Environmental Specialist Etienne Baranshamaje Distance Learning Specialist Madda Ba Coulibaly Information Technology Specialist Emeran Serge M. Menang Evouna Senior Environmental Specialist Mahamadou Bambo Sisoko Senior Procurement Specialist Jean Charles Kra Senior Financial Management Specialist Souleymane Zerbo Architect, Consultant Kalilou Sylla Economist, Consultant Amina D. Denboba Economist, Consultant Mahamadou Ahmadou Maiga Senior Social Development Specialist Nikolai Alexei Sviedrys Wittich Procurement Team Moussa Fode Sidie Program Assistant Alice Sangare Program Assistant Bintou Sogodogo Program Assistant Felly Akiiki Kaboyo Operations Analyst Sandra Beemer ICR Contributor/Main Author Setou Mamadou Diarra ICR Contributor/Financial Analysis Bernardo da Cruz Vasconcellos ICR Team Member/ICR Quality Assurance Page 47 of 68 The World Bank Mali - Higher Education Support Project (P151318) B. STAFF TIME AND COST Staff Time and Cost Stage of Project Cycle No. of staff weeks US$ (including travel and consultant costs) Preparation FY15 20.525 176,653.58 FY16 .325 13,009.57 FY17 0 0.00 Total 20.85 189,663.15 Supervision/ICR FY16 9.482 59,952.78 FY17 9.100 75,673.95 FY18 7.051 77,446.84 FY19 10.656 59,414.62 FY20 37.184 169,402.81 Total 73.47 441,891.00 Page 48 of 68 The World Bank Mali - Higher Education Support Project (P151318) ANNEX 3. PROJECT COST BY COMPONENT Amount at Approval Actual at Project Percentage of Approval Components (US$M) Closing (US$M) (US$M) Component 1: Support to Selected Public Higher 22.00 19.58 89 Education Institutions (HEIs) Component 2: Higher 8.00 8.67 108 Education System Support Component 3: Project Management and Technical 3.00 3.11 103 Assistance Total 33.00 31.361 95.00 1 There was an undisbursed amount of US$1.46 million equivalent and approximately US$0.18 million in US$/SDR exchange rate losses. Page 49 of 68 The World Bank Mali - Higher Education Support Project (P151318) ANNEX 4. EFFICIENCY ANALYSIS Analysis of sector background, and linkages with economic development, poverty reduction and shared prosperity 1. Access to higher education in Mali remains limited, although it has improved slightly over the last decade. Data on the highest education level ever attained (Figure 4.1) shows slightly higher rates of tertiary education completion among 25- to 34-year-olds compared to older generations, but the figure remains low at only six percent for this age group. Data on 19- to 24-year-olds is similar (Figure 4.2), with five percent of the age cohort currently attending tertiary education. While the enrolments of public HEIs in Mali have increased significantly in recent years from 58,884 in 2015/16 to 84,364 in 2018/19, this still represents only a small proportion of the total population. 33 2. Large disparities persist in terms of gender, geography (urban and rural subpopulations), and socioeconomic status. These disparities exist at all levels of education but are particularly pronounced at the tertiary level. Higher education among women, rural residents and the poorest quintile of the population are below two percent among 25- to 64-year-olds (Figure 4.1). Data for 19- to 24-year-olds seems to be moving in the right direction in terms of gender and geography (Figure 4.2), although tertiary education among the rural population is almost non-existent. This points to the need for greater support for marginalized groups. Figure 4.1: Highest education level completed among the Mali population aged 25 to 64, disaggregated by gender, geography, socioeconomic status, and age Quintile Richest 49 11 10 6 9 14 Wealth Gender 25-64 Area 25-64 25-64 Poorest 90 7 0 21 Rural 82 10 4 1 21 Urban 52 12 10 7 8 12 Female 80 9 5 23 2 Male 66 12 7 4 4 7 Age 25-64 74 10 6 3 3 4 Age 55-64 81 9 4 22 3 Age 45-54 79 9 5 22 2 Age 35-44 78 10 4 2 3 4 Age 25-34 65 12 8 4 5 6 0 10 20 30 40 50 60 70 80 90 100 No education Some Primary Some lower secondary Some upper secondary TVET Higher Annual statistics on tertiary education students in Mali (or Annuaire Statistique des etudiants de l’Enseignment Superieur du 33 Mali), 2015/16 and 2018/19. Page 50 of 68 The World Bank Mali - Higher Education Support Project (P151318) Source: World Bank’s estimations based on 2018 EHCVM Figure 4.2: Highest education level completed among the Mali population aged 19 to 24, disaggregated by gender, geography, and socioeconomic status Richest Quintile 33 12 20 14 11 11 Wealth Poorest 72 11 13 0 31 Area Rural 62 14 15 5 32 Urban 31 11 20 16 10 11 Gender Female 56 13 15 8 5 4 Male 44 13 20 11 5 6 Total 19-24 51 13 17 9 5 5 0 10 20 30 40 50 60 70 80 90 100 No education Some Primary Some lower secondary Some upper secondary TVET Higher Source: World Bank’s estimations based on 2018 EHCVM 3. The slight expansion of enrolment at the upper secondary level suggests that tertiary education attainment rates could improve over time, but the net enrollment rate (NER) remains low across all levels of education. Figure 4.3 compares the NER at the primary, lower, and upper secondary levels between 2013 and 2018 (latest year of data available). NER at the primary and lower secondary levels have remained largely stagnant, whereas enrollment at the upper secondary level has increased by four percentage points. The large number of children who are excluded from even primary education is cause for concern, so is the low enrollment rate at the secondary level. The focus should therefore be on getting more children to attend primary education, before facilitating their transition to secondary education and beyond. The higher enrollment at the upper secondary level is an encouraging sign that tertiary education rates could improve. The focus there should be on ensuring that tertiary education programs are labor market relevant. 4. Tertiary education benefits both the individuals, and the economy and society at large. At the individual level, tertiary education results in better employment prospects, higher salaries, and a higher propensity to save and invest for the future. Indirect benefits over the long term could include better quality of life and health outcomes for the individual, but also for their families and children, pointing to potentially large returns to boosting tertiary education Page 51 of 68 The World Bank Mali - Higher Education Support Project (P151318) completion rates.34 With regard to the economy and society, higher investment in research and development, more effective absorption, application, generation and transfer of knowledge, skills and technology, increased tax revenues and consumption stimulus from the private sector, more effective political and social discourse, stronger governance and better public health are some of the channels.35 Figure 4.3: Comparison of NER at the primary and secondary level between 2013 and 2018 (latest year of data available) 60.0% 54.6% 55.7% 2013 2018 50.0% 40.0% 30.0% 23.6% 24.6% 20.0% 12.0% 8.2% 10.0% 0.0% Primary Lower secondary Upper secondary Source: World Bank’s estimations based on 2018 EHCVM 5. Individuals with tertiary education generally start on a higher salary base, and grow their salaries at a faster rate with experience than those with lower levels of education. Among those that are employed, there is a wage premium to tertiary education. Figure 4.4 shows the average wage by education level for Malians aged 25 to 64 disaggregated by age. It is evident that individuals with tertiary education earn significantly higher salaries than those with lower levels of education throughout their lives. The wage differential grows across age cohorts, demonstrating that the experience premium for tertiary educated is larger than that for other age cohorts. 6. The wage premiums may be explained by the impact on higher education on the quality of employment. Figure 4.5 shows that the likelihood of individuals enjoying wage employment (likely with formal employment contracts) increases with education, with individuals having some higher education being most likely to do so. This suggests that higher education results in greater job stability. Figure 4.6 shows that individuals with some higher education are also more likely than individuals with lower education levels to be in the services industry, which is associated with higher wages, as opposed to agriculture, livestock, and mining. The share of tertiary-educated working Schady N., 2011, which found that the schooling and vocabulary of mothers was a strong predictor of children’s 34 See for instance cognitive development in early childhood, based on longitudinal evidence from Ecuador. From Schady, N., Parents' education, mothers' vocabulary, and cognitive development in early childhood: longitudinal evidence from Ecuador. Am J Public Health. 2011;101(12):2299-2307. doi:10.2105/AJPH.2011.300253 35 Bloom, David E., David Canning, Kevin Chan, and Dara Lee Luca. 2014. “Higher Education and Economic Growth in Africa.” International Journal of African Higher Education, 1(1): 23-57. Page 52 of 68 The World Bank Mali - Higher Education Support Project (P151318) in the agriculture, livestock and mining sectors remains low at around 6 percent altogether, despite these sectors accounting for more than 40 percent of GDP and merchandise exports, and there could be significant productivity gains if there are improvements in the education offerings in these domains. The potential productivity gains from a higher share of tertiary educated entering these fields could have significant positive implications for the rural poor, who mostly work in the agriculture and livestock sectors. Figure 4.4: Average wages of Malians aged 25 to 64 disaggregated by age and education level No education Some primary Some lower secondary Some upper secondary 1,000,000 TVET Tertiary 900,000 800,000 Monthly wages in FCFA 700,000 600,000 500,000 400,000 300,000 200,000 100,000 - Ages 25-34 Ages 35-44 Ages 45-54 Ages 55-64 Source: World Bank’s estimations based on 2018 EHCVM Figure 4.5: Breakdown of Malians aged 25 to 64 by job type and education level Some higher education 62 35 3 TVET 46 47 7 Some upper secondary 24 50 26 Some lower secondary 8 52 40 Some Primary 2 42 57 No education 2 38 60 0 10 20 30 40 50 60 70 80 90 100 Wage Non-wage houshold enterprises Agriculture Source: World Bank’s estimations based on 2018 EHCVM Page 53 of 68 The World Bank Mali - Higher Education Support Project (P151318) Figure 4.6: Breakdown of Malians aged 25 to 64 by education level and employment sector Some higher education 3 21 7 88 TVET 7 12 12 78 Some upper secondary 16 01 8 74 Some lower secondary 30 11 14 54 Some primary 53 21 9 35 No education 58 6 1 8 28 0 10 20 30 40 50 60 70 80 90 100 Agriculture Livestock Mining Other industry Services Source: World Bank’s estimations based on 2018 EHCVM 7. The benefits of higher education are mainly transmitted through higher employability, but unemployment among the tertiary-educated in Mali remains common. Tertiary-educated people are more likely to be unemployed compared to those with lower educational attainment, unemployment rates among the tertiary educated are still high (Figure 4.7), particularly at lower ages. Mismatches between higher education programs and labor-market needs could be a key reason for this observation, highlighting a strong need for improving the quality and relevance of tertiary education programs, which was a strong motivation for this Project. Employment prospects for the tertiary educated improve across their lifetimes, and are generally better than those with lower educational attainment. Differences among younger cohorts could also be partially due to individuals with lower education levels entering the marketplace earlier than the tertiary educated, but the latter eventually catch up. Figure 4.7: Unemployment rates by level of education 25.0% 19.6% 19.5% 18.5% 18.8% 20.0% 16.4% 12.3% 17.2% 15.0% 13.2% 11.9% 12.2% 11.4% 10.9% 10.0% 8.8% 7.2% 11.4% 12.6% 5.0% 2.1% 2.2% 0.0% 2013 2014 2015 2016 2017 2018 Advanced education Basic education Intermediate education Page 54 of 68 The World Bank Mali - Higher Education Support Project (P151318) Source: International Labour Organization, ILOSTAT database. Data retrieved on January 29, 2021. 8. There continues to be an imbalance in specialization fields in higher education in Mali in favor of humanities and social sciences, and away from science and engineering, although the bias is less pronounced than noted in the original PAD. The share of science and engineering in tertiary enrolment in Mali was noted as being 3.8 percent in 2011 according to the PAD. However, the share in science and engineering disciplines seems to have risen to 24 percent based on 2018-2019 cohort data obtained from the government. However, the bulk of the science and engineering sector graduates come from the health sector (11 percent), which is certainly of importance to human capital development in Mali. However, the remaining disciplines including agriculture account for only 13 percent, despite the large contribution of fields like agriculture and mining to the economy. The same imbalance could explain the relatively high unemployment rates among tertiary education graduates. Table 4.1: Breakdown of students enrolled in HEIs by areas of study (2018-2019 cohort) Gender Sub-total Area of Study Female Male No. No. No. % Humanities and Social Sciences 22,099 42,125 64,224 76% Letters, Languages and Arts 4,287 9,609 13,896 16% Education and Training Sciences 1,704 3,781 5,485 7% Humanities and Social Sciences 3,404 9,442 12,846 15% Economic Science and Business Administration 8,222 12,614 20,836 25% Legal, Political and Administration Sciences 4,482 6,679 11,161 13% Sciences and Engineering 5,766 14,374 20,140 24% Sciences and Techniques of Physical and Sports 33 144 177 0% Activities, Youth and Leisure Science and technology 1,326 4,923 6,249 7% Agronomic Sciences 1,183 3,556 4,739 6% Health Sciences 3,224 5,751 8,975 11% Total 27,865 56,499 84,364 100% Source: Ministry of Higher Education and Scientific Research, Mali Government Cost-benefit analysis 9. The benefits of the Project were both quantifiable and non-quantifiable. The quantifiable portion accounts for about 60 percent of the total project costs. In particular, the benefits of sub-component 1.1 (Support to selected existing public HEIs) and sub-component 1.2 (Support to the University of Segou’s livestock department) have quantifiable benefits, while sub-component 1.3 (Support to the development of the new Bamako Africa School of Mines), component 2 (Higher Education System Support), and component 3 (Project management and technical assistance) are not easily quantifiable. Like the economic analysis in the original PAD, this analysis is premised on the difference in expected lifetime earnings for student beneficiaries with and without the Project. However, unlike the original analysis undertaken during project preparation, this analysis focuses on private benefits to individuals, rather than benefits calculated at the household or public level. These other sets of benefits are difficult to quantify with Page 55 of 68 The World Bank Mali - Higher Education Support Project (P151318) high fidelity, and have therefore been excluded for the purpose of this ICR. Even without these other benefits, the strong IRR demonstrates the Project’s efficiency in achieving its objectives. The following presents the model assumptions, NPV and IRR for the project. 10. The cost benefit analysis is premised on the project benefitting two different groups of student beneficiaries in slightly different ways. a. The first set of beneficiaries were new students who would otherwise not have enrolled in tertiary education. The four HEIs supported under component 1.1 of the Project received grants in exchange for achieving DLIs, which they used to develop new programs with private sector participation, construct new buildings and facilities, and purchase associated equipment. These investments resulted in higher enrollments for these institutions. Component 1.2 also supported the establishment of a new livestock department at the University of Segou. The analysis presented herein assumed that these newly enrolled students would otherwise not have enrolled in these institutions, and more generally in tertiary education, especially since the HEIs supported under the Project are key institutions in the sectors concerned, and students were unlikely to find other programs of a comparable quality elsewhere. For these students, we have assumed that the alternative would have been to enter employment after having completed only upper secondary education, resulting in a lower probability of employment, as well as lower lifetime earnings. The number of students under this category is captured by the indicator “Number of national students enrolled in licensed programs at participating HEIs” which increased by 26,392 students from the 2015 baseline value of 10,000. b. New and existing students benefitted from the improved adequacy, relevance, and quality of programs. The Project funded the development of curriculum with the involvement of the private sector, and the training of teachers. The improved quality and relevance translated to a high employer satisfaction rate (87.3 percent), which would improve the employability of graduates from the HEIs and programs supported by the Project. We note that the 2019 tracer study did not show a major difference in the insertion rate of graduates from HEIs supported by the HESP compared to graduates from HEIs not supported by the Project (both around 60 percent). However, we note that the study was limited to the first cohort of beneficiary students that completed their Bachelors (License) at the end of 2017-2018. It could take longer for the impact of quality improvements to take full effect, and translate to higher employability rates. The insertion rate measured relatively soon after graduation may also not reflect the long-term probability of these students being employed, as the employment rate for tertiary educated individuals as per ILO data on unemployment among individuals with advanced education was 81 percent (2018) – much higher than the 60 percent suggested by the tracer study. We have therefore assumed that the employment rate for student beneficiaries would be closer to the employer satisfaction rate, which would represent a 6.7 percentage point increase over the 2018 ILO figure. We recognize that the employer satisfaction rate may not be a wholly accurate predictor of employment rate, as the sample in the tracer study was limited to graduates that successfully found employment, presumably graduates of higher quality. We have therefore performed a sensitivity analysis to account for the possibility that quality improvements either do not result in any increase in long term employment prospects, and for completeness, if they improve prospects to a greater degree than the employer satisfaction rate would suggest. In the absence of data to the contrary, we have assumed that project beneficiaries would have the same salaries as the average tertiary graduate in Mali. The number assumed for this category is the full set of 36,392 students enrolled in the licensed programs in participating HEIs during the project period. Page 56 of 68 The World Bank Mali - Higher Education Support Project (P151318) Assumptions for cost-benefit analysis 11. Below are the key assumptions used: a. Earnings: A wage rate is estimated for different levels of education and age, using the 2018 EHCVM (Enquête Harmonisée sur les Conditions de Vie des Ménages ). b. Employment rate (chance of employment): Level specific unemployment rates were estimated for individuals with tertiary (or advanced) education, and those with upper secondary education (or intermediate) based on the ILOSTAT database. As described earlier, the employment rate for project beneficiaries was assumed to be 87.3 percent based on the employer satisfaction rate, to account for the difference in employment prospects due to the enhanced quality of programs offered by the HEIs supported by the Project. c. Discount rate: The discount rate used is 5.14 percent which is different from the 3.5 percent used at the PAD stage. We have used the latest available (2017) lending interest rate (%) for Mali from the International Monetary Fund - International Financial Statistics and data files. d. Inflation adjustment: A 3 percent inflation rate has been used to adjust for earning change overtime. We note that the inflation figure at the PAD stage was not stated clearly. e. Years in the labor market: Assuming the retirement age of 65, project beneficiaries remain in the labor market for up to 43 years following the training, assuming they graduate at the typical age of 22. f. Returns to education: The wage distribution for individuals with different levels of education based on the 2018 EHCVM has been used to determine the difference in lifetime earnings between upper secondary and tertiary education graduates. g. Exchange rate: The exchange rate used is 558 CFA per US$, which was the exchange rate effective January 31, 2015, at the start of the Project. h. Incremental benefits: Incremental benefits estimated by taking the average net present value of the estimated future earnings stream of project beneficiaries. i. Completion: Not all students that enroll in the HEI programs will graduate. The 2019 tracer study performed by ONEF observed that of 6,028 students who were supposed to have completed their bachelor’s between 2015 and 2018 contacted for possible involvement in the study, only 4,953 had received their certificates. We have therefore assumed a graduation rate of 82.2 percent for tertiary education graduates. 12. Table 4.2 below presents the results of the cost-benefit analysis at ICR stage . Both the IRR and NPV estimates show that the Project is economically viable at the end of the project implementation based on actual project beneficiaries. We have shown these estimates for sub-components 1.1 and 1.2 in isolation since the benefits for these are quantifiable. The benefits/cost ratio for these is 1.8, with the NPV of benefits being US$228.9 million, and a strong IRR of 12.5 percent. The benefits for the remaining components are not easily quantifiable. To obtain a lower-bound Page 57 of 68 The World Bank Mali - Higher Education Support Project (P151318) of the benefit-cost ratio, we factored in the costs of these other components but assumed no associated benefits. The benefit-cost ratio is 1.8 even with such an assumption. The analysis therefore strongly supports the assertion that the project offered good value for money for the beneficiaries. Table 4.2: Net Present Value in millions of USD and Internal Rate of Return (base case) Baseline: Employment rate increase by 6.3 percentage points due to the project, from 81% to 87.3% Sub-Components 1.1, 1.2 Total IRR 12.5% 11.9% Discounted cost (present value of costs) $273.2 $283.5 Project cost $15.1 $25.5 Forgone earning $258.1 $258.1 Present value of incremental benefits $502.1 $502.1 NPV (with 2015 as the base year) $228.9 $218.5 Benefit/cost ratio 1.8 1.8 Source: World Bank’s estimation based on assumptions as detailed above, and the Implementation Completion Report for Higher Education Support Project (Borrower’s Final Project Report, 2021). Ministère de l’Enseignement Supérieure et de la Recherche Scientifique. 13. Sensitivity analyses which vary the impact of the project on employment outcomes for graduates reaffirm that the project was efficient. Tables 4.3 and 4.4 below present sensitivity analyses for the cost-benefit analyses at the ICR stage. As noted earlier, we do not have a way of determining at this stage, the impact of higher program quality in the HEIs supported by the project on the employment rate of graduates. As a baseline (presented above) we assumed that the employment rate would correspond to the employer satisfaction rate as measured through the 2019 tracer study. However, we have also performed two sensitivity analyses as shown in the tables below. The first analysis assumes the worst, in that there would be no difference in the long-term employment rate of student beneficiaries of the project, compared to the global employment rate for the tertiary educated in Mali (81 percent in 2018 based on the ILOSTAT database). The benefit-cost ratio under such a scenario is still at least 1.1. The second analysis assumes a rosy scenario, where the project results in a 10-percentage point gain (on top of the 81 percent based on the ILOSTAT database) in employment rate for student beneficiaries relative to graduates of HEIs/programs not supported by the project. In this case, the benefit/cost ratio is obviously higher, at least 2.1. Table 4.3: Sensitivity analysis 1 (worst) Net Present Value in millions of USD and Internal Rate of Return Sub-Components 1.1, 1.2 Total IRR 6.9% 6.5% Discounted cost (present value of costs) $273.2 $283.5 Project cost $15.1 $25.5 Forgone earning $258.1 $258.1 Present value of incremental benefits $326.0 $326.0 Page 58 of 68 The World Bank Mali - Higher Education Support Project (P151318) NPV (with 2015 as the base year) $52.8 $42.5 Benefit/cost ratio 1.2 1.1 Table 4.4: Sensitivity analysis 2 (best) Net Present Value in millions of USD and Internal Rate of Return Sub-Components 1.1, 1.2 Total IRR 15.9% 15.0% Discounted cost (present value of costs) $273.2 $283.5 Project cost $15.1 $25.5 Forgone earning $258.1 $258.1 Present value of incremental benefits $605.5 $605.5 NPV (with 2015 as the base year) $332.3 $321.9 Benefit/cost ratio 2.2 2.1 14. The non-quantifiable benefits of the Project are also likely to be significant. Although component 2 accounted for a smaller share of costs relative to component 1, the benefits of this component could be far reaching. As described in the main body of the ICR, the Project developed a proof-of-concept framework to improve the governance of the higher education sector in Mali. If there are accompanying reforms to extend this model sector wide, all higher education institutions would benefit from a mechanism to sharpen program quality, invest in much needed infrastructure and equipment to deliver more and better programs, and thereby improve the employment prospects for their graduates. Our cost benefit analysis also only looked at the direct student beneficiaries that were enrolled in the programs during project implementation. Many more students will enroll in the same programs for years to come, and benefit from quality and relevant programs. The project has therefore set the stage for a marked improvement in the quality and relevance of higher education in Mali, which would benefit students in future cohorts, sector wide. Strong Financial Management Performance and Judicious Use of Project Resources 87. As earlier discussed, the FM performance rating was satisfactory throughout the project period except early in the Project when the staffing arrangements had not yet stabilized. These included: (i) regular meetings of the steering committee to review annual work programs and validate internal audit tools; (ii) timely and satisfactory submission of IFRs of an acceptable quality on a regular basis; (iii) submission of the required audits to the World Bank in accordance with the Financing Agreement in which the auditors’ opinions were unqualified (clean); and (iv) effective implementation of recommendations that were made by the external audit and Bank supervision missions. Annual financial statements were prepared by the PIU which complied with the Organization for the Harmonization of Business Law in Africa (OHADA) System (SYSCOHADA) which is a system used in West African Francophone countries. The final audit report is due on June 30, 2022, and will cover the period January 1-December 31, 2021. The total IDA project financing was US$33,000,000 equivalent with a total disbursement of approximately US$31.36 million equivalent. There was an undisbursed amount of US$1.46 million equivalent that will be returned to the World Bank and approximately US$0.18 million in US$/SDR exchange rate losses. 15. The Conseil d’Administration (CA) established in the public HEIs was charged with financial management and oversight, which helped ensure the judicious use of resources by HEIs. A key feature of the fund allocation Page 59 of 68 The World Bank Mali - Higher Education Support Project (P151318) mechanism piloted by the Project was increased autonomy for HEIs to manage a portion of their budget, in exchange for meeting certain performance-based conditions (PBC). The CA played the key role of ensuring that this autonomy was well managed in practice. Relevant CA responsibilities included discussing budgets and investment plans for their HEIs, and the preparation of regular audit reports. This helped to ensure sufficient financial oversight at the HEIs. Efficiency of Implementation and Overall Efficiency Rating 16. With a healthy benefits/cost ratio, and the net present value of benefits exceeding costs even after sensitivity analyses, the project demonstrated strong efficiency in investments undertaken during implementation . Substantial internal and external efficiency gains were observed over the life cycle of the Project and the majority of the PDO indicators successfully achieving and significantly improved if their targets were not achieved. Six of the six PDO indicators were met or exceeded and the 10 of the 11 IRIs were met or exceeded. However, few factors negatively impacted the overall implementation of the project and for this reason efficiency is rated Substantial rather than High. The ISR in December 2017 noted some delays in the implementation of quality assurance activities/regulations at the central level, which negatively impacted the ability of the National School of Engineers and the Rural Polytechnic Institute for Training and Applied Research to attain their DLI targets, but this was subsequently resolved. The closing date was extended by six months from December 31, 2020 to June 30, 2021, owing to delays caused by the COVID-19 pandemic and the August 18, 2020 coup d’état, and the resulting pause in disbursements slowed down the Project’s overall implementation progress. Despite these significant events, the HESP largely achieved all its intended outcomes. Based on the analysis presented herein, the overall efficiency rating under the project is Substantial. Page 60 of 68 The World Bank Mali - Higher Education Support Project (P151318) ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS In a confirmation email dated December 16, 2021, the Government confirmed its agreement to the findings of the ICR and had no additional comments. The Government provided their final completion report to the Bank which has been filed in WBDocs for information and reference. Page 61 of 68 The World Bank Mali - Higher Education Support Project (P151318) ANNEX 6. DISBURSEMENT LINKED INDICATORS NOTE: This table provides the list of DLIs, expected protocols for monitoring and disbursement of DLIs, and the original and actual disbursements for each DLI. The achievement of the DLIs was linked directly to the indicators in the results framework. Therefore, the achievement for the DLI is provided in the results framework. Disbursement Disbursement Method and Total Total Linked Indicator Linked Results definition of payment Original Actual Disbursement and Allocated Disbursement Amount per Project Supported HEI DLI 1. Improved Institutional Governance IPRK=US$150,000 ENI=US$200,000 USTTB=US$300,000 UOS=US$300,000 DLR 1: Functional Yes/No IPRK=US$50,000 IPRK=US$50,000 Board of Trustees ENI=US$50,000 ENI=US$50,000 (or University USTTB=US$100,000 USTTB=US$100,000 Council) of HEIs in UOS=US$100,000 UOS=US$100,000 place 4 DLR 1 Requirement Year 0 =1st Board of Trustees meeting that deliberates on the Institutional Year 0 Development Plan, the MOP, the new academic programs, the budget, and other aspects of the management of the HEI DLR 1 Monitoring The HEIs submit to the PIU the following documents: (i) documentation Protocol Year 0 formalizing the appointment of Board members; (ii) a convocation letter convening the Board meeting; (iii) copies of documents submitted to the Board; (iv) a list of attendance with signatures; (v) Board deliberations and approved minutes of the session; (vi) the rules of appointment and operation of the Board, and any official document related to the regulatory framework governing the Board. DLR 1 Years 1-4 = Two meetings of the Board of Trustees conducted that deliberate on Requirements the IDP, the MOP, the new academic programs, the budget, and other aspects of Years 1-4 the management of the HEI DLR 1 Monitoring The HEIs will submit to the PIU the following documents: (i) documentation Protocol Years 1-4 formalizing the appointment of Board members; (ii) a convocation Letter convening the Board meeting; (iii) copies of documents submitted to the Board; (iv) a list of attendance with signatures; (v) Board deliberations and approved Page 62 of 68 The World Bank Mali - Higher Education Support Project (P151318) minutes of the session; (vi) any revision that affects the Board operation adopted since May 1 of 2015, 2016, 2017, and 2018. . DLR 2: Adoption Yes/No IPRK=US$50,000 IPRK=US$50,000 and ENI=US$50,000 ENI=US$50,000 implementation of USTTB=US$100,000 USTTB=US$100,000 management tools UOS=US$100,000 UOS=US$100,000 and procedures of HEIs DLR 2 Requirement Year 0 = Adoption of HEI management tools and procedures Year 0 DLR 2 Monitoring HEIs will submit to the PIU the following documents: (i) the document submitting Protocol Year 0 the draft to the technical review of the national party in charge of proposing the document for approval; (ii) comments/opinions of the Technical Commission; (iii) the document submitting the MP to Board for approval and advice/comments from the Board; (iv) MP finalized taking into account the opinions collected; (v) approval document by the national body together with the final document; (vi) letter of transmittal and draft submitted to IDA for technical advice. DLR 2 Years 1-4 = Publication of statistics of the HEI and Information Management Requirements Systems (IMS) reports; organizational and financial audit report; remedial Years 1-4 measures proposed and validated and implementation report of the previous recommendations DLR 2 Monitoring HEIs will submit to the PIU the following documents: (i) report of implementation Protocol Years 1-4 of the management tools adopted including those for the effective management of enrollments and resources management system; (ii) hard copy and digital format of the statistical information of the previous year accompanied with projections for the following year; (iii) financial, accounting and administration audit report of the HEI for the past year; (iv) letter of transmittal and draft submitted to IDA for technical advice. DLR 3: Approval of A= Target set for the IPRK=US$50,000 IPRK=US$50,000 HEIs IDPs and number of renovated ENI=US$50,000 ENI=US$50,000 yearly operational or new programs USTTB=US$100,000 USTTB=US$100,000 budget with licensed and previously UOS=US$100,000 UOS=US$100,000 indicators approved by the Board with the CP recommendation X=$ (000) B=achieved number of programs If B