Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00004966 IMPLEMENTATION COMPLETION AND RESULTS REPORT ON A GRANT FROM THE GLOBAL PARTNERSHIP FOR EDUCATION IN THE AMOUNT OF US$30 MILLION TO THE FEDERAL DEMOCRATIC REPUBLIC OF ETHIOPIA FOR THE ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT December 27, 2019 Education Global Practice Africa Region CURRENCY EQUIVALENTS (Exchange Rate Effective Dec 27, 2019) Currency Unit = Ethiopian Birr (ETB) ETB 31.92 = US$1 FISCAL YEAR July 8 - July 7 Regional Vice President: Hafez M. H. Ghanem Country Director: Carolyn Turk Regional Director: Dena Ringold Practice Manager: Safaa El Tayeb El-Kogali Task Team Leaders: Girma Woldetsadik, Anna Olefir ICR Main Contributor: Suzana Nagele de Campos Abbott ABBREVIATIONS AND ACRONYMS CPF Country Partnership Framework CSA Central Statistical Agency CTE College of Teacher Education DLI Disbursement-linked Indicator DLR Disbursement-linked Result ECE Early Childhood Education ECERS Early Childhood Education Environment Rating Scale EEP Eligible Expenditure Program EERBF Education Results-Based Financing Project (the Project) EFY Ethiopian Fiscal Year EMIS Education Management Information Systems ESAA Education Statistics Annual Abstract ESDP V Education Sector Development Program V ESPES Enhancing Shared Prosperity through Equitable Services ESPIG Education Sector Program Implementation Grant ETWG Education Technical Working Group FM Financial Management FY Fiscal Year GCO GEQIP Coordination Office GEID General Education Inspection Directorate GEQIP General Education Quality Improvement Project GoE Government of Ethiopia GPE Global Partnership for Education GPI Gender Parity Index IBEX Integrated Budget and Expenditure IDA International Development Association IFR Interim Financial Report IPF Investment Project Financing OP Operational Policy PEFA Public Expenditure and Financial Accountability PFM Public Finance Management SIP School Improvement Program MoE Ministry of Education MoFEC Ministry of Finance NGO Nongovernmental Organization NPP National Procurement Procedures NPV Net Present Value OFAG Office of the Federal Auditor General PAD Project Appraisal Document PBS Protection of Basic Services PDO Project Development Objective PPSD Project Procurement Strategy for Development REB Regional Education Bureau SNNPR Southern Nations, Nationalities, and Peoples Region TABLE OF CONTENTS DATA SHEET .......................................................................................................................... 1 I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 5 A. CONTEXT AT APPRAISAL .........................................................................................................5 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) .......................................9 II. OUTCOME .................................................................................................................... 10 A. RELEVANCE OF PDOs ............................................................................................................ 10 B. ACHIEVEMENT OF PDOs (EFFICACY) ...................................................................................... 10 C. EFFICIENCY ........................................................................................................................... 18 D. JUSTIFICATION OF OVERALL OUTCOME RATING .................................................................... 20 E. OTHER OUTCOMES AND IMPACTS (IF ANY) ............................................................................ 20 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 21 A. KEY FACTORS DURING PREPARATION ................................................................................... 21 B. KEY FACTORS DURING IMPLEMENTATION ............................................................................. 23 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 25 A. QUALITY OF MONITORING AND EVALUATION (M&E) ............................................................ 25 B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE ..................................................... 26 C. BANK PERFORMANCE ........................................................................................................... 27 D. RISK TO DEVELOPMENT OUTCOME ....................................................................................... 27 V. LESSONS AND RECOMMENDATIONS ............................................................................. 28 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 30 ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................... 39 ANNEX 3. PROJECT COST BY COMPONENT ........................................................................... 41 ANNEX 4. EFFICIENCY ANALYSIS ........................................................................................... 42 ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ... 51 ANNEX 6. SUPPORTING DOCUMENTS (IF ANY) ..................................................................... 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name ETHIOPIA EDUCATION RESULTS BASED FINANCING P163608 PROJECT Country Financing Instrument Ethiopia Investment Project Financing Original EA Category Revised EA Category Not Required (C) Organizations Borrower Implementing Agency Federal Democratic Republic of Ethiopia Ministry of Education Project Development Objective (PDO) Original PDO The Project Development Objective is to contribute to improvement of learning conditions in primary schools in targeted regions including in pre-primary classes. Page 1 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) FINANCING Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Financing 30,000,000 30,000,000 21,801,212 TF-A5268 Total 30,000,000 30,000,000 21,801,212 Non-World Bank Financing 0 0 0 Total 0 0 0 Total Project Cost 30,000,000 30,000,000 21,801,212 KEY DATES Approval Effectiveness MTR Review Original Closing Actual Closing 29-Jun-2017 18-Jul-2017 03-Dec-2018 30-Jun-2019 30-Jun-2019 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions KEY RATINGS Outcome Bank Performance M&E Quality Moderately Satisfactory Satisfactory Substantial RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 05-Oct-2017 Satisfactory Satisfactory 0 02 03-Apr-2018 Satisfactory Satisfactory 3.00 03 28-Sep-2018 Satisfactory Moderately Satisfactory 9.60 04 07-Apr-2019 Moderately Satisfactory Moderately Satisfactory 9.60 Page 2 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) 05 28-Jun-2019 Moderately Satisfactory Moderately Satisfactory 9.60 SECTORS AND THEMES Sectors Major Sector/Sector (%) Education 100 Early Childhood Education 33 Public Administration - Education 17 Primary Education 50 Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Human Development and Gender 100 Gender 33 Education 100 Access to Education 100 Education Financing 100 Teachers 100 Education Governance, School-Based 33 Management Standards, Curriculum and Textbooks 33 ADM STAFF Role At Approval At ICR Regional Vice President: Makhtar Diop Hafez M. H. Ghanem Country Director: Carolyn Turk Carolyn Turk Director: Amit Dar Dena Ringold Practice Manager: Sajitha Bashir Safaa El Tayeb El-Kogali Task Team Leader(s): Anna Olefir, Girma Woldetsadik Girma Woldetsadik, Anna Olefir Page 3 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) Suzana Nagele de Campos ICR Contributing Author: Abbott Page 4 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES A. CONTEXT AT APPRAISAL Context 1. At the time of appraisal of the Education Results-Based Financing Project (EERBFP, the Project), Ethiopia had achieved impressive progress in economic, social and human development areas and toward achieving the Millennium Development Goals. Nevertheless, Ethiopia was still the 13th poorest country in the world and the strategies of the Government of Ethiopia (the GoE, the Government) all aimed to progress to a lower middle-income economy by 2025. With rapid economic growth in the past—Ethiopia’s real gross domestic product (GDP) grew by 10.9 percent annually from 2004 to 2014—Ethiopia’s economy was one of the fastest growing in the world. GDP was estimated to slow somewhat due to an El Niño drought, but still grow by a healthy 8.5 percent in 2015/2016. Sustaining this growth going forward would have to rely on the country’s human capital. The Government’s Education Sector Development Program V (ESDP V) 2016-2020 was designed to ensure improved quality education equitably across all levels and to provide schooling that was aligned with the needs of the country’s economy in line with the Government’s middle-income status aspirations. 2. The Government had demonstrated its commitment to investing in education. Public spending on education had increased by 70 percent in real terms between 2003/2004 and 2011/2012, with school enrollment through grade 12 increasing from 10 million to 19 million students during this period, and, further, to 25 million in the 2015/2016 school year. At the same time, the Government had focused on establishing the critical elements needed to improve students’ learning outcomes and completion rates. Funded by several donor partners, the Government’s flagship program to improve quality, the General Education Quality Improvement Project (GEQIP), then in its second phase (GEQIP II, P129828) focused on improvements in general education from preprimary to the secondary level1. It was financing activities in six areas: (a) curriculum implementation and textbook and learning material provision; (b) teacher and school leader development; (c) school improvement program (SIP); (d) strengthening system management and capacity building; (e) improving education quality through information and communication technology; and (f) program planning and coordination, monitoring, and evaluation. 3. Initial results of the quality improvement efforts were promising. GEQIP had increased the supply of qualified teachers, provided teacher training, textbooks and learning materials, and disbursed school grants. As a result, student- input ratios and the quality of inputs had improved markedly. Also, learning outcomes at the primary level had improved, albeit modestly, and the number of students achieving basic proficiency or higher in all subjects between 2011 and 2015 had increased for grades 4 to 8. These results were impressive in view of the huge increases in enrollment: Ethiopia had avoided deteriorations in education quality that often accompanies rapid expansion in access. 4. Progress had not been uniform, however, and there were persistent issues, including: 1 GEQIP II, estimated at US$550.0 million, was financed by several donors (he US Agency for International Development, the UK British Department for International Development, Education for All-Fast Track Initiative, Finland Ministry for Foreign Affairs, Italy’s Development Cooperation Department of the Ministry of Foreign Affairs, Norwegian Ministry of Foreign Affairs, and United Nations Children’s Fund) and an SDR85.8 million (US$130 million equivalent) credit from the International Development Association (P129828) approved on November 12, 2013. Page 5 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) • Low quality schools, especially in the Afar region: There were also large regional variations in school quality, as per Ethiopia’s system of measuring school performance by inspection2. Results from the inspection of about 22,200 primary and secondary schools in 2013/2014 showed wide variations in the percentage of Level 1 schools by region, ranging from 2.4 percent in Tigray to 46.5 percent in Afar. Schools in Afar were struggling to meet basic standards and required a targeted and sustained intervention to address the input-, process- and output- related challenges they faced. • Low quality of the pre-primary program (O-classes school readiness program aimed at 6-year old children): Despite a doubling of children enrolled in O-classes, the quality of preprimary was low. A key challenge was that most O-classes teachers had little or no training in child development and appropriate pedagogical approaches, negatively impacting the effectiveness of early childhood programs. • Equity, especially with respect to students with special needs and gender: The first priority for education in ESDP V focused on equity. But, in primary education only a very limited number of students with special needs had access: primary gross enrollment for these students was only 4 percent in 2014/2015. While significant progress had been made with respect to improving the gender parity index in both primary and secondary education, and in the hiring of female teachers, at the school management level, only 9.4 percent of primary school principals were women in the 2015/2016 academic year. ESDP V aimed to increase the share of school principals that are women to 20 percent by 2020, thereby helping to change pervasive preconceptions on the competencies of women as school leaders. • Efficiency, as reflected in high dropout rates from the primary grades, especially grade 1, and especially in the Southern Nations, Nationalities and Peoples Region (SNNPR): Although Ethiopia had made significant progress in primary school enrollment rates, efficiency continued to be plagued by high dropout rates. After grade 1, only 78 percent of students progressed to grade 2, with large regional variations. Although decreasing, the grade 1 dropout rate for SNNPR was 25.2 percent in 2014/15, a worrisome result given that this is the third most populous region in the country. 5. The EERBFP was to be processed as part of a US$100 million Education Sector Program Implementation Grant (ESPIG) initially envisaged as Additional Financing (AF) to the GEQIP-II (see para. 2). In line with the Global Partnership for Education (GPE) funding model, the variable part grant represents at least 30 percent (US$30 million in the case of ESPIG) and is conditional on achievement on a set of stretch targets established to improve education efficiency, equity and quality. The fixed part of ESPIG was approved on December 2, 2016 in the amount of US$70 million, as AF to GEQIP- II. However, the variable funding was only approved in February 2017 due to the request for more ambitious targets from the GPE than had originally been proposed, requiring additional discussions within the Government and with development partners in the context of the Education Technical Working Group (ETWG). Because of the different timeline and approach of the variable part grant vis-à-vis that of GEQIP II, a decision was made to proceed with a stand- alone project for the variable part, the EERBFP. 6. The US$30 million grant from the GPE was fully consistent with the World Bank Group’s Country Partnership Framework (CPF) for Ethiopia FY18-FY223, with GPE goals calling for inclusive and equitable quality education for all, and with the World Bank’s and donors partners’ education sector assistance strategy in support of Ethiopia’s ESDP V under GEQIP I and II. The Project supported the CPF’s Focus Area 2, Building resilience and inclusiveness, and its objective of 2 Ethiopia’s school inspection system includes annual school self-assessment and external inspection every three years. The inspection framework is robust and consists of seven standards on school inputs, fourteen standards on processes, and five standards on outcomes. Based on the aggregate percentage scores in these 26 standards, schools are classified into four levels of performance with Level 3 and Level 4 meeting the national standard (while Level 1 and Level 2 are below the standard). 3 World Bank, Country Partnership Framework for Ethiopia FY18-FY22, Report No. 119576-ET dated May 22, 2017. Page 6 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) “Improved basic education learning outcomes”. It also contributed to the World Bank’s Education 2020 strategy: “investing smartly” in education “investing early” and “investing for all”. Theory of Change (Results Chain) 7. The Project’s Theory of Change, reflect the pilot nature of the activities financed, and the overall expected outcome of developing tested models that could be scaled up to eventually improve learning outcomes nationally. The pilots were selected in response to an assessment of some of the key pressing needs in the most vulnerable areas of the country and were designed to complement the much larger GEQIP II nation-wide program of support and investment. For each of the pilots, there were plans for how to assess the results and draw lessons for future potential scale up. Page 7 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) Project Development Objectives (PDOs) 8. The Project Development Objective (PDO), as stated in the Project Appraisal Document (PAD), is to contribute to improvement of learning conditions in primary schools in targeted regions, including in pre-primary classes. The Project also had an important piloting and learning objective. It aimed to support the implementation of discrete pilot interventions in some of Ethiopia’s most educationally disadvantaged regions that could be used to develop and test models that could be scaled up nationally to address critical issues and improve learning conditions to support achievements in three domains: (a) learning outcomes; (b) equity; and (c) efficiency. In this sense, the Project was designed to benefit the broader education sector as a whole, and not only the targeted regions in which the somewhat unconnected interventions were to be implemented as described in the paragraph below. Key Expected Outcomes and Outcome Indicators 9. The following PDO-level Outcome Indicators were defined to measure contributions to improved learning conditions: • Share of low-performing primary schools in the Afar region, with the highest share of these schools decreased from 46.5 to 25.7 percent; • At least 90 percent of O-class teachers in the Benishangul-Gumuz region (>447) trained on newly developed accelerated ECE curriculum package with the end target of 90 percent; • At least 90 percent of O-class teachers in the Gambella region (>160) trained on newly developed accelerated ECE curriculum package with the end target of 90 percent; • Additionally-appointed trained female primary school principals, from 3,150 principals to 6,210; • Utilization and report by each region on supplementary school for children with special needs, reaching to utilization of 4 percent supplementary school grant; and • Grade 1 dropout rate in SNNPR decreased from 25.20 percent to 13 percent. 10. The Results Framework also defined five Intermediate Outcome Indicators, as follows: (a) Level 1 schools in Afar re-inspected; (b) New O-class curriculum package developed; (c) Female primary school principals trained (3,060 principals); (d) Special needs school grant guidelines updated; and (e) Mechanism in place to closely monitor and mitigate dropouts. Components 11. The grant was an Investment Project Financing (IPF) with a first ever results-based financing modality in the education sector under which disbursements were linked to the achievement of pre-agreed results. The EERBFP (using the results-based financing modality), and its sister operation GEQIP II and its AF (under the traditional disbursement method) were to complement each other to strengthen results for the sector while maintaining results clearly attributable to each project. The grant aimed to incentivize transformational effects in the education sector through pilots with ambitious stretch targets in line with the GPE funding model requirements (variable part grant). In line with a recommended practice by the GPE, several indicators had a disbursement rule, “clearly stating the conditions under which variable part financing can be (partially) disbursed”4. 12. The Project included three components, as described below. Financing under all of these components was to be disbursed against achievement of pre-defined disbursement-linked indicators (DLIs). 4 Guidance Note on GPE Variable Part Financing. Education sector planning and program design in the context of results-based financing. 2019. GPE, Washington DC. Page 8 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) Component 1: Improving Learning Conditions in Primary Schools in Targeted Regions (US$10 million) 13. Component 1 was designed to support the following: • Upgrading of 80 low-performing primary schools in the Afar region from Level 1 in inspection standards to Level 2. School improvement plans (SIPs) for Level 1 baseline schools in Afar would be collected and analyzed to identify and implement actions to improve school performance on standards that had been identified as inadequate in their 2015/2016 inspection results. This could include actions to improve performance on inputs, processes and outputs, such as lesson planning, assessment and feedback to students, monitoring of the SIP implementation, support for female and students with special needs, etc. • Development of a new Early Childhood Education (ECE) curriculum package for preprimary O-classes; and train at least 607 preprimary teachers and implement the newly developed ECE curriculum in the O-class setting in the Benishangul-Gumuz and Gambella regions. Component 2: Strengthening Equity and Inclusion in Education (US$10 million) 14. Component 2 was designed to support: • Enhancement of the gender balance in school leadership by training and appointing an additional 3,060 women primary school principals nationwide. Greater gender balance in school leadership would be promoted through prioritization of women applicants for school leadership training and their deployment/placement for new openings of school principal positions. • Development of inclusive learning environments by increasing the school grant allocation to the regions to support children with special needs and report on utilization of the special needs school grants. The Project would support an additional top-up of 4 percent of the overall school grant allocation to each region in the country in the 2017/2018 academic year to allow them to meaningfully support special needs students, thereby stimulating a positive trend in their enrollment and retention and change in attitudes towards these students. Component 3: Improving Internal Efficiency of Primary Schools in SNNPR (US$10 million) 15. Component 3 was designed to support a reduction in the grade 1 dropout rate in SNNPR to 13 percent in line with the ESDP V national target. To achieve this, the MoE would carry out the following activities to help improve attendance: (a) sensitize communities on the importance of student participation/attendance in early years of schooling; (b) support improvement of education quality through teacher training; and (c) strengthen the links between the SIP and the inspectorate system, thereby reinforcing a focus on improved school attendance/completion. School inspectorate results would be analyzed and used, with a particular focus at the woreda and school level, to implement specific actions required across the different school community contexts in SNNPR. B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) Revised PDOs and Outcome Targets 16. The Project’s PDO and outcome targets were not revised. Revised PDO Indicators 17. The Project’s PDO Indicators were not revised. Revised Components 18. The Project’s Components were not revised. Other Changes 19. N/A Page 9 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) Rationale for Changes and Their Implication on the Original Theory of Change 20. There were no changes to the Project. II. OUTCOME A. RELEVANCE OF PDOs Assessment of Relevance of PDOs and Rating 21. The Project’s PDO was relevant at the time of its approval and continues to be relevant at completion5. Given the Project’s short implementation period of two years, the World Bank Group’s CPF in place at the time of approval (Para. 6) is still under implementation at present. Therefore, the Project objectives are still highly relevant. GEQIP-II is expected to close on December 31, 2019, and a follow-on US$2.2 billion project, the General Education Quality Improvement Program for Equity (GEQIP-E, P163050), supported by Ethiopia’s donor partners and an SDR213.6 million (US$300 million equivalent) IDA grant and a US$140 million Multi-Donor Trust Fund Grant, was approved on December 19, 2017. GEQIP- E continues to support the Government’s ESDP V, and its ESDP VI, currently under preparation, that is expected to focus largely on the same themes, while expanding its priorities (e.g., equity, efficiency) given the large unfinished education sector agenda. Based on the above, Relevance of PDOs is rated High. B. ACHIEVEMENT OF PDOs (EFFICACY) 22. The Project’s progress towards the PDO of contributing to improved learning conditions is evaluated in terms of how the objectives were met as measured by the PDO indicators selected to measure progress. Progress towards associated disbursement-linked indicators that incentivized compliance is presented in Annex 4. 23. The Project had an important piloting and learning objective and aimed to support the implementation of discrete pilot interventions in some of Ethiopia’s most educationally disadvantaged regions. Implementation of the Project’s pilots aimed to inform future initiatives designed to improve learning conditions. This was especially relevant to the donors’ and World Bank’s support for ESDP V since the Project’s sister operation, GEQIP-II, and the follow-on GEQIP-E were still under implementation, and any lessons learned from these pilots could serve to adjust immediately similar activities and programs being supported under those projects. To reflect the above, the assessment of progress towards the PDO includes a discussion of the pilots’ implementation, the challenges faced, the lessons learned to refine the implementation model for scale-up and a brief discussion of how the implementation of each of these pilots has fed into the design and implementation of the Government’s ESDP. It is in the nature of pilots that some interventions succeed, and some interventions fail, and that the findings of the pilots are equally valuable in either scenario. Assessment of Achievement of Each Objective/Outcome Pilot Approaches under Outcome Area 1.1: Reducing the proportion of low-performing primary schools (Level 1 in inspection standards) in Afar region with highest share of these schools 24. The Project tested a comprehensive intervention model for upgrading primary schools from Level 1 to Level 2 in the Afar region, which had the highest shares of low-performing primary schools in the country according to 2013/14 and 5This section evaluates relevance, as relevance to the current World Bank CPF/CPS, in accordance the ICR Guidelines for IPF Operations dated September 27, 2018. Comments on the formulation of the Project’s PDO are provided in Section II B and, especially, Section III A. Page 10 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) 2014/15 inspection data. Out of the 385 primary schools that were inspected in the region in 2013/14 and 2014/15, 179 schools (46.5 percent) were at Level 1 (the lowest level). The following inspection of all 579 schools in Afar in the subsequent year revealed that the share of Level 1 schools was even worse at 68.9 percent6. The new data demonstrated that, with the exception of Somali, relative to other regions, schools in Afar were among the most disadvantaged in the country struggling to meet basic standards and therefore required a targeted and sustained intervention to address the input-, process-, and output-related challenges they faced. The Afar region is located in the northeastern part of the country and is home to around 1.8 million people, with a large rural, pastoralist population. The region is one of the hottest places on earth and one of the most geographically remote and inhospitable areas of the country . The pilot in Afar, therefore, represented a significant stretch. This first pilot was designed to reduce the share of low-performing schools in the Afar region, as measured by Ethiopia’s existing school inspection system which includes annual school self-assessment and external inspection every three years (para. 4). The experiences gained in the process of turning around the low- performing schools in Afar were considered to be very important for upgrading schools nationwide since the majority of Ethiopia’s primary and secondary schools do not meet the national standards. 25. Progress towards reducing the share of low-performing schools in the Afar region was Modest: the share of Level 1 schools in Afar fell from 46.5 percent (179 out of 385 inspected) in 2015/16 to 32.2 percent in 2017/18, an impressive improvement in such a short time, but still below the stretch target of 25.7 percent (PDO Indicator 1). As a result of these interventions, the 80 targeted most disadvantaged Level 1 schools in Afar were equipped with 160 blackboards, 196 teacher tables and chairs, and 3,680 student desks. Overall, 120 science kits, 243 book sets and 278 sets of supplementary reading materials together with book shelves were transferred and reading centers or reading corners have been organized and are functioning in targeted schools. In addition, water tanks were procured to support schooling at the time of droughts. Importantly, the schools have seen improvement on a number of inputs, processes and outputs standards, including on setting school’s vision and mission (standard 6); work with and participation of parents and local community (standard 26); preparation of participatory three-year strategic and one year annual school improvement plans (standard 7); organization of continuous professional development for teachers, directors and supervisors (standard 15); teaching and lesson planning and use of resources (standard 11). As a result, most schools that were Level 1 at baseline moved up the inspection performance ladder. Of those, 38 schools are close to the threshold of Level 2 schools with 45-49 percent composite inspection scores and are expected to be upgraded in the next few years (Table 1). These results were achieved by analyzing performance of schools on inspection standards and providing the schools with support to develop strategic and annual School Improvement Plans (a joint effort among school leadership, teachers, local authorities, community leaders, parents and students), and capacity building and learning materials and equipment needed to help improve performance as per the actions in the improvement plan. 26. Due to the composite nature of this indicator, the results described above underestimate progress towards upgrading schools. Inspection standards include input (7), process (14) and output (5) indicators. To improve from Level 1 to Level 2, schools need to comply with at least 50 percent of all indicators which vary in complexity, with some as complex as “adequate school infrastructure”, and the measurement of progress was binary—either schools meet at least 50 percent of the indicators or they do not. In terms of measuring results, there is not recognition for significant progress, i.e., over 45 percent indicator compliance. Schools were not very familiar with the inspection process as the standards had been in place for only about four years, and the schools under the Project had only gone through one, initial, baseline inspection. Because of this, the ICR mission obtained information on schools’ compliance with standards, broken down by the percent of standards complied with, as opposed to only the binary classification of whether they continued at Level 1 or were upgraded to Level 2. The results of this analysis are presented in Table 1 below: 6All level 1 baseline schools have been re-inspected. Re-inspections for the region in 2017/18 and 2018/19 currently cover only 60 percent of all schools, and therefore the comparison will be possible only once all re-inspections are completed. Page 11 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) Table 1: Compliance with School Inspection Standards (%) At Baseline (2015/16) Year 1 Year 2 – (2016/17) At Completion (2017/18) Percent Number of Percent of Number of Percent of Number of Percent of of Schools in Schools in Schools in Schools in Schools in Schools in Indicators Category Category Category Category Category Category Over 50 1 0.6 18 10.1 56 31.3 45 - 49 58 32.4 49 27.4 38 21.2 40 – 44 49 27.4 43 24.0 35 19.6 35 - 39 31 17.3 29 16.2 17 9.5 30 - 34 28 15.6 28 15.6 23 12.8 25 - 29 12 6.7 12 6.7 10 5.6 Under 25 0 0.0 0 0.0 0 0.0 179 100 179 100 179 100 Note: one school among 179 level I baseline schools appeared to be level II school, as became apparent as a result of verification in the field. Overall, 385 baseline schools have been inspected prior to the launch of the project. 27. Based on this additional analysis, a positive trend exists towards improvement of inspection standards’ indicators. From baseline to completion, schools complying with over 45 percent and with over 40 percent of inspection standards’ indicators increased from 33 to 52.5 percent, and from 60.4 percent to 72.1 percent, respectively. On the other side, schools complying with less than 35 percent and 30 percent of inspection standards’ indicators fell from 22.3 percent to 18.4 percent and 6.7 percent to 5.6 percent, respectively. As a result, the share of Level 1 schools is likely to decline in subsequent years as some of the schools might have moved up to Level 2. 28. Lessons learned and incorporated in further expansion. This pilot faced numerous challenges, not the least of which was due to the difficulties of working in the remote, arid area of the country. Nevertheless, the overall progress was very positive, and the lessons learned serve as very relevant input for a further expansion of this pilot to other regions on a national scale. Specific lessons include: (a) different realities among regions require more targeted approach to standards, e.g., construction standards in other regions are not necessarily relevant to Afar where remoteness and very hot and challenging climate conditions pose special challenges (for example the students dropping out of school during droughts); (b) many schools showed improvement in standards following implementation of action plans including the procurement of water tanks and basic school furniture, and while this was not done to the extent required to improve in classification levels, the schools did move up the inspection performance ladder; (c) similarly, indicators and targets, and planned implementation periods need to consider the difficulties when working in unfavorable environments; (d) it was difficult for schools to comply with the re-inspection process as they had only gone through the process once before7; (e) woreda inspections should be supported by the REB Inspection Directorate with sample re-inspections to ensure the robustness of inspection results and to strengthen the capacity of woreda inspectors and to provide assistance to schools; and REB inspectors should be trained by the MoE; and (f) data quality plays an important role in verification process. The Project built upon efforts under GEQIP-II to support schools’ capacity building through the process of developing School Improvement Plans and linking the priorities they contained to improve learning outcomes. Following a review of lessons under this pilot, this school improvement process is being supported nationwide under the GEQIP-E, with an emphasis on improving the quality of preparation of school improvement plans and timeliness of receipt of funding for their 7Inspection Framework was introduced in 2013 and the schools had their first inspection within the three-year period of 2014- 2016. Page 12 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) implementation, and with the plans adapted to reflect regional realities. Pilot Approaches under Outcome Area 1.2. Improving the learning environment of O-classes (pre-primary class) in Benishangul-Gumuz and Gambella regions by increasing the capacity of O-class teachers to deliver an early childhood education curriculum package 29. The second pilot was to introduce preprimary O-Classes in the Benishangul-Gumuz and Gambella regions (two pilots, one in each region, related to different DLIs)8. The selection of the Benishangul-Gumuz and Gambella regions was made in light of their strong engagement and efforts in the design and implementation of formal and non-formal early childhood care and education services and was endorsed by the ETWG. The number of children enrolled in O-classes in Ethiopia doubled within the three-year period prior to the Project and continued to grow. At the same time, almost all teachers in these classes had little or no training in child development and appropriate pedagogical approaches. This negatively affected the effectiveness of early childhood programs in the country. The support and expansion of quality preprimary programs is important. Several studies in Ethiopia show that attendance in preprimary is a significant factor for learning progress, school attendance and promotion, and helps reduce dropout and repetition rates—chronical issues that Ethiopia’s education system currently faces. In Ethiopia, children who had been to preschool had progressed further through primary school by 0.218 of a grade, on average, than those who had not. Also, among student background and household characteristics (such as reading books at home, having three meals a day, etc.), attending preschool is an important factor for learning progress in reading. Children who attend preschool tend to have better cognitive skills later in life than non-preschool peers: 8-year-olds who have attended pre-school scored 43.7 percent higher in vocabulary tests and 51.1 percent higher in the cognitive test than those who did not attend ECE services9. Urban preschool children are 25.7 per cent more likely to complete secondary education than their non-preschool counterparts at the proper age10. 30. Progress towards introducing preprimary O-Classes in the Benishangul-Gumuz and Gambella regions is rated High in the pilots in both regions. A successful, new O-class curriculum package was developed (Intermediate Results Indicator 2), distributed and implemented. Activities included the development of a child-friendly, play-based curriculum, provision of teaching-learning materials and teacher guides in local languages (in five languages in Benishangul-Gamuz). The Project also supported developing a region-wide program of in-service teacher training for O-Class teachers, developing a curriculum package of teaching and learning materials, training of trainers, and delivering the intensive one-month training to teachers in Benishangul-Gumuz and Gambella. To support the expansion of the O-class program, a greater number of teachers than originally planned received training on the curriculum in both Benishangul-Gumuz and Gambella regions: in Benishangul-Gumuz, 489 O-class teachers were trained against a target of 447 O-class teachers, and in Gambella, 218 O-class teachers were trained against a target of 160 O-class teachers. More than 90 percent of the O- classes in both of these regions have implemented the newly developed curriculum (PDO Indicators 2 and 3). The marginal returns are higher for those who attended preschool for two and three years than for those who only attended for one year, and the Government plans to expand the O-class program accordingly11. 31. Lessons learned and incorporated in further expansion. The O-Class curriculum and materials developed under the Project led to a rapid expansion of O-Classes at the national level. To address this rapid expansion, GEQIP-E built upon 8 The O-class one-year school readiness program is aimed at 6-year-olds and delivered in primary school compounds. 9 Woldehanna T., Gebremedhin L. 2012. “The Effects of Pre-school Attendance on the Cognitive Development of Urban Children aged 5 and 8 Years, Evidence from Ethiopia.” Young Lives, University of Oxford. 10 Woldehanna, T. and Araya, M. (2017) Early Investment in Preschool and Completion of Secondary Education in Ethiopia: Lessons from Young Lives, Working Paper 168, Oxford: Young Lives. 11 Ibid. Page 13 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) the Project’s efforts by introducing a Quality Enhancement and Assurance Program that will provide a comprehensive package of interventions to systemically improve the quality of O-Class provision under a coherent framework of support. It will be implemented in about 2,000 schools initially (Phase I) and subsequently rolled out over the following two years to reach 50 percent of all schools. These further efforts consider the following lessons learned under the Project: (a) the need to raise awareness at the REB level and provide technical support at that level; (b) adapting the curriculum and materials into diverse local languages pose special challenges, including the need to find qualified translators, capable to adapt materials to the local context; (c) the poor data quality resulted initially in an underestimation of the number of schools in the regions---good quality data is fundamental; (d) in addition to training O-Class teachers, there is a need to focus also on principals and other leadership to ensure that O-Class is integrated in the school environment and not seen as a separate program; (e) as a corollary to the previous lesson, school leadership needs to be accountable for O-Class requirements (e.g., budgeting, planning, supervising, etc.) as part of their responsibilities; (f) attention is required to class size—some schools require more than one O-Class; (g) strong involvement of the college of teachers’ education is needed in training O-Class teachers for the institutions to own the training; and (h) strong community and parental involvement is required to promote participation in O-Class for six-year old children (and now five-year old children as well). Pilot Approaches under Outcome Area 2.1. Addressing the gender balance in school leadership by increasing the number of trained female primary school principals 32. This pilot aimed at enhancing the gender balance in school leadership by increasing the number of trained female primary school principals. This was a transformational target in light of the societal barriers which women face in the education profession in Ethiopia, directly supporting the commitments to improve gender balance within the civil service as articulated in the national Growth and Transformation Plan 2. Although Ethiopia had made significant progress in gender equity in primary education, school leadership was still largely male--only 9.4 percent of all school principals were women in the 2015/16 academic year. Increasing women in leadership positions was being promoted to provide girls with models to help them excel and, themselves aspire to leadership positions, as well as to make schools more sensitive to gender-specific needs of girls. The verification of results encompassed a qualitative analysis of the impact of female school principals on girls’ experiences in primary schools and their perceptions of the changes this has brought in relation to their retention and engagement within school. The results from the focus group discussions (FGDs) were structured according to the following themes: role model effect, counseling, girls’ participation in extracurricular activities, educational outcomes, gender-based violence and early marriage. One key finding from the FGDs is that female school leaders empower girls, support their self-confidence and motivate them to work harder and aspire for better future (play a role model for girls). Also, girls more openly discuss their problems with female school leaders than male leaders. The FGD findings also revealed that female student participation in extracurricular activities such as girls’ clubs, remedial classes and other school activities increased with the assignment of a female principal or deputy principal. Female school leaders tend to advise girls to continue their schooling; and female leader role was also acknowledged by FGD participants in relation to prevention of gender-based violence and early marriage, in tight collaboration with the gender clubs. 33. Progress towards appointing additional trained female primary school principals was Substantial. Activities under this pilot, supported also by GEQIP-II, included preparing materials for training and assigning of future principals (including an Implementation Manual distributed to all parties involved with objectives of the program, expected outcomes, roles of the MoE, of the Universities and of the REBs, and budget), incorporating input from about 18 universities that later prepared and delivered the three-month residential training program. Potential female school leaders were selected from among teachers in primary schools. Trained females were to be assigned as replacements to principals in existing schools as needed and in newly opened schools. By completion, an additional 2,124 trained females had been appointed as primary school principals or vice-principals, increasing the total number nationwide from 3,150 at baseline to 5,274 upon Page 14 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) completion, against a target of 6,210 (PDO Indictor 4). 34. Lessons learned and incorporated in further expansion. Several challenges affected this pilot’s implementation. First, the Implementation Manual stated that the trained females should be assigned to leadership positions either in the same or nearby schools. Given limitations on open school leadership positions, this was not always possible, and most leadership trainees were not willing to relocate far from their homes. Hence, there was an issue with the availability of positions to be filled by newly trained potential leaders. When planning the target for the number of positions to be filled, it is important to consider expected turnover as well as the creation of new positions. Second, Ethiopia’s civil service regulations require that vacancies in public sector positions be open to competition, that is by both males and females alike. Given the Government’s commitment to gender equality—the President’s cabinet is divided equally among both genders—the MoE has been working with the civil service office to rationalize the need to promote more female leaders (by interpreting the Constitution’s mandate for gender equality in government) and is looking to assign more leadership positions as and when needed to have more vacancies at the school level that can be filled by women. Third, in those regions in which there had never been women school leaders, cultural influences are strong, and it has been hard to change the population’s mindset. Also, some experienced teachers preferred continuing their work as teachers. Further, the work burden in terms of hours per week of school leaders is significantly higher than that of teachers (normally work about 20 to 22 hours per week), yet the salary is not commensurate. The MoE is exploring the possibility of adjusting salary scales to address this inconsistency. Fifth, partially related to the lack of information on the part of leadership trainees, and to the added work burden, there has been some turnover of female leaders that have been assigned as principals or vice-principals. Although relatively small, 34 out of 1,432 trainees assigned to leadership positions in the Project’s second year have left to other positions in the REBs, the woreda level, etc. with positive externalities on gender equality at these levels, beyond solely improved gender balance in school leadership envisaged by the project. Also, while this is not scientifically corroborated, on a very positive note, regional data from the Afar region reports that schools that have female directors have managed to increase the enrollment of girls. 35. Considering the lessons and challenges described above, the MoE has decided to expand the program of training female teachers for leadership positions and given the growing student population aims to have a pool of potential female candidates for the future. In all, it aims to train 10,000 teachers at the primary level, and 500 teachers at the secondary level given that this program is being expanded to that level. Pilot Approaches under Outcome Area 2.2. Encouraging more inclusive learning environments by increasing the school grant allocation to support children with special needs 36. This pilot aimed to promote a more inclusive learning environment by providing critically needed financial support for special needs education at the school level through a 4 percent top-up of each region’s overall school grant allocation. Many children with disabilities (special needs) in Ethiopia are yet to be integrated in the education system. According to the Education Statistics Annual Abstract only 8.1 percent of children with special needs were enrolled in primary schools in 2015/16. Considering this, the ESDP V for 2016–2020 underlined the importance of increasing the provision of inclusive education in mainstream schools, including in the one-year school readiness program aimed at 6-year-olds and delivered in primary school compounds. 37. Progress towards the pilot for allocation, utilizing and reporting on use of the 4 percent supplementary school grant to support education of children with special needs is High. All of the regions have utilized and reported on the utilization of this grant (PDO Indicator 5). The MoE revised the special needs school grant guidelines (Intermediate Results Indicator 4), increasing the top-up to the previously existing 1 percent grant. The increased school grants were transferred Page 15 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) to the regions, and the regions reported utilization of the grants to support more inclusive learning environments for students with special needs. The 4 percent top-up to the school grant was disbursed to all regions to create a more inclusive learning environment. Regions were encouraged to be innovative in the allocation and utilization of these additional grants. 38. Lessons learned and incorporated in further expansion. A qualitative survey sampling beneficiaries’ schools in six regions (e.g., Harar, Ethiopia Somali, Addis Ababa, Afar, Tigray and Gambella) and later in five remaining regions (e.g., Dire Dawa, Benishangul-Gumuz, Amhara, Oromia, and SNNP) revealed that regions and schools demonstrated a wide variety of ways for using the grant. Regions allocated funding to invite medical experts to assess students’ hearing and sight, to provide learning materials to students with special needs or channeled the resources to woredas and schools, based on EMIS data, or to targeted schools. Funding disbursed to schools were used for purchase of additional learning materials (30 percent), assistive devices for individual needs (20 percent) and/or training and awareness activities (16 percent). The perception of the program’s effectiveness on enrollment and retention rates of special needs children varies with 46 percent of principals responding that it is very effective, 36 percent reporting that it is somewhat effective and 70 percent reporting that it made it easier for them to meet the needs of students with disabilities. The REBs found that the amount allocated was too small to cover costs of supporting children with special needs, but that this financing was instrumental in raising the awareness for special educational needs and helped regions implement different strategies to strengthen their inclusive education programs. 39. Building upon the experience under the Project, GEQIP-E is supporting the provision of supplementary school grants to transform 687 cluster center schools to inclusive education resource centers (IERCs) to promote mainstreaming of children with special needs in education. The 4 percent top-up has been modified to supplementary grants that will be used to: (a) organize awareness raising and training events; (b) purchase reference materials on special needs, equipment for assessing needs (e.g. embossers, audiometers), and educational resources for children with special needs (sign language dictionary, slate and stylus and tactile learning materials); and (c) ensure accessible learning environment (e.g., minor refurbishment). A set of eligibility and selection criteria will be used to select IERCs with the aim to maximize the number of beneficiaries. This approach is expected to address the issue of the relatively small amount of resources allocated to some schools while promoting efficiency in resource usage through economies of scale, thereby leading to the sustainability in the use of resources. Pilot Approaches under Outcome Area 3.1. Reducing Grade 1 dropout rates in Southern Nations, Nationalities and Peoples region with the highest grade 1 dropout rate 40. The internal efficiency of Ethiopia’s primary schools is plagued by high dropout rates, which are especially acute after grade 1: only an estimated 78 percent of students progressed to grade 2. Although Ethiopia has made encouraging progress in reducing dropout rates in the last few years, the dropout rates, especially in grade 1, are still stubbornly high. This was particularly true in SNNPR (with the highest grade 1 dropout rate in 2013/14 at 25.2 percent). The SNNPR is an ethnically diverse region, home to around 19 million people, which was subject to social unrest during the second year of implementation. This pilot aimed at promoting a reduction in the grade 1 dropout rate in SNNPR, the region with the highest grade 1 dropout rate. The grade 1 dropout rate in SNNPR decreased from 25.20 percent in 2014/15 to 16.61 percent in 2017/18, somewhat short of the ambitious stretch target of 13 percent (PDO Indicator 6) and progress towards the pilot objective is rated as Substantial. Further, SNNPR made significant progress in reducing the dropout rate before the unrest, rapidly approaching the national grade 1 dropout rate. More importantly, by 2018/19 the SNNPR grade 1 dropout rate had fallen significantly below that of the national level. The figures for grade 1 dropout rates at the national level and in SNNPR are presented in Table 2. Page 16 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) Table 2: Grade 1 Dropout Rates in SNNPR and at the National Level (%) 2014/2015 2016/2017 2017/2018 2018/2019 National 21.4 18.1 19.5 25.0 SNNPR 25.2 22.5 16.61 16.8 41. The progress is the result of both the dropouts monitoring and mitigation interventions implemented by the MoE, the REB, woredas and schools, and better registration of new enrollees, repeaters and re-admitters at the school level. To improve school attendance and ensure children’s successful transition and retention in Grade 1, the RBE sensitized communities on the importance of student participation/attendance in early years of schooling through awareness raising and working with parent associations, supported improvement of education quality through teacher training and strengthened the links between the School Improvement Plans and the inspectorate system, thereby reinforcing a focus on improved school attendance/completion (Intermediate Results Indicator 5). To impact the dropout rate, the REB provided resources for initiatives aimed at retaining children in class, including school feeding, support for purchase of uniforms for those in need, training staff at the REB, woreda and school levels, and identifying areas where grade 1 dropout was most prevalent to focus special attention in those areas. At school level, Asmelash committees (comprised of parents, teachers and community members) worked with out-of-school children and their parents to ensure that those who dropped out of school re-entered and continued their education. These activities were largely successful in reducing grade 1 dropout during the first year. However, during the Project’s second year of implementation, SNNPR witnessed social unrest that led to the closure of schools and the displacement of people, and to a further deterioration in the reliability of weak data used to measure progress, inter alia because of difficulties in delivering updated data. 42. Lessons learned and incorporated in further expansion. The lessons learned from this pilot are mostly related to the fact that SNNPR is the most heterogenous region in the country and that it was subject to social unrest during implementation. They include: (a) instability (e.g., social unrest leading to school closure, displacement of communities, economic consequences, etc.) can disrupt any efforts to improve efficiency and/or educational outcomes, but a project could not fully mitigate this risk during the short lifetime of the project; (b) partly because of costs the regions cannot fully provide initiatives such as school feeding aimed at promoting demand for schooling; (c) instability also leads to high turnover in educational institutions such as the REB, making data collection and monitoring more difficult; (d) there were issues with grade 1 dropout data in that it did not take into account transfers to other schools; and (e) there is a need to identify the REBs’ capacity for data collection and provide technical support where needed. 43. The MoE continues to assign high priority to increasing internal efficiency in primary by reducing dropout and absenteeism in primary (and secondary) schools. Supported by GEQIP-E, this objective is now being pursued at the national level through school-based monitoring of student attendance, and incentives for schools to improve performance on two specific internal efficiency indicators: the ratio of grade 2 enrolment to grade 1 enrolment and increase in survival rates to grade 5. The school-based monitoring system by and large provides continuity to the model developed under the Project for the SNNPR pilot: targeted communications to raise awareness of the issues related to grade repetition and readmission, absenteeism, and dropout. The results of monitoring activities at the school, woreda, and regional levels will be captured in school report cards that will be publicly disseminated. Further, the MoE will provide annual awards to the top 10 percent of primary schools nationwide ranked in accordance with a formula that calculates year-to-year improvements in the indicators mentioned above. Page 17 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) Justification of Overall Efficacy Rating 45. Overall efficacy is rated Substantial, based on the High achievement of three pilots, the Substantial achievement of two pilots and the Modest achievement of one pilot. These results are even more impressive given the Project’s pilot nature, the stretch targets encouraged by GPE, and the short implementation period – an extension of the Closing Date was not granted but would have allowed greater progress during implementation. This is relevant since as per the ICR Guidelines, efficacy is defined in terms of whether a project has achieved its objectives (intended outcomes) or is likely to do so, and two of the pilots are very likely to meet the established targets in the coming year(s). While a certain reasonableness is expected in terms of an expected timeframe for “likely to do so”, the Project is making important progress towards these two pilot targets. Specifically, in addition to 55 upgraded schools in Afar (DLI 1), most baseline level I schools have gained experience with the inspection process and standards, have moved up the inspection performance ladder, and 38 schools (close to the threshold of level II) are expected to be upgraded in the next couple of years. Also, the female teachers trained under the Project are expected to be appointed to leadership positions in the coming years, as vacancies become available. The education sector has a large role to play in promoting the gender equality ambitions and commitments of the Government, following the Prime Minister’s announcement in 2018 of the gender progressive Cabinet which comprises 50 percent female Ministers – an unprecedented and historical push for gender equality in the country12. Over and above these achievements, the implementation of these pilots under the Project has already had a major impact on the design and implementation of important education sector programs (e.g. upgrading of schools, training of preprimary teachers and other efforts to improve learning conditions in schools) as described above. C. EFFICIENCY Assessment of Efficiency and Rating 46. Economic Efficiency. At appraisal, the preparation team conducted a cost-benefit analysis and internal rates of returns (IRRs) and net present values (NPVs) were calculated under three scenarios (low, base and high case scenarios). The analysis calculated the cost and benefit streams over a time span of 20 years from 2017 to 2036. The analysis was conducted for the last results area (reducing grade 1 dropout rates in the SNNP region) only given the absence of evidence for the remaining results areas for Ethiopia which, consequently, were not part of the estimations. At the appraisal stage, IRR estimates ranged from a low of 13.7 percent to a high of 15.7 percent. Similarly, NPV estimates ranged from approximately US$213.1 million to US$318.6 million. 47. A simple ex-post cost benefit analysis has been carried out for the ICR that focuses on the results area aimed to reduce grade 1 dropout rates in SNNP region (Outcome Area 3.1) based on similar methodology and assumptions as used at appraisal, updated to reflect actual numbers resulting from implementation with several assumptions. The full analysis is presented in Annex 4, and its conclusion is summarized below. The IRR of costs and benefits of this ex-post analysis shows that the Project was economically viable upon completion, based on data from actual number of project beneficiaries. Notwithstanding the less than expected disbursements over the Project’s life, and the slightly lower than targeted achievement in the reduction of the grade 1 dropout rate, the results from the cost-benefit analysis continue to show that the Project remains economically viable. The corresponding NPV of Project benefits is US$183.2 million, 12 Source: https://www.globalcitizen.org/en/content/ethiopia-cabinet-ministers-women-abiy-ahmed/ Page 18 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) somewhat lower than the US$264.1 million estimated at appraisal; the IRR associated with this NPV is 12.1 percent, which is lower than the estimated IRR at Appraisal (14.7 percent); and the benefit-cost ratio of monetary benefits deriving from the reduction in grade 1 dropout rate to be 2.5 compared to 3.1 at Appraisal. Further, the analysis captures only part of the Project’s benefits (those that are quantifiable) and therefore provides the lower bound for its economic efficiency. 48. The principal benefits from upgrading the schools under Outcome Area 1.1 stem from the fact that individuals are expected to be more employable and productive by gaining access to better quality schooling and developing relevant skills which are required to participate productively in life. Quality of schooling directly affects individual earnings and economic growth.13 The principal benefits from supporting quality O class program including its expansion under Outcome Area 1.2 stem from improved cognitive development, better schooling outcomes and increased productivity in life. As mentioned above, in Ethiopia 8-year-olds who have attended pre-school scored 43.7 percent higher in vocabulary tests and 51.1 percent higher in the cognitive test than those who did not attend ECE services14. Likewise, preschoolers are 38 percent more likely to be proficient readers than non-preschoolers15. Children from urban areas are 25.7 per cent more likely to complete secondary education than their non-preschool counterparts at the proper age16. Investments in ECD yield long-lasting high returns. A growing body of literature demonstrates that the returns to investments in children’s early years are substantial, particularly when compared to equivalent investments made later in life; and high returns to investments in ECD have been consistently delivered across a range of contexts, with a typical rate of return between 2:1 and 8:1 per dollar invested17. Addressing the gender balance in school leadership (Outcome Area 2.1) supports girls’ educational attainment contributes and in a long-term contributes to women’s economic empowerment and more inclusive economic growth. Developing more inclusive learning environments (Outcome Area 2.2), including in low- income countries brings better social, academic, health and economic opportunities to children with disabilities than attending segregated schools, benefitting them individually but also contributing to their families, communities and society at large18. There are positive externalities associated with higher educational attainment. There are also other important individual gains from investing in education; nonetheless, many of them were not quantified in this analysis due to the lack of data for Ethiopia (such as returns to skills), and therefore, this estimation is to be considered a conservative estimation of the benefits of the Project due to data limitations. The Project’s economic efficiency is rated Substantial. 49. Implementation Efficiency. The Project’s implementation efficiency is difficult to evaluate for two reasons: the stretch targets required by GPE that serve to incentivize strong performance, and the short implementation period (two years) in which those targets were to be achieved. Further, despite the request from the Government, the Grant’s Closing Date of June 30, 2019 was not extended, so no extra time to achieve the stretch targets was given. However, the stretch targets and short implementation period were known before hand and implementation efficiency will be evaluated with that as a reference. The Project’s implementation was off to a strong start during Year 1, but delays in procuring inputs for upgrading of schools in Afar in line with school improvement plans (including water tanks, blackboards and basic 13 Hanushek, Eric A.; Woessmann, Ludger. 2007. The Role of Education Quality for Economic Growth. Policy Research Working Paper; No. 4122. World Bank, Washington, DC. World Bank. 14 Woldehanna T., Gebremedhin L. 2012. “The Effects of Pre-school Attendance on the Cognitive Development of Urban Children aged 5- and 8-Years Evidence from Ethiopia.” Young Lives, University of Oxford. 15 Kim J. 2019. Unpacking the Role of Early Learning in Student Learning Outcomes: Evidence from National Reform of Pre-Primary Education in Ethiopia, University of Cambridge. 16 Woldehanna, T. and Araya, M. (2017) Early Investment in Preschool and Completion of Secondary Education in Ethiopia: Lessons from Young Lives, Working Paper 168, Oxford: Young Lives. 17 Neuman, Michelle J.; Devercelli, Amanda E.. 2013. What Matters Most for Early Childhood Development: A Framework Paper. Systems Approach for Better Education Results (SABER) Working Paper Series No. 5. World Bank, Washington, DC. World Bank. 18 The Economic Costs of Exclusion and Gains of Inclusion of People with Disabilities. Evidence from Low- and Middle-Income Countries. 2015. London. Page 19 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) furniture) and O-class materials (and consequent delivery delays), issues with data collection from various schools and REBs and delivery to the MoE and resulting reporting delays began to affect implementation and disbursement of grant funds. Because the Closing Date of June 30, 2019 was not extended, the verification of activities completed by June 30, 2019 was conducted during the grace period, extended to December 31, 2019. As of November 30, 2019, US$21.8 million out of US$30 million grant was disbursed for attained results, while US$8.2 million remained undisbursed. The undisbursed amount related to moderate shortcomings in the achievement of expected outcomes on upgrading of schools in Afar, to the appointment of female school leaders and of very ambitious dropout reduction target in SNNPR (the first two outcomes are expected to be achieved after the Grant’s Closing Date as explained above). DLIs are priced based on needed incentives for implementation, not the actual costs to implement, and disbursements are made only when the results are achieved and verified. The MoE made incentives flow to the responsible actors including REBs, colleges of teacher education (CTEs), woredas and schools and communicated DLIs at the service delivery level. The final amounts disbursed, by DLI, are presented in Table 4.3 (Annex 4). In light of their importance, the stretch targets were incentivized by higher DLI amounts. The Government was able to meet the targets with a considerably lower amount of funding. In effect, the Project achieved Substantial Efficacy with less than full disbursements from the Grant, which were complemented by Government funding. The Project’s implementation efficiency is considered Substantial as pilot interventions were highly efficient in providing valuable inputs and lessons into the country-wide programs being developed and implemented under the GEQIP series, and in light of the ambitious stretch targets by the GPE funding model and short duration of the first ever results-based financing in the sector, which required changing the mindset of the Ministry, REBs, woredas and schools, with steep learning curve on verification of results. Assessment of Efficiency and Rating 50. The Project’s Efficiency is rated Substantial, based on its strong economic efficiency, and efficient implementation. There were some shortcomings as discussed above related to the somewhat lower rates of return than estimated at appraisal for one of the pilots and to some more minor delays incurred in the provision of inputs to the pilots. D. JUSTIFICATION OF OVERALL OUTCOME RATING 51. Overall outcome is rated Moderately Satisfactory based on a High rating for Relevance, a Substantial rating for Efficacy, and a Substantial rating for Efficiency, and the moderate shortcomings in the achievement of expected outcomes under the Project’s pilot for Outcome Areas 1.1, 2.1 and 3.1. E. OTHER OUTCOMES AND IMPACTS (IF ANY) Gender 52. The Government has made significant progress in reducing the gender gap in both primary and secondary education. Hence, the Project did not have a specific objective of improving gender balance among students. At the school management level, however, only about 9.4 percent of school principals were women in 2015/16. Therefore, the Project aimed to address gender by increasing the number of females in managerial positions in schools. It was expected that this would improve perceptions about women’s potential as school leaders, and hence have a positive impact on enrollment, attendance and performance of female students. Progress towards this is discussed under PDO Indicator 4, in para. 33. Page 20 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) Institutional Strengthening 53. The Project did not have a specific institutional strengthening objective, per se, as the institutional strengthening activities were provided mostly within the framework of GEQIP-II. It did, however, serve to introduce the use of results- based financing both at the MoE at the national level and among RBEs, woredas and schools. Changing the mind-set of officials at all levels took time since the new approach required a fundamental change (e.g. monitoring mattered, timeliness mattered, etc.) from traditional input-based financing. Continuous implementation support, and follow-up and coordination by the MoE were instrumental in making the new results-based approach successful, and in helping strengthen some, if not all, RBEs and a handful of schools to implement a broader and increased amount of activities under the GEQIP-E Program for Results operation. Mobilizing Private Sector Financing 54. N/A Poverty Reduction and Shared Prosperity 55. The Project did not have a specific poverty reduction objective. Although regional disparities in consumption levels and poverty are limited in Ethiopia, the country’s pastoral regions lag on human development outcomes. These pastoral regions include four of the regions under the Project: Afar, Beninshangul-Gamuz, SNNP and Gambella, where boys’ net attendance in primary school was 62.6 percent, 81.3 percent, 75.9 percent and 86.9, respectively, against a figure of 98.2 in Addis Ababa. Corresponding figures for girls’ net attendance in primary school for these regions were 63.7 percent, 72.8 percent, 76.3 percent, 89.6 percent in Afar, Beninshangul-Gamuz, SNNP and Gambella, respectively, against a figure of 92.1 in Addis Ababa. Other Unintended Outcomes and Impacts 56. The Project’s unintended impacts, which were in reality expected but not made explicit nor considered in its expected outcomes, related to the significant progress made in incorporating its lessons in the Government’s ongoing activities under its ESDP. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION 57. Preparation within the context of sectoral assistance program by the WB. The World Bank, together with Ethiopia’s other donor partners, have provided consistent support to the Government’s national strategy for education since 2009. The General Education Quality Improvement Program (GEQIP), funded under a pooled arrangement by IDA, a Multi-Donor Trust Fund (MDTF) 19, and the Global Partnership for Education (GP) was originally approved as a two-phase Adaptable Program Loan (APL) in support of the Government’s ESDP IV that translated the Government’s strategic goals into a program of action for the education sector. This multi-phase approach was deemed appropriate since the expected 19The MDTF is funded by the United Kingdom’s Department for International Development (DFID), Finland, Italy , Norway, the Netherlands, the United National Children’s Fund and the United States Agency for International Development. Page 21 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) implementation period of a single project was not considered to provide enough time to yield quality improvements. With the extinction of the APL instrument in the Investment Lending Reform of 2013, the second phase of GEQIP was processed as a stand-alone Investment Project Finance. At the time, GEQIP-I was witnessing progress towards improvement of quality education through the enhancement of learning conditions. GEQIP-II was approved on November 12, 2013, with a continuation of pooled funding by donor partners, to provide continued support to ESDP IV. 58. GPE Funding. The ERBFP, which was the GPE variable part grant disbursing against DLIs (stretch indicators in GPE’s terminology), had originally been planned as part of a US$100 million Education Sector Program Implementation Grant (ESPIG), originally envisaged as Additional Financing to GEQIP-II. The fixed part of the GPE Grant (US$70 million as AF to GEQIP-II) was approved on December 2, 2016. The remaining 30 percent (US$30 million in financing) was to be a variable part grant according to GPE’s funding model. Discussions regarding the stretch targets were prolonged to ensure even more ambitious stretch targets, and after approval of the World Bank as the Grant Agent for the variable part grant in March 2017, the World Bank made the decision to proceed with a stand-alone project (the ERBFP) for the grant in light of the different implementation period (GEQIP-II was planned to close earlier than the variable part grant) and approach (results-based financing) vis-à-vis that of GEQIP-II. 59. Coordination with donors in the context of integrated assistance. The Project was included within the framework of the donor coordination in the context of the ESDP, including the planning for ESDP V (2015/16-2019/20), the ESDP Joint Review Meeting in April and the Annual Education Conference in November of each year. As part of ESDP coordination, all donors supporting education participate in the Ethiopia Education Technical Working Group (ETWG) that meets monthly and reports to the MoE’s Director of Planning and Resource Mobilization Directorate and a representative of a rotating donor partner. 60. Decision to proceed with “pilot” approach to test mechanisms to address key issues. The ERBFP was designed to address serious problems in some of Ethiopia’s most educationally disadvantaged regions. In doing so, it would help identify, implement and monitor five, discrete interventions to serve as pilots in different regions that could then be used to develop models to be scaled up nationally to address those issues on a much broader scale. 61. “Framing” of PDOs and Project Components. The Project’s PDOs, as formulated, gives the impression that the activities and programs it financed would contribute to improved learning conditions in selected project-supported regions by supporting a package of activities to this end. In reality, because of its “pilot” nature described above, the Project financed discrete activities that varied in each of the several regions in which it operated; some of the pilots were national in scope (e.g., appointing female primary school principals). As a result, although primary and pre-primary schools in each of the regions were expected to witness some improvements in learning conditions, this would be because a single, differing initiative would be implemented in each region as a pilot. It was not that all initiatives would be implemented as a package in all regions, although as mentioned above, some were national in scope. This interpretation is corroborated by the Project’s PDO indicators which are region- and initiative-specific (and which are used to evaluate progress to this narrower interpretation of the PDO as included in the PAD). 62. Therefore, a more suitable wording for the PDO would have been, “to develop, pilot and learn from different initiatives that could contribute to improved learning conditions in both selected regions and nationally to inform the Government’s ESDP”. Similarly, Project component titles should have highlighted the pilot nature of the initiatives that it financed. For example, Component 1 (Improving Learning Conditions in Primary Schools in Targeted Regions) would have been better titled “Piloting Different Approaches in Targeted Regions and Nationally to Help Improve Learning Conditions in Primary and Pre-Primary Education”. This broader interpretation of the Project’s PDO is also used to evaluate progress, Page 22 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) in addition to that supported by the Project’s Results Framework, as described above. 63. First application of results-based financing. This was the first application of results-based financing for Ethiopia’s education sector. This approach was expected to incentivize the Government to deliver transformational education outcomes. It was also seen to provide a unique opportunity to test and transition towards the results-based financing model that was being considered for the follow-on sectoral operation that was being prepared under the Program for Results modality. 64. Incorporation of lessons learned. The Project’s design incorporated important lessons, especially with respect to the implementation of results-based operations. Of these, perhaps the most important was that of the need for strong monitoring and evaluation arrangements and very precise protocols to determine the achievement of DLIs, agreed and endorsed up front to ensure transparency in their measurement and to avoid diverging interpretations of achievement. This lesson, together with the requirement that technical aspects of the verification exercise be discussed and agreed with the ETWG, were incorporated very well in the Project’s design. 65. Identification of risks. Project preparation correctly identified risks, and to the extent possible, actions to mitigate the risks both during preparation and later in implementation. During preparation, the team conducted consultations with stakeholders from the central and regional governments to ensure understanding of and build ownership for the Project. Also, the ETWG was fully engaged in the selection of the Project’s DLIs. Finally, the Project’s stretch targets were closely aligned to activities under the ongoing GEQIP-II and were complementary to help achieve ESDP V targets. B. KEY FACTORS DURING IMPLEMENTATION 66. Implementation Timeline. The Grant was approved by the GPE Board on February 15, 2017, then by the World Bank on June 29, 2017, signed on July 17, 2017 and became effective on July 18, 2017. Project activities were completed by June 30, 2019, and confirmation of all disbursements against DLIs were finalized following the verification and endorsement of the verification reports by October 10, 2019 (US$21.8 million), with US$8.2 million remaining undisbursed. The implementation support team has stated that it believes that more could have been achieved, and hence a greater amount disbursed, if the Project had been granted additional time through a Closing Date extension, and especially, if its implementation had not been affected by the conflict in SNNPR. 67. Government commitment. The Government’s commitment to its ESDP V, and hence to the GEQIP series and the Project was present throughout, even following the change in Government leadership in early 2018. There was commitment from the national and regional governments to pursuing improvements in learning conditions and subsequently in educational outcomes in some of the country’s most educationally disadvantaged regions, but data collection from REBs faced continuing difficulties and delays. Also, the follow-on ESDP VI, originally scheduled for approval by August 2019, is still work in progress. On the positive side, the Government is working on a number of strategic documents which are nearing completion, including an education sector roadmap through 2030 and the drafting of a new Education Act. 68. Donor coordination. For the ERBFP, the ETWG routinely considered decisions with respect to compliance with DLIs and corresponding authorization for disbursement of grant funds. Ethiopia is considered a model of harmonized donor coordination, and the donors in the education sector through the ETWG have been consistently supporting capacity building of government systems. This is important as funding from the GEQIP series and from the Project are channeled through existing government financial management and other systems. Page 23 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) 69. Support of CSA for results verification. The Implementation Support team counted upon valuable input provided by the Central Statistical Agency (CSA) that was selected as the independent verification agency. Attainment of DLIs was verified by CSA. CSA was provided with very detailed instructions for the Technical Aspects of the Verification Exercise for the Terms of Reference for Services of Independent Verification of Performance/Attainment of the Disbursement- Linked Indicators. For each DLI, CSA prepared reports with: (a) the detailed description of verification activities carried out to check the accuracy of results reported by the implementing agency; (b) verified results for all DLIs along with recommended disbursement levels for each DLI; (c) highlights of major discrepancies between reported and verified results along with explanations for the differences; (d) critical analysis for why achievement against specific DLIs could be above or below targets, major barriers and constraints that impeded progress, and recommendations and plausible solutions for the attainment of the agreed targets. CSA presented its findings to the MoE, the development partners, including the World Bank, at ETWG meetings. Because of issues with the quality of data (para. 71), CSA verification was an important factor to enable disbursements. 70. Implementation at the Regional level. The Project’s pilots were implemented in some of Ethiopia’s most remote, pastoral and ethnically-diverse regions. This was known at appraisal, and the impact of social and political developments on the possible disruption of education service delivery was rated Substantial. There are few, if any, measures that can be applied to mitigate this risk short of working in different regions. Violence in the country significantly increased and spread to new areas, and four times as many people were displaced in 2018 compared to 2017 according to the Internal Displacement Monitoring Center. In effect, social unrest in SNNPR affected the results of PDO Indicator 6 due to social unrest that began during the second year of implementation and is still ongoing. Because of the conflict, schools were closed, and even when they were open, many children did not attend classes. The conflict led to the internal displacement of many persons, including students and teachers. Also, support was required from the educational system, including in the form of psychological support for teachers and students alike. Even in Regions that were not affected by conflict, implementation was challenging and subject to delays in delivery of materials due to their remoteness, difficulties in communications, etc. (e.g., Afar). Importantly, the project made significant achievements in several areas in time of rising ethnic tensions, violence over border disputes and ongoing internal displacement in the country. 71. Continuity in Project Coordination and among Education Sector Projects. Project coordination was provided by an office (the GEQIP-II Coordination Office) headed by a Project Coordinator responding to the State Minister for General Education in the MoE. The Project Coordinator was responsible for coordinating project activities both at the MoE at the national level as well as by project RBEs for pilots implemented at the regional levels, for coordinating financial management, disbursements, procurement, monitoring and evaluation and other project-related initiatives, representing the projects at the ETWG meetings and submitting periodic reports to the World Bank. More importantly, this office provided not only considerable continuity in coordination of education sector projects--the Project Coordinator has been in this position since 2009, when GEQIP-I was prepared, and is responsible for coordinating preparation and implementation of the ongoing GEQIP-II, the Project, and the follow-on GEQIP-E—but ensured complementarity among activities under each of these projects. 72. Progress in implementation of GEQIP series, and support to Project objectives. Both GEQIP II and GEQIP-E projects had a synergistic relationship with the Project, in which they not only provided the overall framework to help guide the Project’s implementation (including in the procurement of needed goods and materials) and the sectoral dialogue and coordination to help it succeed, but they also benefited from the pilot experiences that the Project was implementing. The ETWG and the Project’s coordination by the GEQIP II Coordination Office helped to reap the benefits of this synergy. Page 24 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E) M&E Design 73. The Project’s Results Framework was succinct, and well-designed, with six PDO Indicators and four Intermediate Results Indicators, all of which contained baseline data, and Year 1 and End-of-Project targets. They were also well aligned with the Project’s objectives, although there was some ambiguity with the formulation of the PDO itself (para. 61). The PDO indicators were very well aligned with the DLIs that were defined to trigger disbursements. Detailed protocols were elaborated for each of the Project’s DLIs, including the stretch indicators, the baseline values, the targets (Year 1, Year 2, end-of-project), the disbursement rule (whether disbursement could be prorated) and the indicator verification process (including inspection timing process and sample). These were described in detail in Annex 1a of the Project Appraisal Document. The targets were very ambitious in line with the GPE’s variable grant stretch model. Further, the indicator for appointment of school principals did not consider the fact that appointment depended upon the availability of vacancies, which turned out to be less than originally reported by the REBs. 74. The GEQIP-II Coordination Office, including the two Task Forces and the concerned Directorates in the MoE, together with the REBs, were responsible for monitoring activities and results in the project regions and submission of reports on indicators to the MoE’s Directorate of Planning and Resource Mobilization Directorate. At the regional level, REBs were also responsible for monitoring project activities, and together with city administrations for field visits and follow-up on implementation of DLIs in their respective woredas and schools. The CSA was responsible for verifying the attainment of DLIs (para. 69). Broader education indicators were captured by and monitored on the basis of information in the Government’s Education Management Information System (EMIS) that was being strengthened under GEQIP -II; project- specific indicators (e.g., teachers trained on new O-Class curriculum) were reported on the basis of information provided to the MoE by the REBs. M&E Implementation 75. The Project’s Results Framework was not adjusted during implementation. The CSA provided scheduled reports often with delay and presented the results of their findings to the ETWG. However, as this was the first results-based financing in education, there was a very steep learning curve for both the MoE and the CSA, as changing existing processes takes time. As a result, the World Bank’s implementation support team had to provide intensive, hands -on assistance, especially with verification processes. The data quality was a concern, mainly due to manual data collection and entry at the school and woreda levels, and difficulties by the woredas in delivering data to the REB. The conflict in SNNPR also led to delays in data collection as the schools had problems in travelling to the woreda level to deliver data. Nevertheless, the World Bank’s implementation support team, and the local donor group were able to rely upon the verified data both for monitoring and disbursement purposes, and were confident that the verified data represented actual project achievements. M&E Utilization 76. Dependable monitoring data is especially important for results-based financing operations, as monitoring data becomes the basis for disbursements of project funding. The findings of verification reports also provided recommendations to improve implementation performance, that can feed into the design and implementation of similar programs in other regions in Ethiopia. Page 25 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) Justification of Overall Rating of Quality of M&E 77. The Overall Rating of Quality of M&E is Substantial, based on a rating of Substantial for design, Modest for Implementation, and High for Utilization. B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE 78. Environmental Compliance. The Project was classified as Environmental Assessment (EA) Category C Project. As a result, none of the World Bank policies related to environmental safeguards were triggered. The activities financed by the Project also did not have any issue of environmental safeguards concern. 79. Social Compliance. The Project was implemented mostly in pastoral and agro-pastoral areas of Ethiopia, including regional states of Afar, Gambella, Benishangul-Gamuz and SNNPR. The social assessment carried out during preparation of GEQIP-II (that finances complementary activities), assessed the potential impact of that Project on the more vulnerable and disadvantaged groups and emerging regions, and identified strategies for mitigating the respective risks. Conclusions and recommendations of GEQIP-II’s social assessment were followed and implemented under the RBF Project, including for example providing O-Class education in local languages in pastoral and agro-pastoral areas and providing materials, training teachers, providing support to children with special needs, and incentivizing girls’ education through increased appointment of female school principals. GEQIP-II Social Assessment implementation performance review was conducted, and action plans agreed. The EERBF supported the implementation of the performance assessment action plan. 80. Fiduciary Compliance. The Project was to rely on the country’s public financial management (PFM) system for the budget process, classification system and compliance with financial regulations, and for the expenditure management and control (that was benefiting from substantial technical assistance), and internal control. Structured as a results-based financing project, disbursements would be triggered upon compliance with DLIs, reviewed twice yearly, and made against expenses under eligible expenditure programs (EEPs). The Project’s recurrent expenditure EEPs related to two -line items of the MoE’s budget codes: the budget line items of 2100000 (Compensation to Employees) and 2200000 (Goods and Services). A financial management (FM) review was carried out for the Project four times during its implementation. Visits were made to the Ministry of Finance (MoF), the MoE, selected regional Bureaus of Finances (BoFs) and REBs. The World Bank’s reviews focused on the Project’s EEP and reviewed procedures for budgeting, accounting, internal control, funds flow, financial reporting and external audit. The Project submitted all quarterly financial reports required but with some delay. The audit report for the ministry as a whole was received on May 6, 2019, with a month delay and contained an adverse audit opinion on the MoE’s financial statements (raising issues on documentation of several expenditures and internal control weaknesses relating to weak documentation, weak monitoring of advance and payable balances as well as unsupported variation orders on various constructions). The Ministry responded to the findings, but the auditors indicated they require further documentation to conclude that the expenditure is justified properly. However, the auditor’s findings on the MoE’s audit relate mostly to construction of universities that are outside the scope of this Project. The World Bank requested an update on the audit by June 30, 2019, and as a result of the review, excluded questioned expenditures from the Project’s EEP and downgraded the FM rating to Moderately Satisfactory. As a lesson learned, defining EEPs is very critical in an IPF with DLIs. The EEPs should be well thought through and risks arising from the line items should be mitigated as much as possible. 81. RBF project was a results-based financing project disbursing on the basis of EEPs covering recurrent expenditures, including procurable recurrent expenditures of small contract values. That said, procurement under the Project was largely carried out under the sister operation GEQIP II (p. 40 of the PAD) to ensure synergies and complementarities needed to strengthen the results for the education sector as a whole while maintaining distinguishable results attributable to each Page 26 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) of the projects. C. BANK PERFORMANCE Quality at Entry 82. Bank Performance for Quality at Entry is rated Satisfactory. The preparation team worked with GPE and donors to design a contained project that piloted approaches to address some of the most pressing sectoral issues in educationally disadvantaged regions, all within the framework of previous and ongoing World Bank assistance to the Government’s ESDP through the GEQIP series. The Project and GEQIP-II were sister operations that complemented each other to strengthen the results for the overall sector while maintaining clearly distinguishable results attributable to each project. The Project’s approaches were designed to later be analyzed and adjusted to broader implementation throughout the country. At the same time, the team faced preparation of the first results-based financing in Ethiopia’s education sector. The DLI protocols were detailed, well defined both in terms of what they aimed to achieve, and also in terms of how they would be measured for disbursements to materialize. Further, they coincided with the Project’s PDO and results framework (which was contained and well-defined and contained baseline data and targets for each indicator). On the negative side, the prepared project contained in some cases very ambitious stretch targets as required by GPE, and/or an implementation period that was too short in which to achieve them. However, the World Bank made a conscious decision to proceed with this results framework given the associated external financing and the fact that these could be seen as incentivizing achievement of results. Also, the Project’s PDO did not capture fully its objective to design, pilot and learn from different initiatives aimed to improve learning conditions. On balance, however, the positive contribution of the World Bank’s preparation team far outweighed the negatives. Quality of Supervision 83. Bank Performance for Quality of Supervision is rated Satisfactory. Although the implementation support team was supported by the CSA and the ETWG, its task to provide support for the first results-based financing operation for education was not easy. The different approach to financing required a change in the mind-set of MoE, CSA, regional, woreda and school staff that were accustomed to traditional investment financing, together with continued hand-holding to keep the Project on track. The task team responsible for the Project’s implementation support was the same team responsible for implementation support of GEQIP-II and for preparation of the follow-on GEQIP-E. This approach ensured that the Project’s implementation experience and lessons learned were incorporated in the design and implementation of GEQIP-E, which is also a DLI-based operation and the Government’s Flagship Program supporting the implementation of ESDP V. Not only were experiences with different approaches to sectoral issues incorporated as lessons learned, but the results-based approach was fundamental in the preparation and implementation of the sector’s first Program for Results operation. This team was led by two task managers, one headquarters- and the other country office-based. Justification of Overall Rating of Bank Performance 84. The Overall Rating of Bank Performance is Satisfactory based on similar ratings for Quality at Entry and Quality of Supervision. D. RISK TO DEVELOPMENT OUTCOME 85. The Project financed six, distinct, initiatives, in different educationally disadvantaged regions and some, nationwide. The Government, at preparation, had planned that the Project’s pilots would feed into a nationwide rollout of the program for upgrading the low-performing schools, for reducing grade 1 dropouts and the revised O-class program, providing school grants to support children with special needs and building capacity for school management. Therefore, the risk to Page 27 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) development outcome of these initiatives depends not only on the commitment and initiatives of the respective Regional Education Bureaus in each of these regions, but also upon the enabling environment at the sector level to internalize, build upon and expand these initiatives to the education system and other regions, as well. At the regional level, risks are higher, but these are mitigated mostly by the overall sectoral context through: (a) Government commitment; (b) continued donor funding, at least through 2021, under the umbrella of the GEQIP series; and (c) existing processes for strong donor coordination. The Government’s commitment to education, and more specifically, improvements in access, efficiency, equity and quality continues to be very strong, as it is for improving learning outcomes and sectoral management through better systems. ESDP V is nearing completion, and ESDP VI is expected to be approved by June 2020 and focus on the same themes, including capacity development for improved management, general education quality and general education access, equity and internal efficiency in light of the large unfinished agenda. Following an extension to complete remaining project activities, GEQIP II is closing on December 31, 2019. A new General Education Quality Improvement Program for Equity (GEQIP-E), a Program for Results financing in the amount of US$440 million equivalent was approved on December 19, 2017, again with pooled funding from donors and IDA, to continue supporting the ESDP series. GEQIP-II focused on improving learning conditions and strengthening sectoral institutions; GEQIP-E maintained this focus and expanded focus to improving efficiency and equitable access—both priority programs in ESDP. Existing processes for donor coordination, namely the functioning of the ETWG remain strong, with most sectoral decisions financed by donors channeled through the group for consideration. Although the Government has been contributing an increasing amount of budget resources to the education sector, an important risk, therefore, would be that donors withhold funding, although this is not likely in the near future. Another important risk, and one that is difficult to mitigate, is the impact of ethnic conflicts and violence which affected implementation of the Project in SNPPR and continues to affect implementation of ESDP activities in conflict-affected areas nationwide. Ethiopia is now estimated to have approximately 3.1 million internally displaced people (IDPs), making it the country with the highest number of IDPs in the world. The ongoing conflict in West and Central Gonder, Benshangul Gumuze regional state, West Wellega and West Guji zones of Oromia, Gedeo, Amaro, Basketo zones in SNNPR, and continued violence in the Afar-Somali border disrupt children’s education. Therefore, the Risk to Development Outcome is Substantial. V. LESSONS AND RECOMMENDATIONS 86. The Project had a strong emphasis on learning from pilots, and in view of the importance of learning, the lessons learned from the design and implementation of each of the pilots are presented together with the achievement of results from each of the pilots in Section II B. Some of the lessons were positive, others were negative - but these too have provided important feedback that has influenced the design of the Government’s further roll-out of the programs supported by the pilots. This section presents broader lessons with respect to project design and implementation, including the Project’s design as a results-based financing operation. 87. An open and fluid relationship between an implementing agency and the independent verification agency supports provision of valuable inputs needed to promote improvements in work processes. It greatly facilitated the work of the World Bank’s implementation support, and of the Project Coordination Unit in the MoE. 88. Composite indicators (e.g., achieving Level 2 standards that implies progress on input, process and output indicators) are tricky as when using the composite indicator to evaluate progress towards objectives (e.g., as a PDO Indicator), important details regarding that progress may be lost. For example, a project may make 95 percent progress towards the total of all sub-indicators, but unless 100 percent is achieved a composite indicator will not recognize progress. Hence, a composite indicator, if warranted, should include individual sub-indicators that monitor progress towards expected outcomes as defined in the PDO. Composite indicators may be useful as DLIs—especially if they are Page 28 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) already established and monitored composite indicators--in that they facilitate disbursement, but monitoring of sub- indicators is preferred for evaluating overall progress. 89. The GPE funding model with highly incentivized stretch indicators presents its own unique challenges in terms of implementation that needs to be taken into consideration in evaluations of these projects. Stretch indicators can be extremely valuable in incentivizing strong performance. Still, they may reflect targets that are almost impossible to achieve and measure, especially when implementation is decentralized across several regions and when used in a project involving a major change in the modus operandi (e.g., traditional disbursement to results-based financing approach which requires changing mindset and capacity building for verification). Further, in a project implemented in diverse regions subject to distinct realities, these stretch targets and their related timelines need to take into consideration region-specific peculiarities such as variety of languages, topography, remoteness, etc. 90. Related to the above, there is an interesting choice to be made in selecting regions for piloting different approaches, and project design needs to consider whether pilots should be carried out in regions with the worst performance or those with somewhat better performance. In the case of the Project, two of the pilots were implemented in regions with poor if not the poorest performance for the activities and indicators being piloted: the pilot for improving school standards in Afar and the pilot for reducing grade 1 dropouts in SNNPR. In both, regional characteristics affected performance, although it did not affect achievement of important progress. In Afar, remoteness affected the delivery of goods and subsequent verification, and in SNNPR, conflict affected efforts to keep children in school. Both of these extremes provided valuable input for expanding their pilot experience to other regions. But the question remains, would having implemented the pilot in a more average performing region have provided more lessons that would then be applicable to the poorest performing regions. In any event, conflict spreads further, the education projects in Ethiopia would need to include interventions aimed at mitigating the impact of conflicts, including interventions that help address psychological trauma of children and teachers, catch-up classes, development interventions for both internally displaced and host populations. 91. Pilot operations, such as those under the Project, can provide a powerful vehicle when complemented by continuous assistance under broader projects supporting sectoral programs and reforms. The GEQIP II and GEQIP-E projects supported the Project’s activities and processes such as project coordination, procurement, monitoring, etc. At the same time, the Project provided valuable inputs and lessons into the various country-wide programs being developed and implemented under the GEQIP series. Together, the Project and GEQIP II and GEQIP-E had a very strong synergistic effect that was very relevant in the face of continuity in Government commitment and that would not have materialized had the projects not been implemented simultaneously. 92. Establishing DLIs with associated disbursement amounts and PDO and Intermediate Indicators in results-based financing operations present several challenges. DLIs are often dated, and even if they are not or if specific dates are not binding, disbursements related to DLIs are ultimately constrained by a financing agreement’s Closing Date. Compliance with targets for PDO Indicators are not similarly constrained as it does not involve the cutoff that applies to DLIs. So, while a results-based operation may not meet all of its DLIs in time for full disbursement, it may still achieve its objectives as measured by its results indicators. 93. The design and implementation of results-based financing operations requires a strong effort on the part of the World Bank’s task team to assist the implementing agency. Results-based financing operations require a change in the mindset of implementing agency. Further, significant support is required during implementation to understand the .requirements of monitoring and reporting, and the importance of verification in the disbursement process. Page 29 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: to contribute to improvement of learning conditions in primary schools in targeted regions Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Share of low performing Percentage 46.50 25.70 32.20 primary schools in Afar region (of baseline schools in the 15-Mar-2017 15-Jun-2017 26-Jun-2019 region) Comments (achievements against targets): The share of Level 1 schools in Afar fell from 46.5 percent (179 out of 385 inspected) in 2015/16 to 32.2 percent in 2017/18, an impressive improvement in such a short time, but still below the stretch target of 25.7 percent. This pilot faced numerous challenges, not the least of which was due to the difficulties of working in the remote, arid area of the country. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion At least 90 percent of O-class Percentage 0.00 90.00 100.00 Page 30 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) teachers in Benishangul- 15-Mar-2017 15-Jun-2017 31-Aug-2018 Gumuz region (> 447) trained on newly developed accelerated ECE curriculum package Comments (achievements against targets): In Benishangul-Gumuz, 489 O-class teachers were trained against a target of 447 O-class teachers, which is more than 100% achievement. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion At least 90 percent of O-class Percentage 0.00 90.00 100.00 teachers in Gambella (> 160) trained on newly developed 15-Mar-2017 15-Jun-2017 22-Mar-2019 accelerated ECE curriculum package Comments (achievements against targets): In Gambella, 218 O-class teachers were trained against a target of 160 O-class teachers, which is more than 100% achievement. Page 31 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Additionally appointed Number 3150.00 6210.00 5274.00 trained female primary school principals and deputy 15-Mar-2017 15-Jun-2017 26-Jun-2019 principals Comments (achievements against targets): An additional 2,124 trained females had been appointed as primary school principals or vice-principals, increasing the total number nationwide from 3,150 at baseline to 5,274 upon completion, against a target of 6,210. Given limitations on open school leadership positions, and most leadership trainees were not willing to relocate far from their homes, the target was not achieved. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Utilization and report by each Text 1% supplementary detailed report on detailed report on region on suplementary school grant (no utilization of 4% utilization of 4% school grant support for report on utilization) supplementary school supplementary school children with special needs grant grant 15-Mar-2017 15-Jun-2017 17-Sep-2018 Comments (achievements against targets): All of the regions have utilized and reported on the use of the four percent supplementary school grant to support education of children with special needs. Page 32 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Grade 1 dropout rates in Percentage 25.20 13.00 16.61 SNNP region 30-Jun-2016 15-Jun-2017 22-Mar-2019 Comments (achievements against targets): The grade 1 dropout rate in SNNPR decreased from 25.20 percent in 2014/15 to 16.61 percent in 2017/18, somewhat short of the ambitious stretch target of 13 percent. SNNPR is the most heterogenous region in the country and that it was subject to social unrest during implementation, affecting efforts to reduce dropouts. A.2 Intermediate Results Indicators Component: 1: Improving Learning Conditions in Primary Schools in Targeted Regions Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Level 1 schools in Afar re- Yes/No N Y Y inspected 15-Mar-2017 15-Jun-2017 19-Mar-2018 Comments (achievements against targets): Page 33 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion New O-class curriculum Yes/No N Y Y package developed 15-Mar-2017 15-Jun-2017 19-Mar-2018 Comments (achievements against targets): Component: 2: Strengthening Equity and Inclusion in Education Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Female primary school Number 0.00 3060.00 2124.00 principals trained 15-Mar-2017 15-Jun-2017 17-Sep-2018 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Special needs school grant Yes/No N Y Y guidelines updated 15-Mar-2017 15-Jun-2017 19-Mar-2018 Page 34 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) Comments (achievements against targets): Component: 3: Improving Internal Efficiency of Primary Schools in SNNP Region Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Mechanism in place to Text No timely mechanism Dropout rates are Dropout rates are closely monitor and mitigate since EMIS data is determined using data determined using data dropouts lagged by one school disaggregated at the disaggregated at the year level of schools and level of schools and used in policy making used in policy making 15-Mar-2017 15-Jun-2017 17-Sep-2018 Comments (achievements against targets): Page 35 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) B. KEY OUTPUTS BY COMPONENT 1. Component 1: Improving Learning Conditions in Primary Schools in Targeted Regions. The principal objective of this component was to contribute to improving the learning conditions in primary schools in targeted regions, including in the preprimary O-classes. Support under this component was provided for (a) reducing the proportion of low-performing primary schools (Level 1 in inspection standards) in the region with highest share of these school (Afar region) and (b) improving the learning environment of preprimary O-classes in two emerging regions (Benishangul-Gumuz and Gambella regions) by increasing the capacity of O-class teachers to deliver an ECE curriculum package. Figure 1.1 School Classification Standards Inputs Standards (25%) Process Standards (35%) Standard 1 - Classrooms, education facilities, Standard 8 - Students’ learning and participation textbooks, and other recourses Standard 9 - Students’ progress in learning Standard 2 - Financial resources Standard 10 - Students’ attitude toward school Standard 3 - Qualified directors, teachers Standard 11 - Teaching and lesson planning and use of and staff resources Standard 4 - Conducive teaching-learning Standard 12 - Teachers’ knowledge of subject environment Standard 13 - Use of modern teaching methods Standard 5 - Education development army Standard 14 - Support for female and special need students Standard 6 - School’s vision and mission Standard 15 - CPD for teachers, directors and supervisors Standard 7 - Preparation of participatory Standard 16 - Teamwork by directors, teachers, and students School Improvement Plan Standard 17 - Knowledge and effective application of curriculum Standard 18 - Assessment and feedback to students Standard 19 - Monitoring of the school plan implementation Standard 20 - Proper utilization of resources Standard 21 - Partnership with parents and the community Output Standards (40%) Standard 22 - National education goals on access, internal efficiency, and education sector goals Standard 23 - Classroom and regional and national exam results Standard 24 - Students’ behavior and values Standard 25 - Communication and interaction among the school teachers, leaders, and support staff Standard 26 - Support and participation of parents and local community Page 36 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) 2. To achieve the first target, the Government relied on a system of measuring school performance through inspection20. Figure 1.1 presents the standards in the input, process, and output domains. The following activities were conducted to upgrade the low performing schools: (i) regional task force created with experts from the Inspection, SIP and Planning Directorates; (ii) indicator level analysis of the inspection results conducted, and feedback provided to schools outlining areas for improvement under each standard and indicator; (iii) Trainings of school principals, supervisors and woreda education office heads on inspection, school leadership and school improvement program (SIP); (iv) schools developed and implemented strategic and annual SIP plans based on the identified gaps; (v) teaching and learning materials, blackboards, student chairs and tables, water tanks procured and distributed to target schools based on the identified gaps; (vi) supervision visits conducted by regional task force; and (vii) woreda inspections supported by the REB Inspection Directorate with sample re-inspections to ensure the robustness of inspection results and to strengthen the capacity of woreda inspectors; re-inspection of all level I baseline schools completed. Performance of schools on inspection standards was analyzed, and the schools were supported to develop strategic and annual school improvement plans (a joint effort from school leadership, teachers, local authorities, community leaders, parents and students). The leaders of schools of upgraded schools were invited to share their experiences with the peers. The developed approach will be scaled up in the region, beyond baseline schools, and nationally. The experiences gained in the process of turning around the low-performing schools are very important for upgrading schools since the majority of primary and secondary schools in the country do not meet the national standards. 3. For improving the learning environment of preprimary O-classes in two emerging regions (Benishangul-Gumuz and Gambella regions) the Project supported the development of new O-class curriculum package and relevant teacher training. Also, a minimum package of locally available preprimary teaching and learning materials were defined and made available in two regions. This was done to tackle the low teacher competencies and a lack of classroom learning materials which appear to be the biggest barriers to ensuring children’s school readiness. Piloting the delivery of O-class curriculum in the Benishangul-Gumuz and Gambella regions provided important lessons to other emerging regions and to the nation as a whole. The nation-wide scale-up of the O-class program is now being conducted under the follow-up operation GEQIP-E, financed by IDA, DFID, Norway, Finland and UNICEF. 4. Component 2: Strengthening Equity and Inclusion in Education. The objective of this component was to strengthen equity and inclusion in primary schools of the country. Support under this component was provided for (a) enhancement of the gender balance in school leadership by increasing the number of trained female primary school principals and (b) development of inclusive learning environments by increasing the school grant allocation to the regions to support children with special needs. Number of additionally appointed trained female primary school principals increased from 3,150 principals to 5,274 (vis-à-vis planned target of 6,210; 69 percent on target). The appointment of newly recruited and trained female school leaders was done in newly opened schools and as replacement to principals and deputy principals who leave their positions. 20 Ethiopia’s school inspection system includes annual school self-assessment and external inspection every three years. The inspection framework consists of 7 standards on school inputs, 14 standards on processes and 5 standards on outcomes. Based on the aggregate percentage score in these 26 standards, schools are classified into 4 levels of performance, with Level 3 and Level 4 meeting the national standards. Page 37 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) 5. Utilization and reports by each region on utilization of 4 percent supplementary school grant to support education of children with special needs have been made (target met). The top-up to the school grant was disbursed to all regions to create a more inclusive learning environment. Regions were encouraged to be innovative in the allocation and utilization of these additional grants. To have a better understanding of how these extra resources were used and whether they were considered useful, a qualitative survey was conducted with purposeful sampling of beneficiaries’ schools in six regions of the country: Harar, Ethiopia Somali, Addis Ababa, Afar, Tigray and Gambella. In 2009 E.C., a similar questionnaire was conducted in the 5 remaining regions (Dire Dawa, Benishangul-Gumuz, Amhara, Oromia as well as SNNP). Over two years of project implementation, all regions have been part of the survey which aimed to increase Government’s understanding on the effectiveness of this intervention. Regions and schools demonstrated a wide variety of ways for using the grant. Regions used funding to invite medical experts to assess the level of hearing and visual impairment before making a decision on which hearing or visual aids are needed by the students, to provide learning materials to the students with special needs or channeled the resources to woredas and schools, based on EMIS data or to targeted schools. The majority of schools purchased additional learning materials (30%), assistive devices for individual needs (20%) and/or in training and awareness activities (16%). The perception of the effectiveness of the program regarding the enrollment and retention rates of children with special educational needs varies among schools. In general, 46% of principals responded that the program is very effective and 36% answered it is somewhat effective. 70% of respondents reported that the received support made it easier for them to meet the needs of students with certain disabilities. Although it was reported by the REBs that the amount allocated is too little to cover costs needed by the schools to support the children with special needs, this financing proved to be instrumental in raising the awareness for special educational needs and helped regions implement different strategies to strengthen their inclusive education programs. 6. Component 3: Improving Internal Efficiency of Primary Schools in SNNPR. The objective of this component was to contribute to internal efficiency of primary education in SNNPR. Support under this component was provided for reduction in the grade 1 dropout rate in SNNPRs in line with the ESDP V national targets. The MoE, jointly with REBs and woredas, undertook a number of activities to help improve attendance: (a) mobilized and sensitized communities on the importance of student participation/attendance in early years of schooling; (b) provided school feeding, and (c) strengthen the links between the SIP and the inspectorate system, thereby reinforcing a focus on improved school attendance/completion. Grade 1 dropout rate in Southern Nations, Nationalities, and People's Region (SNNPR) decreased from 25.20 percent to 16.9 percent (vis-à-vis the planned target of 13 percent; 68 percent on target). The linear trend (without the project) suggests the dropout rate of at least 19 percent, with non-linear projections suggesting the dropout rates of over 20 percent. The progress is reported to be the result of both the dropouts monitoring and mitigation interventions implemented by the MoE, Regional Education Bureaus (REBs), woredas and schools, and better registration of new enrollees, repeaters and re-admitters at school level. The results are commendable since SNNPR witnessed social unrest and ethnical conflicts during the project lifetime, limiting progress in this important area. Page 38 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION A. TASK TEAM MEMBERS Name Role Preparation Anna Olefir, Girma Woldetsadik Task Team Leader(s) Fitsum Zewdu Mulugeta Team Member Shimelis Woldehawariat Badisso, Pascal Tegwa Procurement Specialist(s) Meron Tadesse Techane Financial Management Specialist Biftu Wordofa Team Member Rosario Aristorenas Team Member Mohammad Nadeem Team Member Chukwudi H. Okafor Social Specialist Maiada Mahmoud Abdel Fattah Kassem Team Member Hiroshi Saeki Team Member Tersit Berhane Ghiday Team Member Samuel Lule Demsash Social Specialist Yacob Wondimkun Endaylalu Social Specialist Supervision/ICR Girma Woldetsadik, Anna Olefir Task Team Leader(s) Shimelis Woldehawariat Badisso, Pascal Tegwa Procurement Specialist(s) Fitsum Zewdu Mulugeta Team Member Ademe Zeyede Hailu Team Member Meron Tadesse Techane Financial Management Specialist Yacob Wondimkun Endaylalu Environmental Specialist Samuel Lule Demsash Social Specialist Page 39 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) Qi Shen Young Team Member Tersit Berhane Ghiday Team Member Hiroshi Saeki Team Member Maiada Mahmoud Abdel Fattah Kassem Team Member Chukwudi H. Okafor Social Specialist Mohammad Nadeem Team Member Rosario Aristorenas Team Member Biftu Wordofa Team Member Tsebaot Bekele Habte Team Member B. STAFF TIME AND COST Staff Time and Cost Stage of Project Cycle No. of staff weeks US$ (including travel and consultant costs) Preparation FY17 .225 4,719.89 FY18 11.868 50,609.65 FY19 3.000 87,056.43 FY20 0 39.70 Total 15.09 142,425.67 Supervision/ICR FY18 0 62,882.72 FY19 0 81,742.36 FY20 3.800 10,177.27 Total 3.80 154,802.35 Page 40 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) ANNEX 3. PROJECT COST BY COMPONENT Amount at Approval Actual at Project Percentage of Approval Components (US$M) Closing (US$M) (US$M) Component 1: Improving Learning Conditions in 10.00 8.43 84% Primary Schools in Targeted Regions Component 2: Strengthening Equity and Inclusion in 10.00 8.36 83% Education Component 3: Improving Internal Efficiency of Primary 10.00 5.00 50% Schools in SNNP Region Total 30.00 21.80 73 Page 41 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) ANNEX 4. EFFICIENCY ANALYSIS Economic and Financial Analysis 1. This annex presents the Project’s economic and financial analysis and examines the efficiency of investment in education in Ethiopia based on key indicators and estimates of internal rates of returns (IRR) of actual indicators observed at the end of the project implementation. The annex is structured as follows. First, an overview of the major challenges in education sector is presented. Second, an analysis of the returns to education are provided. Third, a cost-benefit analysis of the project is presented providing evidence of the efficiency of investments undertaken under the project. The last section briefly discusses education sector financing. Challenges in the Education Sector 2. Ethiopia made good progress in terms of expanding universal education. However, access, quality and equity remain the main challenges of the education system. Internal efficiency of the education system is still overwhelmed by high dropout rates, causing primary school completion rates and secondary school enrollment rates to fall well below their peers. The problem remains particularly acute in grade 1, with large regional variations. For example, nearly half a million primary students dropped out of school in Southern Nations, Nationalities, and Peoples Region (SNNPR) in 2018 (SNNP region education bureau). Over 2.6 million school-aged children are estimated to be out of school and are excluded from learning (UNESCO, 2015). Moreover, learning poverty in Ethiopia is 3.3 percentage points worse than the average for the Sub-Saharan Africa region and 0.2 percentage points worse than the average for low income countries (Figure 4.1)21. Figure 4.1: Learning poverty 100 89.8 90 91 86.7 83 90 74 80 70 60 50 40 30 20 8.7 10 0 HICs Senegal Uganda SSA LICs Ethiopia Mali Source: World Bank, 2019. 3. The Government of Ethiopia has made a strong commitment to improve access to preprimary education. Its national sector strategy aims to achieve an average 80 percent gross enrollment rate in preprimary education by 2020. This has led to a massive expansion of the preprimary sector in the last six years. The 21Learning Poverty means being unable to read and understand a short, age-appropriate text by age 10; and the indicator brings together schooling and learning. High Income Countries (HICs); Low Income Countries (LICs); and Sub-Saharan African Counties (SSA). Page 42 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) growth has come mainly from the provision of O-Class, a one-year preprimary program provided in public primary school compounds. It began in 2011/2012 and rapidly expanded to 2.5 million children in 2017/2018, with an impressive annual growth rate of about 20 percent. Most importantly, it is now available in many previously underserved remote and rural areas. However, this expansion has also led to many challenges. Almost all the teachers in these classes have little to no training on childhood development. Classroom observations also show a lack of appropriate pedagogical approaches and learning materials. Frontal teaching dominates teachers' classroom interaction, with limited encouragement of contact between students. Addressing these barriers to children's school readiness has become an urgent priority. 4. The World Bank is supporting Ethiopia in improving condition for quality preprimary education and increasing access of children to early childhood education (ECE) programs in O-Class through the Ethiopia Education Results Based Financing (EERBF) project as pet the Project Development Objective (PDO). The project is financed by GPE to test pilots considered for scale-up under the General Education Quality Improvement Program for Equity (GEQIP-E) financed by IDA and development partners (DFID, Embassy of Finland, UNICEF and Embassy of Norway). In December 2018, the GEQIP-E Program for Results (PforR), a five-year program, was launched with IDA financing of US$300 million, other DPs financing of US$140 million (DFID US$117m, Finland US$19m, Norway US$11m and UNICEF US$4m) and government financing of US$1,460 million. 5. The EERBF project supported the development and implementation of intensive in-service training in two emerging regions: Benishangul-Gumuz and Gambella. The training program focused on play-based activities through art, songs, and indoor/outdoor games to help children develop early literacy and numeracy as well as social, creative thinking, problem solving and motor skills. The EERBF project also supported the development and availability of a minimum package of pre-primary teaching and learning materials to ensure child-friendly learning environment in O-Classes. Teachers were actively involved in production of learning materials using locally available resources. Following in-service training, the regional education bureaus continue to provide support through coaching and supervision of O-Class teachers, as well as training of school management staff. 6. The EERBF project highly contributed to increase female school principals as a means of demonstrating an equitable education profession and a gender inclusive system. This is directly supported the commitment to improve gender balance within the civil service as articulated in the second Growth and Transformation Plan (GTP II). Increased participation of female in education leadership provided adolescent girls with role models to motivate them to excel in their studies; and made schools more sensitive to gender-specific needs and provided girls with additional support. The project also supported to increase financing for students with special needs enrolled in schools, which in turn triggered attitudinal change in the society. In addition, the project played a pivotal role in improving the learning conditions in primary schools in the most educationally disadvantage region of the country – Afar region. Returns to Education 7. Education is a key determining factor of livelihoods and higher level of education is associated with better employment opportunities and higher earnings in Ethiopia. For example, according to the 2016 Ethiopian Household Consumption/Expenditure Survey (HCES), those with secondary education earn 28.8 percent more than those with incomplete secondary education; those with incomplete secondary education earn 2.6 percent more than those with only primary education; those with primary education, in turn, earn Page 43 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) about 6.5 percent more than those with only incomplete primary education; and the latter earn about 13.1 percent more than individuals who have no formal education (Figure 4.2). Figure 4.2: Average monthly earnings by level of education (in Birr) 2500 2143 2000 1622 1664 1523 1500 1347 1200 1000 908 700 461 500 300 0 No formal education Incomplete primary Primary education Incomplete secondary Secondary education education education 2011 2016 Source: Data from Ethiopian Household Expenditure Survey (HCES), 2011 and 2016. 8. Education is an attractive investment for individuals and higher level of education is also associated with higher rate of returns. Returns to education in terms of consumption have increased, both in urban and in rural areas. According to the latest data of 2019 Ethiopia Poverty Assessment, whereas in urban areas post-secondary education had a 45 percent return in terms of household consumption in 2011 (relative to a household with an uneducated head), it had a 64 percent return in 2016 (Table 4.1). In rural areas, the return of having a head who completed primary educated doubled from 10 percent in 2011 to 20.8 percent in 2016. Table 4.1: Returns to education in terms of consumption in Ethiopia (urban areas), 2011 and 2016 Returns to education in terms of consumption (%) Level of education 2011 2016 Urban Rural Urban Rural Adult education 10.4 -2.4 12.5 6.2 Primary incomplete 10 4.6 16.6 11.3 Primary complete 11.3 10 24 20.8 Secondary incomplete 18.2 12 32.5 22.8 Secondary complete 28.4 38.3 40 47.8 Post-secondary 44.5 --- 63.9 --- Source: Ethiopia Poverty Assessment, World Bank (2019) 9. Moreover, investments during the early years yield considerably higher results than equivalent investments made during the primary or secondary school years or beyond (Figure 4.3). Investments in ECE also have proven highly cost-effective and a wise use of limited government resources. According to Engle et al, (2011), increasing preschool enrollment to 50 percent of all children in low- and middle-income Page 44 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) countries could result in an estimated benefit of lifetime earnings ranging from US$15 to US$34 billion22. Figure 4.3: Comparison of returns on investment for education interventions23 Source: Heckman and LaFontaine, 2007 Cost-Benefit Analysis 10. The EERBF project was implemented over a two-year period between 2016/17 and 2017/18 and consists of five results areas: (i) reducing the proportion of low performing primary schools in Afar region; (ii) improving the learning environment of O-classes (pre-primary education) in two emerging regions (Gambella and Benishangul Gumuz) by increasing the capacity of O-class facilitators to deliver an early childhood education curriculum package; (iii) addressing the gender balance in school leadership by increasing the number of trained female primary school principals; (iv) encouraging more inclusive learning environments by increasing the school grant (SG) allocation to support special needs; and (v) reducing grade 1 dropout rates in the SNNP region. 11. Cost-benefit analysis was conducted during the project appraisal, and internal rates of returns (IRR) and net present values (NPV) were calculated under three scenarios (low, base and high case scenarios). The analysis calculated the cost and benefit streams over a time span of 20 years from 2017 to 2036. Under the appraisal stage, IRR estimates from the analysis ranged from a low of 13.7% to a high of 15.7%. Similarly, NPV estimates ranged from approximately US$213.1 million to US$318.6 million. However, this analysis was conducted for the last results area (reducing grade 1 dropout rates in the SNNP region); and given the absence of evidence for the remaining results areas for Ethiopia which, consequently, were not part of the estimations of this analysis. Similar to the project appraisal stage, a simple cost benefit analysis is undertaken for the ICR focuses on the results area aimed to reduce grade 1 dropout rates in SNNP region based on available data with several assumptions (Table 4.2). 22 Engle et al. 2011. Strategies for Reducing Inequalities and Improving Developmental Outcomes for Young Children in Low- income and Middle-income Countries. 23 Much of the cost-benefit analysis of ECE investments are based on interventions in OECD countries (most of them, in the United States). Page 45 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) 12. Benefits Stream: To quantify the likely benefits because of tackling grade 1 dropouts in SNNP region through RBF project implementation, first the number of additional students who were retained in the education system and progress to grade 2 as a direct result of the project investment is estimated, adjusted for increasing enrollments. As a result, the number of beneficiaries is estimated to increase to 72,455 in the first year and to 70,770 in the second year. This result area was designed to support a reduction in the grade 1 dropout rate in SNNP region from 25.2 percent in 2015/16 to 13 percent in 2017/18, in line with the ESDP V national target. Despite the encouraging results achieved in the first year (grade 1 dropout rate reduced to 16.6 percent from 25.2 percent), due to serious ethnic clashes and regional instability, second year project implementation faced increasing problems. The 2017/18 prolonged drought and insufficient regional budgets for the ongoing school feeding program also contributed for the lower achievements (grade 1 dropout rate continued to be high, 16.8 percent in 2017/18 compared to 13 percent of the original target set to be achieved). 13. Second, the benefits for the students who participate or graduate from primary and secondary education are estimated based on the updated data on dropout rate by grade level, labor force participation and wage rates, assuming that following the decline of dropouts in grade 1, similar achievements would be observed for other grades, that is, the dropout rates for grade 2–12 also mirror the regional average. Third, the benefits are derived from the higher expected earnings generated by a more qualified and educated pool of students. Note that wage gain (earnings) is estimated from the difference between the average wage of those with some or completed primary or secondary education in relation to those without formal education, adjusted for annual wage increases of 6 percent. Table 4.2 Summary of assumptions for cost-benefit analysis (reducing grade 1 dropout rates in SNNPR) Items/Indicators Appraisal Stage ICR Stage Number of grade 1 students in SNNPR 897,385 842,500 Enrollment reduction in 2017/18 from the base year --- 54,885 Grade 1 dropout rate in (year 1) SNNPR, % 25.2 → 17 16.6 Grade 1 dropout rate in (year 2) SNNPR, % 17 → 13 16.8 Children gain (Year 1), adjusted for decreasing enrollments 73,586 72,455 Children gain (Year 2), adjusted for decreasing enrollments 109,481 70,770 Additional enrollments grade 1 per year, % 3.3 3.53 Transition rate grade 8 to grade 9, % 55.0 57.7 Labor force participation rate 82.9 80 Annual wage gain: incomplete primary, US$ 105.6 100 Annual wage gain: completed primary, US$ 262.3 156 Annual wage gain: some secondary, US$ 398.7 179 Annual wage gain: completed secondary, US$ 590.2 451 Annual nominal earnings growth rate, % 3.0 6 Education recurrent spending (2014/15; 2018/19), ETB 2,561,735,597 6,200,000,000 Exchange rate 20.5 27.4 Teacher salaries in education recurrent spending, % 80.0 66 SNNPR student-teacher ratio, grades 1–8 50.0 38 SNNPR student-teacher ratio, grades 9-12 28.0 21 Discount rate, % 7 7 Source: Calculations based on World Bank, HCES and MoE-ESAA data. Page 46 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) 14. Costs Steam: The costs streams constituted: (i) project costs of US$21.8 million (Table 4.3); (ii) primary and secondary education public expenditure; (iii) costs of additional teachers required to provide education services to beneficiary students in line with the current student-teacher ratio in primary and secondary schools; and (iv) the per capita school grants. Table 4.3a. Summary of initial budgets and actual disbursements (in USD) Budget at the Actual DLIs appraisal stage disbursement DLI 1. Reducing the proportion of low performing primary schools (Level 1 in inspection standards) in Afar region 5,000,000 3,437,500 DLI 2. Improving the learning environment of O-classes (pre-primary class) in two emerging regions by increasing the capacity of O-class facilitators to deliver an early childhood education curriculum package 5,000,000 5,000,000 DLI 3. Addressing the gender balance in school leadership by increasing the number of trained female primary school principals 5,000,000 3,363,712 DLI 4. Encouraging more inclusive learning environments by increasing the School Grant allocation to support special needs 5,000,000 5,000,000 DLI 5. Reducing Grade 1 dropout rates in the SNNP region 10,000,000 5,000,000 Total 30,000,000 21,801,212 Table 4.3b. Details of initial budgets and actual disbursements (in USD) Disbursement-Linked Withdrawal ceiling Payment Disbursement-Linked Result (DLR) Indicator (DLI) (expressed in USD) (in USD) DLI 1. Reducing the DLR 1.1: 40 primary schools upgraded from Level 1 to Level 2 so 2,500,000 1,062,500 proportion of low that the share of Level 1 schools is reduced to 37.1% performing primary schools (Level 1 in inspection DLI 1.2: 40 primary schools upgraded additionally from Level 1 to 2,500,000 2,375,000 standards) in Afar region Level 2 to reach the cumulative number of 80 schools upgraded to Level 2 over the 2-year period to reach 25.7% of Level 1 school in the region DLI 2. Improving the DLR 2.1: More than 90% of 0-class teachers (at least 447) in 4,000,000 4,000,000 learning environment of O- Benishangul-Gumuz region completed training package; O-class classes (pre-primary class) national minimum teaching and learning material package in two emerging regions by defined and made available for Benishangul-Gumuz and Gambella increasing the capacity of regions O-class facilitators to DLI 2.2: More than 90% of 0-class teachers (at least 160) in 1,000,000 1,000,000 deliver an early childhood Gambella region complete training package; more than 90% of education curriculum the O-classes have implemented the newly developed O class package curriculum package DLI 3. Addressing the DLR 3.1: 1,020 trained female primary school principals 1,500,000 1,100,308 gender balance in school additionally appointed to reach 4,170 female principals target leadership by increasing the DLR 3.2: 2,040 trained female primary school principals 3,500,000 2,263,404 number of trained female additionally appointed to reach 6,210 female principals target primary school principals DLI 4. Encouraging more DLR 4.1: 2% top-up to regional school grant allocation received 1,500,000 1,500,000 inclusive learning and utilized by each region to specifically support special needs environments by increasing DLR 4.2: 4% top-up to regional school grant allocation received 3,500,000 3,500,000 the School Grant allocation and utilized by each region to specifically support special needs to support special needs DLI 5. Reducing Grade 1 DLR 5.1: Grade 1 dropout rate reduced from 25.2% to 17% at the 5,000,000 5,000,000 dropout rates in the SNNP end of 2016/17 (8.2 percentage points reduction) region DLR 5.2: Grade 1 dropout rate reduced from 17% to 13% at the 5,000,000 0 end of 2017/18 (4 percentage points reduction) Total 30,000,000 $21,801,212 Page 47 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) 15. The Project justified its investment: Table 4.4 below compares findings from the cost-benefit analysis carried out during project preparation stage and from the analysis carried out at the ICR stage. The internal rates of return (IRR) of costs and benefits shows that the Project was economically viable and sound both at the project preparation stage and project closing based on data from actual number of project beneficiaries. Notwithstanding with limited disbursements over the life of the Project, the results from the cost-benefit analysis continue to show that the project remains economically viable. The corresponding NPV of program benefits is US$183.2 million somewhat lower compared with US$264.1 million at the project preparation stage. The IRR associated with this NPV is 12.1 percent which is lower than the estimated IRR (14.7 percent) at project preparation, which reflects that the Project partially achieved the planned targets in SNNP region due to different factors as explained above. The analysis also shows the benefit-cost ratio of monetary benefits deriving from the reduction in grade 1 dropout rate to be 2.5 (3.1 at project preparation). 16. The model employed in the analysis captures only part of the Project’s benefits and therefore provides the lower bound for the impact of the Project. The analysis takes into account only quantifiable benefits and therefore, may underestimate overall gains. There are positive externalities associated with higher educational attainment and better cognitive and noncognitive. There are also other important individual gains from investing in education; nonetheless, many of them were not captured in this analysis, and therefore, this estimation is to be considered a conservative estimation of the benefits of the project due to data restrictions. As a result, it is likely that the computed internal rate of return presents a lower bound for the impact of the Project. Table 4.4: Net Present Value (NPV), Benefit-to-cost ration and Internal Rate of Return (IRR) Items/Indicators Appraisal Stage ICR Stage Present value of costs (US$, million) 125.0 124.7 Present value of incremental benefits (US$, million) 389.2 308.0 Net present value (NPV), (US$, million) 264.1 183.2 Benefit-to-cost ratio (BCR) 3.1 2.5 Internal rate of return (IRR), % 14.7 12.1 17. Through a simple sensitivity test, it can be safely concluded that these estimations hold generally, and the IRR does not present major changes. To test for overestimation of benefits, a sensitivity test was performed by increasing the discount rate. By increasing the discount rate to 8 percent and 10 percent, benefit-cost ratios of 2.4 and 2.2 were obtained, respectively. These results are not as high as the calculated 2.5; however, they are not low enough to cast doubt on the benefits of the Project. Moreover, the assumptions are already on the conservative side, and by decreasing them even further, the changes in IRR and benefit-cost ratios were only minimal. 18. For the reaming results areas, due to lack of evidence for Ethiopia, it is difficult to quantify the benefits of creating appropriate learning conditions. In fact, better learning conditions tend to result in higher learning outcomes and earnings of people throughout their lifetime, as well as faster economic growth at the national level. Increasing the number of female school principals as a means of demonstrating an equitable education profession and gender inclusion system could play a crucial role, e specially on girls’ education achievement, retention, and reducing repetition rates. Supporting a more inclusive learning environment by providing financial support for special need education significantly stimulate a positive trend in the enrollment and retention of children with disabilities. There are many beneficiaries of this Page 48 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) type of intervention including parents and staff at different levels of the education systems. 19. Children who attend ECE tend to have better cognitive skills (literacy and numeracy) later in life than other children. The ECE promotes school achievements, numeracy and early mathematical skills of children which later help them solve more complicated mathematical problems in high school. For example, Young Lives data from Ethiopia indicate that children who attended preschool perform better in primary school as compared to their peers: 8-year-olds who have attended pre-school scored 43.7 percent higher in vocabulary tests and 51.1 percent higher in the cognitive test than those who did not attend preschool24. Likewise, preschool education promotes children reading proficiency in Ethiopia (Table 7); and preschoolers were 38 percent more likely to be a proficient reader than non-preschoolers (Kim J., 2019)25. However, given the absence of the evidence between the cognitive skills and earnings in Ethiopia, these benefits are not quantified in this analysis. Table 4.5. Preschool attendance and early grade reading assessment (EGRA) outcomes EGRA Task (1) 2010 OLS (2) 2016 OLS (3) 2016 School-Fixed Coef. Effect Size Coef. Effect Size Coef. Effect Size (SE) (SE) (SE) (SE) (SE) (SE) Oral reading fluency 0.46 0.02 4.15*** 0.20*** 2.48*** 0.12*** (1.55) (0.07) (0.78) (0.03) (0.61) (0.03) Letter sounds 3.33** 0.11** 6.48*** 0.20*** 3.46*** 0.11*** (1.53) (0.05) (0.91) (0.03) (0.87) (0.03) Familiar words 1.46 0.08 4.25*** 0.21*** 2.28*** 0.11*** (1.38) (0.07) (0.60) (0.03) (0.59) (0.03) Invented words 1.61** 0.11** 2.84*** 0.19*** 1.60*** 0.11*** (0.80) (0.06) (0.46) (0.03) (0.43) (0.03) Reading comprehension 1.34 0.05 4.67*** 0.17*** 2.78*** 0.10*** (% of correct answer) (1.92) (0.07) (0.85) (0.03) (0.82) (0.03) Listening comprehension 0.21 0.01 1.85** 0.07** 1.20 0.04 (% of correct answer) (1.64) (0.06) (0.76) (0.03) (0.77) (0.03) Observation 9,121 9,121 8,332 8,332 8,332 8,332 Source: Kim J. 2019. Unpacking the Role of Early Learning in Student Learning Outcomes: Evidence from National Reform of Pre- Primary Education in Ethiopia, University of Cambridge. Data significant at different levels: * p<0.1, ** p<0.05, *** p<0.01. 20. Education Sector Financing: The Government of Ethiopia is highly committed in investing in education and the sector accounts for the largest proportion of the government’s spending at 22 percent of total expenditure in 2017/18 reflecting the country meets the internationally agreed targets set out by the Education for All (EFA) coalition of 20 percent of total government expenditure heading to finance education expenditure. Public spending on education in Ethiopia increased by 104.3 percent in real terms between 2013/14 and 2017/18. This increase is largely a result of the expansion of the higher education and school enrollment up to grade 12 rising from 22 million to more than 27 million. However, only 38 24 Woldehanna T., Gebremedhin L. 2012. “The Effects of Pre-school Attendance on the Cognitive Development of Urban Children aged 5- and 8-Years Evidence from Ethiopia.” Young Lives, University of Oxford. 25 Kim J. 2019. Unpacking the Role of Early Learning in Student Learning Outcomes: Evidence from National Reform of Pre-Primary Education in Ethiopia, University of Cambridge. Page 49 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) percent of the education spending was headed to finance primary and secondary education (with 28 percent and 10 percent shares, respectively) in 2017/18. Improvements are thus required in the composition of education spending by reallocating government expenditure within the education sector towards general education for equity considerations. The total education expenditure as percentage of GDP remained above 4 percent in recent years. Page 50 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS The Government submitted its Project Completion Report on June 29, 2019. Comments on a draft of this Implementation Completion and Results Report (ICR) were received from the Government on November 20, 2019 and from the Education Sector Working Group, in representation of donors, on November 25, 2019. The Government’s main comments were related to disbursement figures, number of ‘O’ class teachers trained, and editorial changes and have been addressed in the ICR. The Education Sector Working Group’s main comments were related to clarifications on training of ‘O’ class teachers, impact of female leaders on girls’ enrolment, and students with special needs and have been addressed in this ICR. Page 51 of 52 The World Bank ETHIOPIA EDUCATION RESULTS BASED FINANCING PROJECT (P163608) ANNEX 6. SUPPORTING DOCUMENTS (IF ANY) World Bank, Implementation Status Reports, Nos. 1-4. World Bank, Implementation Support Mission Aide Memoires. Education Technical Working Group, Summary of Meetings. Federal Democratic Republic of Ethiopia, Central Statistical Agency, Independent Verification Reports (several) on the Performance/Attainment of various Disbursement-Linked Indicators. World Bank, Country Partnership Framework for Ethiopia FY18-FY22, Report No. 119576-ET dated May 22, 2017. World Bank, Project Appraisal Document, Ethiopia Education Results-Based Financing Project, P163608, Report No. PAD2429, dated June 30, 2017. World Bank, Project Appraisal Documents, Ethiopia General Education Quality Improvement Program-I and Ethiopia General Education Quality Improvement Program-II, P106855 and P129828, Report Nos. 45140-ET and PAD476, dated November 24, 2008, and October 17, 2013 respectively. Federal Democratic Republic of Ethiopia, Ministry of Education, Education Statistics Annual Abstracts. Federal Democratic Republic of Ethiopia, Education Sector Development Program V (ESDP V) for 2016–2020. Hanushek, Eric A.; Woessmann, Ludger. 2007. The Role of Education Quality for Economic Growth. Policy Research Working Paper; No. 4122. World Bank, Washington, DC. Woldehanna T., Gebremedhin L. 2012. “The Effects of Pre-school Attendance on the Cognitive Development of Urban Children aged 5- and 8-Years Evidence from Ethiopia.” Young Lives, University of Oxford. Kim J. 2019. Unpacking the Role of Early Learning in Student Learning Outcomes: Evidence from National Reform of Pre-Primary Education in Ethiopia, University of Cambridge. Woldehanna, T. and Araya, M. (2017) Early Investment in Preschool and Completion of Secondary Education in Ethiopia: Lessons from Young Lives, Working Paper 168, Oxford: Young Lives. Neuman, Michelle J., Devercelli, Amanda E. 2013. What Matters Most for Early Childhood Development: A Framework Paper. Systems Approach for Better Education Results (SABER) Working Paper Series No. 5. World Bank, Washington, DC. The Economic Costs of Exclusion and Gains of Inclusion of People with Disabilities. Evidence from Low- and Middle-Income Countries. 2015. London. Page 52 of 52