Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00005556 IMPLEMENTATION COMPLETION AND RESULTS REPORT 5726XK ON A CREDIT IN THE AMOUNT OF SDR 7.9 MILLION US$11 MILLION EQUIVALENT TO THE REPUBLIC OF KOSOVO FOR THE KOSOVO EDUCATION SYSTEM IMPROVEMENT PROJECT JANUARY 31, 2021 Education Global Practice Europe And Central Asia Region CURRENCY EQUIVALENTS Exchange Rate Effective October 15, 2021 Currency Unit = SDR 1 SDR = US$ 1.4118 US$1 = SDR 0.708273 FISCAL YEAR January 1 - December 31 Regional Vice President: Anna Bjerde Country Director: Linda Van Gelder Regional Director: Fadia M. Saadah Practice Manager: Harry Anthony Patrinos Task Team Leader: Angela Demas ICR Main Contributor: Aisuluu Bedelbayeva ii ABBREVIATIONS AND ACRONYMS AI Administrative Instruction BFD Budget and Finance Division CBA Cost-Benefit Analysis CPF Country Partnership Framework CPS Country Partnership Strategy COVID-19 Coronavirus disease 2019 DSME Division for Standards, Monitoring and Evaluation EMIS Education Management Information System FM Financial Management GDP Gross Domestic Product GoK Government of Kosovo HD Human Development ICR Implementation Completion and Results Report IDA International Development Association IFR Interim Unaudited Financial Reports IDEP Institutional Development for Education Project IRI Intermediate Results Indicators IRR Internal Rate of Return ISR Implementation Status and Results IT Information Technology KCF Kosovo Curriculum Framework KAS Kosovo Agency of Statistics MEST Ministry of Education, Science and Technology M&E Monitoring and Evaluation MoF Ministry of Finance MTR Mid-term Review MED Municipal Education Directorate NSAF National Student Assessment Framework NDS National Development Strategy NPV Net Present Value OECD Organisation for Economic Cooperation and Development PAD Project Appraisal Document PCU Project Coordination Unit PDO Project Development Objective PIRLS Progress in International Reading Literacy Study PISA Program for International Student Assessment POM Project Operational Manual SBM School-based Management SDG School Development Grants SDP School Development Plans SGB School Governing Boards SRC School Report Card SDR Special Drawing Rights SILC Statistics on Income and Living Conditions TA Technical Assistance TIMSS Trends in International Mathematics and Science Study iii TOC Theory of Change TTL Task Team Leader WB World Bank iv TABLE OF CONTENTS DATA SHEET ......................................................................................................................... VI I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 1 A. CONTEXT AT APPRAISAL .........................................................................................................1 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION ................................................................6 II. OUTCOME .................................................................................................................... 10 A. RELEVANCE OF PDOs ............................................................................................................ 10 B. ACHIEVEMENT OF PDOs (EFFICACY) ...................................................................................... 10 C. EFFICIENCY ........................................................................................................................... 24 D. JUSTIFICATION OF OVERALL OUTCOME RATING .................................................................... 26 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 28 A. KEY FACTORS DURING PREPARATION ................................................................................... 28 B. KEY FACTORS DURING IMPLEMENTATION ............................................................................. 30 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 32 A. QUALITY OF MONITORING AND EVALUATION (M&E) ............................................................ 32 B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE ..................................................... 34 C. BANK PERFORMANCE ........................................................................................................... 35 V. RISK TO DEVELOPMENT OUTCOME ............................................................................... 37 VI. LESSONS AND RECOMMENDATIONS ............................................................................. 37 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 40 ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................... 49 ANNEX 3. PROJECT COST BY COMPONENT (ORIGINAL AND RESTRUCTURED) ........................ 51 ANNEX 4. EFFICIENCY ANALYSIS ........................................................................................... 52 ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ... 60 ANNEX 6. GENERAL STRUCTURE OF THE KOSOVO EDUCATION SYSTEM ................................ 61 ANNEX 7. SUMMARY OF THE REVISIONS TO COMPONENTS ................................................. 62 ANNEX 8. SUMMARY OF PROJECT RESTRUCTURINGS ........................................................... 64 ANNEX 9. ORIGINAL THEORY OF CHANGE (AS INTERPRETED FROM PAD, TABLE 1) ............... 65 ANNEX 10. UPDATED THEORY OF CHANGE (APRIL 2019 RESTRUCTURING) ........................... 66 ANNEX 11. SUPPORTING DOCUMENTS & BIBLIOGRAPHY ..................................................... 67 v The World Bank Kosovo Education System Improvement Project (P149005) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name P149005 Kosovo Education System Improvement Project Country Financing Instrument Kosovo Investment Project Financing Original EA Category Revised EA Category Not Required (C) Not Required (C) Organizations Borrower Implementing Agency Ministry of Education, Science, Technology and Republic of Kosovo Innovation Project Development Objective (PDO) Original PDO The Project Development Objective is to strengthen selected systems that contribute to quality, accountability, and efficiency improvements in education in the Republic of Kosovo. FINANCING Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Financing 11,000,000 7,515,200 6,597,008 IDA-57260 Total 11,000,000 7,515,200 6,597,008 Non-World Bank Financing 0 0 0 Total 0 0 0 Total Project Cost 11,000,000 7,515,200 6,597,008 vi The World Bank Kosovo Education System Improvement Project (P149005) KEY DATES Approval Effectiveness MTR Review Original Closing Actual Closing 10-Sep-2015 13-May-2016 12-Nov-2018 31-Dec-2019 30-Jun-2021 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 25-Apr-2019 .92 Change in Results Framework Change in Components and Cost Cancellation of Financing Reallocation between Disbursement Categories 10-Oct-2019 1.94 Change in Loan Closing Date(s) 07-Dec-2020 4.78 Change in Loan Closing Date(s) KEY RATINGS Outcome Bank Performance M&E Quality Moderately Satisfactory Moderately Satisfactory Substantial RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 30-Oct-2015 Satisfactory Satisfactory 0 02 25-May-2016 Satisfactory Satisfactory 0 03 05-Dec-2016 Satisfactory Moderately Satisfactory 0 04 13-Jun-2017 Moderately Satisfactory Moderately Satisfactory .17 05 15-Jan-2018 Moderately Satisfactory Moderately Unsatisfactory .37 Moderately 06 07-Sep-2018 Unsatisfactory .67 Unsatisfactory Moderately 07 08-May-2019 Unsatisfactory .92 Unsatisfactory 08 17-Jan-2020 Moderately Satisfactory Moderately Satisfactory 2.66 09 24-Jun-2020 Moderately Satisfactory Moderately Satisfactory 3.98 vii The World Bank Kosovo Education System Improvement Project (P149005) 10 16-Mar-2021 Satisfactory Moderately Satisfactory 6.01 11 30-Jun-2021 Satisfactory Moderately Satisfactory 6.03 SECTORS AND THEMES Sectors Major Sector/Sector (%) Education 100 Public Administration - Education 7 Other Education 93 Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Public Sector Management 24 Public Administration 24 Transparency, Accountability and Good 24 Governance Human Development and Gender 76 Education 76 Access to Education 38 Education Financing 38 ADM STAFF Role At Approval At ICR Regional Vice President: Cyril E Muller Anna M. Bjerde Country Director: Ellen A. Goldstein Linda Van Gelder Director: Claudia Maria Costin Fadia M. Saadah Practice Manager: Mario Cristian Aedo Inostroza Harry Anthony Patrinos Task Team Leader(s): Flora Kelmendi Angela Demas ICR Contributing Author: Aisuluu Bedelbayeva viii The World Bank Kosovo Education System Improvement Project (P149005) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES A. CONTEXT AT APPRAISAL 1. Country Context. The Republic of Kosovo is Europe’s youngest country – both based on its new statehood1 and the average age (26 years) of its 1.8 million population. At appraisal,2 Kosovo had been one of only four countries in Europe that recorded positive growth rates in every year of the post-crisis period after 2008. At that time, the country’s economy was managed well, and the average growth of 3.5 percent during 2011-2014 did contrast favorably to the rest of the region.3 Nonetheless, despite sustained economic growth and tangible gains in poverty reduction,4 Kosovo remained one of the poorest countries in Europe demonstrating a negative association between poverty and level of education. In 2011, almost two-thirds of the poor possessed less than secondary education. 2. Widespread unemployment and a lack of quality jobs contributed significantly to poverty and income insecurity. In 2009, with a 45 percent unemployment rate and a low employment rate (29 percent), Kosovo had the weakest track record in Europe.5 The country’s difficult labor market conditions were especially severe for youth and women. According to Labor Force Survey, in 2014, only 11.5 percent of women were employed, compared to 38.7 percent of men.6 The lack of jobs led to migration and remittances that financed large imports and disincentivized work, while exports remained small, thus limiting business expansion. 3. To continue to grow, Kosovo needed to unleash productivity gains and create more quality jobs. This required addressing infrastructure bottlenecks, prioritizing human capital investment, and creating an environment more conducive to private sector development. 4. Sector Context. In 2015, the country’s pre-university education system covered students of pre-primary age and up to age 18, attending the following education levels: (i) pre-school education programs (ages 3-5); (ii) primary school (grades 1-5), lower secondary (grades 6-9) and upper secondary (grades 10-12). The overall general structure of the Kosovo education system is presented in Annex 6. Pre-university education was the largest sub-sector within the education sector and served almost 20 percent of the Kosovo population on a daily basis.7 5. Public spending in education was low relative to regional comparators, indicating the need to prioritize education in public spending. At appraisal, total public spending on education represented about 14 percent of budgetary expenditure, in line with other middle-income countries with similar profiles. However, because of the large number of pupils, Kosovo spends only 13 percent of per capita income per student in both primary and secondary education, which is much less than all countries in Southeastern Europe.8 1 Kosovo declared its independence on February 17, 2008 and is recognized as an independent country by more than 100 United Nations members and by 23 out of 28 members of the European Union (EU). 2 The appraisal of the KESIP was carried out from April 29 to May 5, 2015. 3 World Bank Group: Country Snapshot-Kosovo. Prishtina: The World Bank Country Office in
Kosovo, 2015. http://www.worldbank.org/content/dam/Worldbank/document/eca/Kosovo-Snapshot.pdf 4 From 2012 to 2015, the poverty rate at the 2011 PPP US$3.2 per day fell from 5.7 in 2012 to 2.9 percent in 2015. 5 Kosovo - Country Partnership Strategy, FY12-FY15 (English). Washington, D.C.: World Bank Group. http://documents.worldbank.org/curated/en/738541468047404424/Kosovo-Country-Partnership-Strategy-for-the-period-FY12-FY15 6 Kosovo - Country Partnership Framework, FY17-FY21 (English). Washington, D.C.: World Bank Group. http://documents.worldbank.org/curated/en/297951496160148830/Kosovo-Country-partnership-framework-for-the-period-FY17-FY21 7 Kosovo Education Strategic Plan 2011-2016. 8 Kosovo Education System Improvement Project (English ). Washington, D.C.: World Bank Group. http://documents.worldbank.org/curated/en/823281467991937253/Kosovo-Education-System-Improvement-Project Page 1 of 69 The World Bank Kosovo Education System Improvement Project (P149005) 6. In years leading up to appraisal, considerable progress was made in improving access to education, mostly at the pre-primary and upper secondary levels. Enrollment rates in primary and lower secondary education was almost universal and in upper secondary gross enrollments stood at 84.7 percent.9 In terms of education quality, results of the Program for International Student Assessment (PISA) survey, 2015 revealed inadequate and inequitable outcomes. More than two-thirds of students performed below a basic level of proficiency in science (68 percent), math (78 percent), and reading (77 percent)—a share that is much higher than in neighboring Montenegro and Albania and significantly higher than the Organisation for Economic Cooperation and Development (OECD) average of 21 percent.10 Within Kosovo, inequities were also prominent; for example in the area of science, rural students were about one year of schooling behind their urban peers, and students from low-income households were about 1.5 years of schooling behind their well-off peers.11 Kosovo’s education system also lacked transparency and reliable monitoring mechanisms, hindering management and planning decisions in the sector. While the Government of Kosovo (GoK) undertook a range of reforms in pre-university education, addressing issues of quality, access, relevance and management, efforts needed to be better coordinated and institutionalized, and continued support was required to support the reforms in a concerted manner. 7. In higher education, the GoK policies, focused on setting up the accreditation system for both public and private higher education providers, and in expanding access to higher education. 8. Project context. The Kosovo Education System Improvement Project (KESIP) was built on the on-going reform efforts and achievements made in the education sector at the time of its design. It was envisioned that KESIP would roll out certain activities supported by its predecessor, the Institutional Development for Education Project (IDEP, 2007- 2013), focusing on system-level improvements to enable stronger system management, effective policy making, and efficient resources allocation. The following results were achieved under the IDEP: (i) the Law on pre-university education and respective by-laws were developed and approved, laying out the foundation for reform in relevant areas, including school autonomy, financial decentralization, teachers’ qualifications and career development, assessment and examination; (ii) improvements to the budget allocation process were introduced; (iii) the capacities of school managers and teachers were strengthened through school-based management activities supported through school development grants; (iv) the State Council for Teacher Licensing was established and led the preparation of the legal framework for teachers’ licensing; (v) the Education Management Information System (EMIS) was improved; (vi) assessment policy reporting standards were supported; (vi) school mapping was carried out and standards for school designs were developed; (vii) the Law on Higher Education was revised. The interventions envisioned under the follow-up KESIP project were focused on expanding the IDEP efforts and ensuring sustainability of the core reform areas. 9. Rationale for Bank Support. The World Bank (WB) had been providing continuous and flexible support to Kosovo’s education system which was one of the top priority sectors of the government since the conflict in 1999. The Bank’s support to the education sector was considered critical for Kosovo’s development including the post-conflict period under transitional United Nations administration, declaration of independence, and international recognition. With youth unemployment and underemployment at persistently high levels, education continued to be central to the development and to the improvement of the lives of young people in Kosovo. The World Bank’s comparative advantage in Kosovo was to use its technical expertise, experience in supporting educational reforms in a post- 9 Kosovo Education Strategic Plan 2017-2021. 10 For each of the three subjects, “below basic proficiency” is defined as scores below Level 2—the baseline level of proficiency that all students should be expected to attain when they leave compulsory education. 11 Kosovo Country Partnership Framework for the period FY17-FY21 (English). Washington, D.C.: World Bank Group. Page 2 of 69 The World Bank Kosovo Education System Improvement Project (P149005) conflict environment as well as in the countries aiming to join the European Union, and its convening power to facilitate a more coordinated and holistic approach in supporting education. 10. Higher level objectives to which the project contributes. KESIP was designed to support implementation of the key reform priority areas in the education sector as outlined in the government’s Education Strategic Plan 2011-2016. Its objectives were aligned with the World Bank Group’s Country Partnership Strategy (CPS) for FY12-1512 (Report No. 66877-XK) for the Republic of Kosovo which, in turn, supported the government’s priorities identified in the National Development Plan and the Medium-Term Expenditure Framework for 2011–14. The first pillar of the CPS sought to accelerate broad-based economic growth and employment generation through actions in six priority areas, one of them being investing in education and skills to improve its quality and relevance. The project also contributed to the World Bank’s goals of reducing poverty and boosting shared prosperity through the project’s school -level investments that benefitted children from rural areas and localities with lower levels of socioeconomic development. Theory of Change (Results Chain) 11. The flow chart presented in Annex 9 depicts KESIP’s original theory of change inferred from the Table 1 results chain presented in the Project Appraisal Document (PAD), since there was no separate Theory of Change diagram in the PAD. KESIP’s results chain presents the key inputs, outputs, intermediate results, and outcomes which the project sought to achieve. The high-level outcomes, which were beyond the Project Development Objective (PDO) and to which the project interventions aimed to contribute, were improvements in quality, accountability, and efficiency of the Kosovo education system. Project Development Objectives (PDOs) 12. The PDO was to strengthen select systems13 that contribute to quality, accountability, and efficiency improvements in education in the Republic of Kosovo.14 Key Expected Outcomes and Outcome Indicators 13. As explained in paragraph 15 of the PAD, “By their very nature, most of the improvements in quality, accountability, and efficiency will materialize only with a considerable lag after the Project’s close. Therefore, the PDO indicators are defined to measure the achievement of expected outcomes that are directly attributable to project interventions (see Table 1). They serve as proxy measures of progress towards the ultimate results of improved quality, accountability, and efficiency throughout Kosovo's education system.” 14. The PAD identified four PDO-level indicators with eight expected project outcomes contributing to higher level, longer-term results of improved quality, accountability, and efficiency of Kosovo education system.15 The project’s performance against the identified outcomes was to be measured through the four PDO-level indicators which were supported by 15 Intermediate Results Indicators (IRIs), including two corporate indicators. Together, these were intended to track progress against achievement of the stated PDO. Table 1,16 verbatim from the PAD, shows the 12 The CPS for FY12-15 was the first to be prepared since Kosovo declared independence in February 2008 and became a member of the World Bank Group in mid-2009. That CPS followed a series of Interim Strategy Notes (ISNs) implemented over the previous decade and was closely aligned with the national development priorities set by the Government. This CPS was subsequently extended by one year. 13 Table 1 in this report lists the select systems, as described in the PAD, that the project sought to strengthen. 14 As specified in the PAD, dated August 4, 2015, and in the Financing Agreement, dated November 20, 2015. 15 KESIP PAD, page 16. August 4, 2015. Report No: PAD1015 16 Source: KESIP PAD, page 16. August 4, 2015. Report No: PAD1015 Page 3 of 69 The World Bank Kosovo Education System Improvement Project (P149005) original PDO indicators and their linkage with the project sub-components, expected project outcomes, and ultimate results or long term objectives to which KESIP’s selected system strengthening sought to contribute. Table 1. PDO level results indicators and results chain17 Strengthened Systems Contributing to Ultimate Ultimate Original PDO Indicators Corresponding Results Results (PAD) Sub-components (Expected Project Outcomes) (long term) 1. Percent of (i) Strengthened accountability through Improved municipalities 1.1 - Strengthening strategic planning transparent budget allocation accountability allocating resources to and financial management capacities schools according to (ii) Improved and rationalized allocation of Improved revised formula resources efficiency 1.2 - Enhancing existing Education (iii) Improved education data collection, Management Information System management, and utilization 2. Percent of school (EMIS) (iv) Improved education quality monitoring report cards Improved through student assessments distributed and quality and 2.2 - Strengthening capacities of key publicly displayed at (v) Strengthened school-level accountability and accountability institutions for monitoring schools quality through information dissemination on educational outcomes and student performance and sources/uses of funds assessment of student learning 3. Percent of beneficiary schools disseminating (vi) Strengthened school-level accountability Improved reports on grant 1.3 - School Development Grants mechanisms through information dissemination accountability budget allocation and on uses of funds for school level intervention utilization 2.3 - Improving accountability and (vii) Improved higher education quality Improved quality assurance of higher education monitoring through piloting and assessment of a 4. Number of licensing quality through the development of key licensing mechanism exams piloted professional licensing statutes in (viii) Improved accountability of higher Improved priority economic fields education institutions for learning outcomes accountability Note: Sub-component 2.1 on the Teacher Career System was to be measured with intermediate indicators. Components 15. At the time of project appraisal, the KESIP project consisted of three components and six sub-components, to be financed by an International Development Association (IDA) credit of Special Drawing Rights (SDR) 7.9 million (US$11 million equivalent).18 A summary of each component and related sub-components is provided below. 16. Component 1: Enhancing strategic planning, financial management, monitoring and decision–making in the education sector (estimated cost US$5.2 million equivalent; actual cost: US$3.34 million). This component aimed to strengthen planning, financial management and evaluation capacities at different levels of the education sector, including the national level through the Ministry of Education, Science and Technology (MEST), the municipal level through the Municipal Education Directorates (MEDs), the school level, and universities. The component included three sub-components: 17. Sub-component 1.1: Strengthening strategic planning and financial management capacities (estimated cost: US$ 1.1 million; actual cost: US$0.10 million). This sub-component aimed at strengthening strategic planning and financial 17Table 2 in section IB illustrates revised results chain following first restructuring. 18The first project restructuring approved in April 2019, entailed a partial cancellation of US$3.5 million from the IDA credit, related to the 2016-2018 unused funds, thus bringing the total project cost down to US$7.5 million. Page 4 of 69 The World Bank Kosovo Education System Improvement Project (P149005) management capacities of MEST, MEDs, schools and universities to improve spending efficiency and increase accountability. It sought to specifically support the following activities at the central level: (i) review of the national allocation formula for the education specific grant going to pre-university education; (ii) review of local budgeting processes in pre-university education to assess the complete municipal-to-school formula and the incentives yielded by its parameters; (iii) development of annual monitoring procedures for national and local budgeting processes and resource allocation based on the revised per capita financing formula, which would standardize information about basic indicators and allocation trends; (iv) development of education spending indicators to assess and monitor education spending efficiency. At the municipal level, the sub-component sought to provide series of trainings and on-the-job support for MEDs and schools on use of formula and application of the guidelines provided by the MEST. At the school level, this sub-component aimed to support further expansion of school financial autonomy by strengthening the financial management capacities of school directors. In addition, this sub-component aimed to support a small number of municipalities in developing and implementing school rationalization plans. At the university level, the sub-component aimed to finance the following activities: (i) development of the criteria and programs for full institutional financial autonomy; (ii) defining accountability norms and standards through bylaws developed within the context provided by the new law on higher education; and (iii) developing procedures for assessing performance of university leaders and management. 18. Sub-component 1.2: Enhancement of the existing EMIS (estimated cost: US$2.1 million; actual cost: US$0.03 million). This sub-component aimed to enhance and modernize the systems and practices of data management and analysis in the education sector. It sought to specifically support the following activities: (i) establishing rules and procedures for effective information management, use, data analysis and dissemination; (ii) enhancing data availability by implementing critical registries (students and school/university registries); (iii) developing a secure platform to integrate existing information systems and databases for further automation; (iv) enhancing existing EMIS capabilities to extract and aggregate data for expansion of the catalogue of indicators and more advanced statistical analysis; and (v) capacity building and training to schools, MEDs, universities, and central government staff (including MEST departments) on the use of all systems that would be implemented. The sub-component also sought to develop data protection systems aligned with Kosovo’s legislation. 19. Sub-component 1.3: School Development Grants (estimated cost: US$2.0 million; actual cost: US$3.21 million). This sub-component aimed at strengthening the capacity of schools to plan and manage resources, and promote citizen engagement through the participation of teachers, parents and communities in educational planning, implementation, and monitoring of school activities. It sought to finance activities that would: (i) strengthen the school development planning system; (ii) support selected primary schools to prepare and implement multi-year school development plans (SDPs) and offer training on management, planning and self-evaluation; and (iii) finance school development grants (SDGs) ranging US$10,000 to US$15,000 to selected primary schools. 20. Component 2: Strengthening management capacity and accountability to enhance quality of education (estimated cost: US$5 million; actual cost: US$2.55 million). This component aimed to support and institutionalize the teacher career system and facilitate the use of international best practices in assessing the quality of education. The design of the component included three sub-components: 21. Sub-component 2.1: Implementation and Improvements to the Teacher Career System (estimated cost: US$1 million; actual cost: US$0.54 million). The sub-component aimed to support MEST to: (i) implement the career system for teachers’ professional development, performance assessment and promotion; and (ii) provide po licy support to further strengthen, complement and systematize the teacher career system and licensing scheme. It sought to finance the: (i) development and implementation of a plan for teacher performance assessment aligned with local capacities; (ii) development of competency-based teacher licensing and promotion scheme that defines key Page 5 of 69 The World Bank Kosovo Education System Improvement Project (P149005) competencies for each type of license in the existing scheme and links these with financial incentives; and (iii) setting up of an induction program for new teachers, and other activities aimed at improving and institutionalizing teacher performance assessment and professional development. 22. Sub-component 2.2: Strengthening capacities of key institutions for monitoring educational outcomes and assessment of student learning (estimated cost: US$3 million; actual cost: US$1.97 million). This sub-component aimed to revise and upgrade the current system of national student assessment and examination in grades 5, 9 and 12, and bring it in line with international best practices. It sought to finance the following activities (i) setting up a model national assessment program incorporating procedures, technical standards and reporting methods aligned with international best practices; (ii) developing feedback systems for examination for decision makers and schools that are timely, and provide comprehensive information on learning outcomes; (iii) establishing an integrated Information Communication Technology system for processing examinations and assessments; and (iv) participating in international student assessments, including analyzing results. Through these activities, the sub-component also sought to strengthen the capacity of the MEST’s Division for Standards, Monitoring and Evaluation (DSME) to design, administer and process high quality examinations and large-scale student assessment surveys. 23. Sub-component 2.3: Improving accountability and quality assurance of higher education through the development of key professional licensing statutes in priority economic fields (estimated cost: US$1 million; actual cost: US$0.04 million). This sub-component aimed to strengthen the capacities of MEST to carry out professional licensing, and establish a mechanism for holding higher education institutions accountable for outcomes through: (i) development of a coherent national strategy, including bylaws, on professional licensing; and (ii) piloting of licensing activities in key professions to provide quality assurance to the labor market related to the knowledge and skills of graduates from public higher education institutions in the targeted fields of study. The sub-component sought to provide technical support, software and financing for developing, piloting and conducting a pilot in one or two key professional fields. 24. Component 3. Implementation and Communications Support (estimated cost: US$0.8 million; actual cost: US$0.71 million). This component sought to support project implementation by financing consultant services, non-consultant services, training directly related to project implementation, and operating costs for the Project Coordination Unit (PCU). It also aimed to support communication, including workshops, communication materials, and outreach events to key stakeholders and beneficiary communities. B. SIGNIFICANT CHANGES DURING IMPLEMENTATION 25. Revised PDOs and Outcome Targets. The PDO remained unchanged throughout the life of the project. Despite significant changes introduced to the PDO indicators and adjustment of project scope and activities, the project’s orientation towards achievement of the original PDO remained constant. The outcome targets were more realistically framed in terms of achievability, following the mid-term review (MTR) and approved as part of April 2019 restructuring in accordance with adjustments made to PDO indicators, project scope, and activities (see next section for more details). 26. Revised PDO Indicators. PDO indicators were revised, as part of the first restructuring, to reflect project simplification and changes in scope and financing of the project components. Two of the four original PDO indicators were dropped, and one new indicator was added to address the quality objectives of the PDO. Two outcome targets were adjusted given simplification and reduced scope of project activities. In total, the project had three PDO-level indicators following the April 2019 restructuring. The revisions to the PDO Indicators are presented in Table 2. Page 6 of 69 The World Bank Kosovo Education System Improvement Project (P149005) 27. Revised Results Framework. Nine of the original 14 intermediate results (IR) indicators related to cancelled sub- components and activities, were also dropped, including two corporate indicators, and the scope of three IR indicators related to teacher career system and assessment of student learning were reduced, reflecting project simplification and reduced scope of project activities. Three newly adopted Bank core indicators were added later as IR indicators, first appearing in the Implementation Status & Results (ISR) Report #10 but were not formalized through restructuring. Table 2. Revised PDO Indicators (from April 2019 restructuring) PDO Indicator Changes Rationale / Target Adjustment Original PDO Indicator 1: Dropped Dropped to reflect the decision to cancel pre-university and Municipalities allocating resources university level activities for budget formulas (subcomponent 1.1) to schools according to revised and reduce project scope. All associated IR indicators were dropped formula (percentage) as well. Original PDO Indicator 2: School Revised: Revised to reflect project simplification and reduced scope of school report cards distributed and School report report card activities. The indicator was revised to remain at a pilot publicly displayed at schools cards piloted level covering 20 percent of primary schools (against 90% as the (percentage) at school level original target) and include learning assessment results and school (percentage) level statistics Original PDO Indicator 3: Unchanged This PDO indicator remained unchanged, but the end-target was Beneficiary schools disseminating Renumbered adjusted from 80 percent to 60 percent of beneficiary schools report on grant budget allocation as PDO reporting given project simplification and uncertainty on whether and utilization (percentage) Indicator 1. the project would be extended. The restructuring paper indicates that this indicator contributed to efficiency as well as accountability. Original PDO Indicator 4: Dropped Dropped based on the decision to cancel university level activities. Licensing exams piloted (number) All associated IR indicators were dropped as well. New PDO Indicator 3: Reforms to Added Added as a new PDO Indicator 3 to capture the project’s contribution standardized 12 grade assessment to education quality improvements and reflect the decision to focus applied (Yes/no) on Grade 12 examination. Revised Components 28. Project components were modified under the first restructuring, reflecting the decision made to simplify the project design and reduce its scope. The revisions to each component are summarized in Table 3 and detailed in Annex 7. Page 7 of 69 The World Bank Kosovo Education System Improvement Project (P149005) Table 3. Summary of original and restructured KESIP components and sub-components following restructuring Original component & sub- Status following Comments components (PAD) restructuring (25-Apr-2019) Component 1. Enhancing Strategic planning, financial management, monitoring and decision-making in the education sector 1.1. Strengthening strategic Cancelled While the decision was made to drop this sub-component due to limited planning, financial policy and implementation progress, the project did support a small TA management and decision- which produced recommendations for updating the original per capita making capacity school financing formula and its implementation, but this work was carried out under the SDG sub-component 1.3. 1.2. Enhancing the existing Cancelled While the sub-component was dropped in its entirety, the MEST later EMIS expressed an interest in a scaled back version of EMIS upgrade and the Project aimed to use some unprogrammed funds to support programming of a national student registry, links to EMIS and a sub-registry Matura 12 exam entrants. Due to a one- time shortfall in budget allocated to the project by the government for year 2021, this activity wasn’t carried out. 1.3. School Development Maintained (andWith restructuring it was recognized that this subcomponent’s activities Grants later expanded contributed to efficiency improvements in addition to accountability after extensionimprovements. It also added the small TA from 1.1 to assess and produce simulated options for revising the existing per capital school financing of closing date) formula which was related to improving efficiency. Component 2. Strengthening management capacity and accountability to enhance quality of education 2.1. Implementation and Revised/ KESIP limited its contribution to teacher career system to two areas: a improvements to the teacher simplified strategic framework on teacher development and the teacher performance career system assessment pilot. In addition, the project supported school-based teacher training programs through SDGs. 2.2. Strengthening capacities Revised/ Simplified scope included: (i) development and adoption of the National of key institutions for simplified Student Assessment Framework; piloting of the reformed Grade 12 Matura monitoring education exam; (iii) capacity building of MEST staff; (iv) support to PISA results outcomes and assessment of analysis and report writing. In addition, the project supported the design student learning and piloting of the school report card. 2.3. Quality assurance of Cancelled Institutional commitment and capacity to implement higher education higher education activities within sub-components 1.1 and 2.3 were limited. Component 3. Implementation and communication support 3.1. Implementation & Maintained communication support Other Changes 29. Restructurings. The KESIP project underwent three level 2 restructurings. The first restructuring was approved on April 25, 2019 and was a result of changes agreed from the midterm review to make the project more manageable to implement, eliminate certain activities, and reduce the credit amount accordingly. The second restructuring approved on October 10, 2019 was to extend by 12 months the project closing date after the project successfully demonstrated that it met the implementation milestones agreed during restructuring by July 2019. The third Page 8 of 69 The World Bank Kosovo Education System Improvement Project (P149005) restructuring was approved on December 7, 2020 to extend the project closing date by 6 months due to implementation challenges brought on by COVID-19 school closures. The summary of changes introduced during restructurings is provided in detail in Annex 8. 30. Extension of the project closing date. In total, the original closing date was extended by eighteen months with the final closing date moved to June 30, 2021. With two extensions, KESIP’s total implementation period was 5.2 years since effectiveness. 31. Cancellation of funds. A total of SDR 2,500,000 or US$3.5 million equivalent was cancelled as part of the first restructuring. 32. Project Component Costs. As a result of project restructuring and cancellation of project sub-components and activities, there was a reallocation of funds among the components and sub-components as specified in Annex 3. Rationale for Changes and their Implication on the Original Theory of Change 33. The adjustments and changes introduced through the first restructuring were necessary to offset the broad project design targeting multiple policy areas, overestimated institutional capacity, and the unsteady commitment of the GoK to implement complex reforms. Modifications introduced to the project components reflected a joint decision taken by the GoK and the World Bank to simplify the project design and focus on school-based interventions, thus reducing the scope and overall ambition of the project. Those changes had a positive impact because they increased the prospects of achieving the KESIP targets and objectives. 34. The extension of the project closing date as a result of the project reaching agreed implementation milestones set as goals in the first restructuring, allowed KESIP to (i) provide school development grants (SDGs) to an additional 80 schools in 2020; (ii) adopt and implement the National Student Assessment Framework; (iii) design the school- and cluster-based Teacher Professional Development Program; (iv) design and pilot the School Report Card (SRC); (v) provide support to international assessment results analyses and reporting; (vi) complete full implementation of SDGs in 2021; (vii) test and finalize the SRC in 2021; and (viii) complete the technical assistance (TA) on the review and revision of the school financing formula. All these activities were critical for the achievement of the PDO. 35. Adjustments introduced through the first restructuring had an implication for the original Theory of Change (TOC). The updated TOC is provided in Annex 10 (depicted as a flowchart) and in Table 4 below that shows the revised PDO indicators and their linkages with the project sub-components, expected project outcomes (following restructuring approved in April 2019), and ultimate, longer-term PDO results to which the KESIP sought to contribute. Page 9 of 69 The World Bank Kosovo Education System Improvement Project (P149005) Table 4. Revised Results Chain, post restructuring. Corresponding Expected Project Outcomes (PAD) Ultimate Results PDO indicators, post- № Components & Sub- Strengthened Systems Contributing to (long term, high restructuring component Ultimate Results level objectives) 1. Percent of beneficiary 1.3 - School Objective/outcome 5: Strengthened school- schools disseminating Development Grants level accountability mechanisms through Improved reports on grant budget information dissemination on uses of funds efficiency and allocation and utilization for school level intervention accountability 2. Reforms to standardized 12 2.2 - Strengthening Objective/outcome 4: Improved education grade assessment applied capacities of key quality monitoring through student (Yes/No) institutions for assessments monitoring Improved quality 3. Percent of school report Objective/outcome 6: Strengthened school- and accountability cards piloted at schools educational outcomes level accountability and quality through and assessment of information dissemination on student student learning performance II. OUTCOME A. RELEVANCE OF PDOs Assessment of Relevance of PDOs and Rating 36. The relevance of objectives is rated high. The PDO was highly relevant at the time of approval and remained equally relevant at the project closing stage. The project directly responds to one of the three focus areas of the World Bank’s Country Partnership Framework (CPF) FY17-21, specifically Focus Area 2 - Strengthening Public Service Delivery and Macro-Fiscal Management” (Objective 4), which uses two outcome and two intermediate results indicators and targets of KESIP and makes a direct reference to the project as a key contributor. 37. The CPF, in turn, selectively supports Kosovo’s development agenda as outlined in the National Development Strategy (NDS) 2016-2021. The NDS is divided into four thematic pillars: human capital, good governance and rule of law, development of competitive industries, and development of infrastructure. The GoK is placing high priority on developing its human capital by making it the first pillar of the strategy and proposing interventions aimed at enhancing the efficiency of public spending in education sector, increasing accountability mechanisms and practices, establishing a more credible, transparent and comparable evaluation of student performance and enhancing quality of teaching. The KESIP addressed these priority policy areas of the NDS and is also aligned to the Kosovo Education Strategic Plan 2017-2021 which has focus areas that include capacity building at local and school level, including school governing boards, and strengthening of accountability mechanisms through teacher performance assessment development. B. ACHIEVEMENT OF PDOs (EFFICACY) Assessment of Achievement of Each Objective/Outcome 38. The April 2019 restructuring resulted in the reduced overall scope and ambition of the KESIP with cancelled activities, sub-components and outcomes as well as changes to indicators and targets. Consequently, a split rating methodology Page 10 of 69 The World Bank Kosovo Education System Improvement Project (P149005) is applied to assess the performance of the project using objectives and outcome targets at the time of restructuring and at completion. 39. Originally, the PDO was measured by four PDO indicators aligned to eight expected project outcomes contributing to ultimate, longer-term results of improved quality, accountability, and efficiency of Kosovo education system. The first restructuring resulted in no changes to the PDO and a scaling back of project scope which resulted in (1) the dropping of PDO indicators #1 and #4 and adding of one indicator; (2) elimination of subcomponents; and (3) elimination of five outcomes. Three outcomes were carried forward. Table 5 depicts the changes to expected project outcomes pre- and post-restructuring. 40. In accordance with the ICR Guidelines on the application of the split rating method, this ICR assesses the project’s efficacy, across the entire project lifetime to assign separate outcome ratings against both, the original and revised, outcomes and outcome targets. The project indicators and achievement of each expected outcome and long term, high level objectives are discussed in detail in the following paragraphs. Table 5. Expected project outcomes, pre- and post-restructuring Objectives/outcomes Expected Project Long term, Original Post Comments Outcomes High Level restruct- (PAD, p.5) Outcomes uring Objective/outcome 1: Improved Related sub-component was Objective 2. strengthen Strengthened accountability dropped selected systems that accountability through + contribute to accountability transparent budget improvements in education allocation Objective/outcome 2: Improved Even though the related sub- Improved and rationalized efficiency component was dropped, the allocation of resources project strengthened systems Objective 3. strengthen contributing to efficiency selected systems that + + improvements by supporting contribute to efficiency improved and rationalized improvements in education resource allocation through the revision of the per-capita school financing formula. Objective/outcome 3: Related sub-component was Improved education data dropped + Objective 1. strengthen collection, management, selected systems that and utilization contribute to quality Objective/outcome 4: Carried forward – strengthening improvements in education Improved education quality systems contributing to quality + + monitoring through Improved improvements Objective 2. strengthen student assessments quality and selected systems that Objective/outcome 5: accountability Carried forward – strengthening contribute to accountability Strengthened school-level systems contributing to improvements in education accountability and quality accountability improvements through information + + dissemination on student performance and sources/uses of funds Page 11 of 69 The World Bank Kosovo Education System Improvement Project (P149005) Objective 2. strengthen Objective/outcome 6: Improved Carried forward – strengthening selected systems that Strengthened school-level accountability systems contributing to contribute to accountability accountability mechanisms accountability improvements improvements in education through information + + Objective 3. Strengthen dissemination on uses of New - aligned to strengthening selected systems that funds for school level systems contributing to contribute to efficiency intervention efficiency improvements at improvements in education school level in use of funds Objective/outcome 7: Improved Related sub-component was Objective 1. strengthen Improved higher education quality dropped selected systems that quality monitoring through + contribute to quality piloting and assessment of improvements in education a licensing mechanism Objective 2. strengthen Objective/outcome 8: Improved Related sub-component was selected systems that Improved accountability of accountability dropped + contribute to accountability higher ed. institutions for improvements in education learning outcomes Outcome 1: Strengthened accountability through transparent budget allocation Outcome 2: Improved and rationalized allocation of resources 41. The achievement of both outcomes was to be measured by the single PDO indicator #1 and three intermediate results indicators (IRI) as presented in Table 6. Both outcomes were to be achieved under the same sub-component 1.1 and are assessed jointly. The original theory of change for these two aspects of the PDO was that ensuring a predictable and equitable resource allocation based on an optimal per capita financing formula, channeled through an effective and transparent budget distribution system, and supported with training and managerial capacity building at the central, municipal, school, and university levels would contribute to more efficient resource allocation and strengthened accountability in education. Table 6: PDO and Intermediate Indicators for Outcome 1 and Outcome 2 Indicators (original, PAD) Unit of Baseline End target Revised Actual Comments measure (2015) (31-Dec- end target Achieved 2019) (Dec. 2019) (June 2021) PDO1: Municipalities allocating Percentage 0% 80% Dropped Dropped resources to schools according to revised formula IRI: National and municipal-to-school Text No Revised Dropped Dropped Partially per capita funding formulas officially formulas are achieved revised and used for resource in use (modest) allocation IRI: Budgeting and financial Text Bylaws Bylaws in use Dropped Dropped management rules or bylaws for not in school autonomy approved and used place IRI: Rules for full financial autonomy in Text No rules Rules Dropped Dropped universities developed and approved in place implemented Page 12 of 69 The World Bank Kosovo Education System Improvement Project (P149005) 42. The project supported a consultancy resulting in an initial diagnosis on pre-university education financing in 2018, but the policy dialogue within the MEST on reforming pre-university education financing did not advance to guide the project to (i) develop annual monitoring procedures for national and local budgeting processes and resource allocation based on the revised per capita financing formula, and (ii) develop education spending indicators to assess and monitor education spending efficiency, along with guidelines explaining the indicators and their usage. A consultancy was procured to strengthening financial autonomy and capacity of universities, but the work was not completed due to lack of ownership and progress in higher education investment areas. Given the overall limited implementation progress under this sub-component by the mid-term review, at the first restructuring sub-component 1.1 and its corresponding PDO indicator #1 and three intermediate results indicators were dropped, as summarized in Table 6. 43. A technical assistance (TA) to support reforms of the per capita school financing formula remained an eligible activity for financing within the restructured KESIP. The TA successfully carried out (i) a review of the implementation of the existing school financing formula, (ii) consultations at the municipal and school levels, and (iii) proposed policy options and recommendations for the revision and implementation of the formula to ensure greater transparency and equity in resource allocation. A series of presentations using the findings of the assessment and revised policy options, based on six simulations, built consensus and ownership around proposed formula reforms. While this work has been successfully carried out, no new indicator was established to measure the contribution of this TA towards the ultimate higher-level objective of improved efficiency in education. With the successful execution of the TA on the financing formula, the project has partly achieved the formerly dropped objectives of (i) strengthening accountability through transparent budget allocation; and (ii) improved and rationalized allocation of resources (efficiency). 44. Based on the above, the level of achievement of Outcome 1 before April 2019 restructuring is rated Negligible as the PDO1 and two corresponding intermediate indicator targets were not attained. There is no rating of this outcome post restructuring as it was dropped. 45. The rating for Outcome 2 is considered Modest before April 2019 restructuring and Modest thereafter, given the progress made on revising the per capita financing formula and contribution towards the improved and rationalized allocation of resources (stated project outcome) and ultimate result of improved efficiency (long-term objective). Outcome 3: Improved education data collection, management, and utilization Outcome 4: Improved education quality monitoring through student assessments Outcome 5: Strengthened school-level accountability and quality through information dissemination on student performance and sources/uses of funds 46. At appraisal, the achievement of outcomes 3, 4, and 5 were to be measured by PDO indicator #2 and five intermediate results indicators as presented in Table 7. Outcome 3 was to be achieved under sub-component 1.2 (enhancing existing EMIS) and Outcomes 4 and 5 were to be achieved under subcomponent 2.2 – Strengthening capacities of key institutions for monitoring educational outcomes and assessment of student learning. Page 13 of 69 The World Bank Kosovo Education System Improvement Project (P149005) Table 7: PDO and Intermediate Indicators for Outcomes 3, 4, and 5. Indicators (original, PAD) Unit of Baseline Original target Revised Actual Comments measure (2015) (31-Dec-2019) target Achieved (Dec. (June 2021) 2019) Original PDO2: School report Percentage % 90% - - Indicator cards distributed and publicly was displayed at schools revised IRI: Education institutions Text No 80% of schools Dropped Dropped updating student-level data in student- and providers EMIS twice a year level data update student- in EMIS level data 2X/year IRI: EMIS integrated, and data Text EMIS not Integrated and Dropped Dropped exchange automated with four integrated data automated databases on teacher licensing, with all student assessment, school map, databases and expenditures IRI: Grade 9 and 12 examinations Text Grade 9 & Revised Grade 9 - - Indicator & revised in line with Kosovo 12 exams & 12 exams target Curriculum Framework (KCF) and not aligned piloted and were evaluated after piloting with KCF evaluated revised IRI: System for learning Yes/No No Yes Dropped Dropped These assessment at the primary level corporate IRI: Rating scale for primary-level Supplemental 0 2 Dropped Dropped indicators learning assessment system Value were Subtype dropped 47. Outcome 3. The original ToC for Outcome 3 (Improved education data collection, management, and utilization) was that a solid EMIS is fundamental
to Kosovo’s capacity to produce, manage and monitor education data. The objective of the corresponding sub-component 1.2 (Enhancing the existing EMIS) was to enhance and modernize the systems and practices of data collection and analysis in the education sector through development of an integrated EMIS. Legal and institutional foundations were needed to guarantee the use of the EMIS software and hardware packages, such as rules and procedures for effective information management, use, data analysis and dissemination. There was a shared understanding that without the necessary bylaws and implementation capacity in place, the large EMIS investments in software and hardware would not yield expected results. With extensive support from the WB Information Technology (IT) and EMIS specialists, the analytical work, guidelines for the legal and institutional structures within MEST, and development of technical specifications for the core central systems for integrated information management were completed in advance in 2017. 48. However, the implementation of this sub-component suffered due to uncompleted bylaws and delays and lack of transparency of the procurement evaluations. Following a mid-term review, critical milestones were set to decide if it was feasible to continue with the proposed EMIS investments under KESIP: (i) the adoption of minimal legislation to introduce the central registries had to be made before the contract for Central Systems could be signed; and (ii) the bidding process for Central Systems had to be finalized, contract signed, and work commenced by the end of January 2019. The agreed milestones were not reached as of February 2019, and the World Bank restructuring missions concluded that the risks of a failed and low-quality implementation were too high within the remaining implementation time. MEST agreed to cancel the EMIS investments, as part of April 2019 project restructuring, and use the preparatory Page 14 of 69 The World Bank Kosovo Education System Improvement Project (P149005) work completed prior for possible implementation with alternative financing sources. Consequently, all activities aimed at achieving Outcome 3 under sub-component 1.2, and corresponding intermediate results indicators were dropped. 49. Later, the MEST expressed an interest in a scaled back version of EMIS upgrade and KESIP aimed to use some unprogrammed funds to support programming of a national student registry, links to EMIS and a sub-registry for the Grade 12 Matura exam entrants. However, due to a one-time shortfall in budget allocated to the project by the government for year 2021, this activity was not carried out. Bidding documents were revised and updated and can still be used for financing under another source. Given the above, the level of achievement of Outcome 3 before April 2019 restructuring is rated Negligible and Dropped thereafter with minimal contribution to the long-term objective of improved quality and accountability. Outcomes 4 and 5 were to be achieved under sub-component 2.2. 50. Outcome 4. To successfully achieve the outcome of improved education quality monitoring through student assessments, Grade 9 and 12 exams were expected to be revised in line with the Kosovo Curriculum Framework (KCF), piloted, and evaluated, based on functioning system for learning assessment at the primary level. The Grade 5 sample- based national assessment survey would also be revised, piloted, and conducted during project life. To achieve this outcome a specialized twinning firm to help build capacity was hired in early 2018. The original timeframe required a full three years of implementation, so with two years of implementation time left, activities contributing to the achievement of Outcome 4 were adjusted to concentrate on (i) drafting, approval and launching the National Student Assessment Framework (NSAF); (ii) grade 12 Matura examination reform; (iii) capacity building of MEST in data analysis, report writing, dissemination and policy design based on PISA 2015 and PISA 2018 results data; and (iv) capacity building for other national assessments. Up to April 2019, the project had made progress with all four activities. It had supported the MEST to develop a first draft of the NSAF, concentrate on the reform of the national Matura 12 exam, and deliver training to build capacity of the MEST under (iii) and (iv) above through the twinning firm and an additional international consultant. The restructuring introduced a new PDO indicator #3 to capture the project’s contribution to education quality improvements and reflect the decision to focus on Grade 12 examination. Because Grade 5 and 9 exams were periodic (not annual) and were sample based, the project would instead focus on supporting the MEST to participate in Trends in International Mathematics and Science Study (TIMSS), Progress in International Reading Literacy Study (PIRLS) and PISA (also sample-based exams), roughly for the equivalent grades. Outcome 4 was to be further measured by two intermediate results indicators as presented in the Table 8. Page 15 of 69 The World Bank Kosovo Education System Improvement Project (P149005) Table 8: Revised PDO and Intermediate Indicators for Outcomes 4 and 5 (post-restructuring) Indicators Unit of Baseline Revised end target Actual achievement (June Comments measure (2015) (31-Dec-2019) 2021) Revised PDO2: School report Percentage 0% 20% 20% Target cards piloted at school level achieved IRI: Grade 12 examination Text Grade 12 Revised Grade 12 Grade 12 Matura exam Target revised and piloted in line exam not exam piloted and was revised & piloted. achieved with Kosovo Curriculum aligned with evaluated Results analyzed & items Framework KCF evaluated in line with KCF. IRI: Number of large-scale Number 0 4 4 Target learning assessments achieved20 completed at the primary level or secondary levels19 51. All milestones and targets set to achieve the outcome of improved education quality monitoring through student assessment were met. The project strengthened the capacity of MEST through its DSME to design, administer and process high quality examinations. This was accomplished under the twinning arrangement with Cito,21 which provided trainings to DSME staff, subject specialists, psychometricians, statisticians, and item writers on the principles of assessment, item writing and test design, statistical analysis, and reporting. The reformed Grade 12 Matura examination target was also met. The new Matura test items, aligned to curricular reforms, were designed, and then piloted in June 2019. The analysis of the pilot provided further guidance to the DSME for final adjustments to the reformed Matura 12 standardized school assessment. Despite COVID-19 restrictions, MEST was able to conduct Grade 12 Matura exam in August 2020, in accordance with newly established health and safety precautions, and using new items developed and tested. In October 2020, MEST approved and launched the National Student Assessment Framework (NSAF), which was developed with KESIP support. The MEST can now use the NSAF to guide, design, and plan execution of high-quality national student assessment. In addition, with project support, the DSME conducted in-depth analysis of PISA 2015 and PISA 2018 results data, including trends by comparing the 2015 and 2018 data, and produced reports, which were discussed with local stakeholders and disseminated. 52. The level of achievement of Outcome 4 before April 2019 restructuring is rated Substantial given that certain achievements were made in strengthening the capacity of the DSME in order to successfully conduct sample-based assessments and produce the first draft of the NSAF, which was a big milestone to achieve. The rating of this outcome post-restructuring is assessed as Substantial given that the outcome has been fully achieved. Targets for improved education quality monitoring were met. The capacity of MEST under the twinning arrangement continued to grow and enabled the DSME to design, administer, and analyze high quality examinations and large-scale student assessment surveys. The DSME is also now capable of developing testing instruments that are comparable across years as demonstrated through the revision to the Matura 12 exam which piloted new subject items and is now aligned with the competency-based curricula. These outcomes contribute substantially to the longer term (higher level) objective of contributing to quality improvements in education. 53. Outcome 5. As per original design, the achievement of the outcome of strengthened school-level accountability through information dissemination on student performance was to be measured by the PDO indicator 2 “School report cards distributed and publicly displayed at schools.” Following project simplification and subsequent restructuring, this PDO indicator was revised to remain at a pilot level as shown in Table 8. 54. With the project support and under the guidance of the Pre-University Director of the MEST, the School Report Card (SRC) concept and design were completed by the SRC working group in 2020. The design of the SRC included learning Page 16 of 69 The World Bank Kosovo Education System Improvement Project (P149005) assessment results, school statistics, and the municipalities averages for comparison purposes. It was initially planned that the data for the SRCs would be generated automatically from the upgraded EMIS, but this subcomponent was dropped. Despite not having automation, the MEST was able to manually generate the SRCs from the existing EMIS and pilot them during the month of March 2021 in the 160 schools that received SDGs in 2018-2019. After the pilot, DSME put forth a solution that could automate the production of SRCs using SPSS programming22 that would enable the Ministry to go to scale for all pre-university schools, reducing the time to one week. In addition to the SRC template, the Administrative Instruction for the SRCs and a user manual with indicator definitions were designed and used in the pilot. The project team reported that overall experience with the SRC pilot was very positive and that SRCs evoked a greater spirit in schools to provide good data as the data included in the SRCs comes from both school reporting to the central level as well as centrally generated data on individual schools. A survey of the SRC pilot schools indicated that schools welcome the SRC and information in the SRCs on school performance facilitated a greater degree of collaboration with the community and among schools. Paired with the School Development Plan, it is another informational tool that helps to focus attention on school improvement goals and account for school efforts to achieve those goals by clearly stating the learning outcomes achieved, school statistics and how the school compares to municipal averages. 55. Based on the above, the rating for Outcome 5 is considered Modest before April 2019 restructuring and Substantial after the restructuring given that the outcome target has been fully achieved. Through project support, the MEST is now well equipped to regularly produce school report cards which has enabled schools to use performance data for better school improvement planning and community support. This outcome contributes substantially to the long term, higher level objective of improvement in accountability. Outcome 6: Strengthened school-level accountability mechanisms through information dissemination on uses of funds for school level intervention 56. As per original design, the achievement of this outcome was to be measured by PDO indicator 3 “Beneficiary schools disseminating report on grant budget allocation and utilization” and three intermediate results indicators, as summarized in Table 9. These outcomes were to be achieved under subcomponent 1.3 – School Development Grants. PDO indicator #3 remained unchanged but renumbered as PDO Indicator #1. The target for the PDO indicator was adjusted from 80% to 60% of schools receiving SDGs reporting, since at the time of restructuring it was not known if the project would meet the milestones set for measurement in July 2019 and be eligible for an extension of closing date. One intermediate results indicator was revised, and one new intermediate indicator was added, both measuring effectiveness of SDGs. 57. At restructuring, subcomponent 1.3 which aligned with outcome 6 and originally only pointed to contributing to the long term, higher level objective of improved accountability, added ‘improvements to efficiency.” The theory of change to support this addition is as follows. Given the literature on school grants and school-based management contributing to more efficient use of funds because decision-making and management of those funds are closer to the local level, it was an opportunity to expand alignment of this subcomponent to contribute to the long term, higher-level objective of improvements in efficiency in education albeit limited to the school level. At the school level, this sub-component aimed to support school financial autonomy by strengthening the financial management capacities of school directors and boards. This would still be measured through the PDO of Percent of beneficiary schools disseminating reports on grant budget allocation and utilization. An intermediate indicator was added to measure the number of schools developing plans for quality investments through school grants. Also to measure this subcomponent’s contribution toward the 19 This new/revised corporate indicator was added in March 2021 and reflected in ISR#10 20 Learning assessments included in this indicator: PISA 2018, TIMSS 2019, Annual Matura 12 exam in 2019 and 2020. 21 The twinning firm selected to support the work on student assessments 22 Software package used for interactive, or batched, statistical analysis. Page 17 of 69 The World Bank Kosovo Education System Improvement Project (P149005) longer term, higher level objective of efficiency improvements, the project relied on an independent evaluation of the grants for measuring the effectiveness of KESIP School Development Grants in enhancing school-level capacities which covered among others school governance – strengthened financial management (better use of funds towards achieving school development plan activities) as demonstrated through the ability to apply for, manage, and report on use of SDG funds. Table 9. Original and revised PDO indicator and IRIs for Outcome 6 Indicators Unit of Baseline End target Revised end Actual Comments measure (2015) 31-Dec-19 target Dec-19 Achievement (June 2021) Original PDO3/ % 0% 80% 60% 99.58% Target Revised PDO1: Beneficiary (Restructuring exceeded schools disseminating 1) 90% (post- report on grant budget restructuring allocation and utilization target)* IRI: Number of schools that Number 0 At least 150 240 Target receive SDGs 150 exceeded IRI: Number of beneficiary Number 0 At least 150 240 Target schools in which parents 150 exceeded and/or community members were involved in grant planning or implementation (citizen engagement indicator) New IRI added: Schools Number 0 - 150 240 Target developing plans for quality exceeded investments through school grants Original IRI: Percent of % 0 Increase - - IRI was school directors reporting from year 2 revised improved planning capacity collection Revised IRI: Feedback from Text No Assessment Endline survey Target school directors on assessment findings on completed. 77% say achieved improved planning capacity principals’ SDGs effectively capacity improved planning building capacity *Note: The end-of-project target was revised upward after the first restructuring took place and reflected in sub-sequent ISRs. 58. The KESIP achieved Outcome 6 through its successful implementation of the SDGs despite obstacles and challenges that caused delays in channeling funds to municipalities and schools, at various stages, to finance the school development plans. As per the original design, at least 150 schools were expected to have received the SDGs and 80% of those were expected to be disseminating their reports on grant budget allocation by project completion. By the end of 2019 (original closing date), the project had already met the PDO indicator target established during the restructuring (60% of beneficiary schools disseminating report on grant budget allocation and utilization) and exceeded the end targets of three IRIs (with 160 schools23 having developed, with parents and community members participation, school development plans for enhanced teaching and learning, and having received school grants to implement those plans). Given the satisfactory performance in execution of SDGs to 160 schools, the decision was made by the project and the 23Out of those 160 schools, 140 schools are pre-selected schools from disadvantageous backgrounds, selected against pre- defined criteria, and 20 schools are competitively selected. Page 18 of 69 The World Bank Kosovo Education System Improvement Project (P149005) MEST to finance a second round of SDGs to 80 additional schools on a competitive basis (i.e., the quality of the school development proposals). The SDG amount was increased for the second batch of 80 schools from US$10,000 to US$15,000 each. Schools from the second call for SDG proposals were awarded in February 2020 and the PCU managed the whole process of working with schools to implement their plans during COVID-19. All 80 selected schools, received required training, developed their school development plans, and executed their SDGs to finance activities aimed at enhancing the quality of teaching and learning. 59. A qualitative survey was carried out in a sample of 140 beneficiary schools to study the effectiveness of the SDGs on enhancing the functionality of the school governing mechanisms (School Governing Boards (SGB), parents, student, and teacher councils). The baseline, midline and endline surveys were conducted by an external firm allowing comparisons to the baseline data. Overall, survey respondents assessed KESIP as highly effective in improving the functionality of school governance mechanisms. Figure 1 below presents results of the survey: almost 85% of respondents assessed KESIP as most effective in improving the performance of teacher councils, to be followed by improved functionality of SGBs (77.1%), student councils (63.6%), and parent councils (37.9%). When asked specifically in which areas they observed most improvements in comparison to the pre-SDG period, most respondents (64.3%) ranked participation in SGB meetings as an area with most improvement since the implementation of KESIP grants. Figure 1: KESIP SDG Evaluation Results – Reported impact on improving functionality of school governance mechanisms Source: DOTS. 2019. Measuring the Effectiveness of ESIP School Development Grants in Enhancing School-Level Capacities: Endline Survey, Figure 35. 60. In total, KESIP awarded 240 schools with SDGs on the basis of their school development plans, which were developed with participation of teachers, parents, students and communities to enhance quality of teaching and learning environments in schools. Thus, the end-of-project target of 150 for all three intermediate indicators was exceeded by 60 percent, supported, in turn, by a higher budget allocation to SDGs during the restructuring. With respect to the PDO indicator (Percentage of beneficiary schools disseminating report on grant budget allocation and utilization), the revised end-of-project target of 90 percent was exceeded by 11 percent. Based on this, the rating for Outcome 6 is considered Substantial before April 2019 restructuring and Substantial after the restructuring given the noteworthy achievements made under the SDGs. The capacity of publicly financed schools was substantially strengthened to plan and manage resources, and promote citizen engagement through the participation of teachers, parents and communities in planning, implementing, and monitoring school activities. Subcomponent 1.3 and Outcome 6 contribution to the long term, high level objective of improved accountability is deemed high and the contribution to Page 19 of 69 The World Bank Kosovo Education System Improvement Project (P149005) the high level objective of improved efficiency is modest since the project was limited to influencing improvement in efficient use of funds at the school level through the SDGs. With the two combined, the contribution is Substantial. Outcome 7: Improved higher ed. quality monitoring through piloting and assessment of a licensing mechanism Outcome 8: Improved accountability of higher education institutions for learning outcomes 61. KESIP included activities and sub-components focused on higher education, even though the project was mainly covering pre-university education. Outcomes 7 and 8 were to be achieved through implementation of activities under sub- component 2.3: Improving accountability and quality assurance of higher education through the development of key professional licensing statues in priority economic fields. The objective of this sub-component was to strengthen the capacities of MEST to carry out professional licensing and establish a mechanism for holding higher education institutions accountable for outcomes. 62. The achievement of Outcomes 7 and 8 was to be measured by PDO indicator 4 “Number of licensing exams piloted” and one intermediate results indicator “Bylaws on professional licensing developed and approved.” From the early stages of project implementation, the higher education activities showed little progress, including no action plans prepared or policy discussions held. Institutional commitment and capacity to lead investments in higher education policy was limited. Following the mid-term evaluation at restructuring in April 2019, sub-component 2.3 was dropped along with corresponding PDO-level indicator 4 and intermediate indicator specified above. The level of achievement of Outcomes 7 and 8 before April 2019 restructuring is rated Negligible and Dropped thereafter . There was no contribution to the long term, high level objective of quality and accountability. Table 10 presents the summary of Efficacy rating results for each outcome prior and post-restructuring by both eight detailed objectives/outcomes (as defined in the PAD) and by three aggregated outcomes showing that overall efficacy rating remains the same under both approaches. Page 20 of 69 The World Bank Kosovo Education System Improvement Project (P149005) Table 10: Summary of Efficacy ratings PDO: to strengthen select systems that contribute to quality, accountability, and efficiency improvements in education in the Republic of Kosovo Long Term Outcomes Outcomes Rating At restructuring At completion Improved accountability Objective/outcome 1: Strengthened Negligible Dropped accountability through transparent budget allocation Improved efficiency Objective/outcome 2: Improved and rationalized Modest Modest allocation of resources Improved quality and accountability Objective/outcome 3: Improved education data Negligible Dropped collection, management, and utilization Improved quality and accountability Objective/outcome 4: Improved education quality Substantial Substantial monitoring through student assessments Improved quality and accountability Objective/outcome 5: Strengthened school-level Modest Substantial accountability and quality through information dissemination on student performance and sources/uses of funds Improved accountability and efficiency Objective/outcome 6: Strengthened school-level Substantial Substantial accountability mechanisms through information dissemination on uses of funds for school level intervention Improved quality Objective/outcome 7: Improved higher education Negligible Dropped quality monitoring through piloting and assessment of a licensing mechanism Improved accountability Objective/outcome 8: Improved accountability of Negligible Dropped higher education institutions for learning outcomes PDOs: OVERALL EFFICACY RATING Modest Substantial Objective 1. strengthened selected Objectives/outcomes 3, 4, 5, 7 Modest Substantial systems that contribute to quality improvements in education Objective 2. strengthened selected Objectives/outcomes 1, 3, 4, 5, 6, 8 Modest Substantial systems that contribute to accountability improvements in education Objective 3. strengthened selected Objectives/outcomes 2 and 6 Modest Modest systems that contribute to efficiency improvements in education OVERALL EFFICACY RATING Modest Substantial 63. The assessment of the project outcomes as described above did not include the critical intermediate results related to the development and strengthening of the teacher professional development and teacher performance assessment. The interventions aimed at improvements of the teacher career system were not reflected in the PDO or the eight expected project outcomes as depicted in Table 1 and their progress was measured instead through the intermediate results indicators. The revised post-restructured Results Framework included two intermediate results indicators related to teacher development and assessment and both of them were met, as shown in Annex 1. The simplified design of sub- component 2.1 provided TA to develop the policy framework for school-based teacher professional development, and Page 21 of 69 The World Bank Kosovo Education System Improvement Project (P149005) design and pilot the teacher performance assessment, thus contributing to the strengthened management capacity and accountability to enhance quality of education. Justification of Overall Efficacy Rating 64. Overall, the project achieved its revised outcome indicators and intermediate results targets, with several exceeded. The restructuring approved in April 2019 improved the project’s outcome rating by reducing the scope of the project from a complex program focused on multiple policy areas, to three core sub-systems: SDGs, learning assessments, and teacher assessment and professional development. These investments still directly responded and contributed to the project’s ultimate, long-term, objectives of improving quality, accountability, and efficiency of education in Kosovo. Overall efficacy, therefore, is rated Modest at the April 2019 restructuring and Substantial at completion. Table 11 provides an overview of the selected systems supported under KESIP, the PDO indicators, long term outcomes, eight expected outcomes, and efficacy ratings. Page 22 of 69 The World Bank Kosovo Education System Improvement Project (P149005) Table 11: KESIP Select Systems, PDOs, Outcomes, and Efficacy Ratings PDO: Strengthen select systems that contribute to quality, accountability, and efficiency improvements in education in the Republic of Kosovo. Efficacy Rating PDO indicator Status of PDO Long term Outcome Restruc Comple Achievement Outcomes turing tion (Higher level) System 1: Strategic Planning and financing management capacities (1.1) (dropped) % of municipalities allocating resources to schools Dropped but Accountability 1 - Strengthened accountability through N - according to revised formula. Partially achieved and efficiency transparent budget allocation under Outcome 2 2 - Improved and rationalized allocation of M M resources System 2: EMIS (1.2) (dropped) 3 - Improved education data collection, N - management, and utilization System 3: School-level accountability and efficient use of resources (SDGs) (1.3) % of beneficiary schools disseminating reports on Exceeded Efficiency and 6 - Strengthened school-level accountability S S grant budget allocation and utilization accountability mechanisms through information dissemination on uses of funds for school level intervention System 4: Institutional Capacity to Monitor Educational Outcomes and Assessment of Student Learning (2.1&2.2) Reforms to Grade 12 Standardized Exams (new) Achieved Quality 4 - Improved education quality monitoring S S through student assessments % of school report cards distributed and publicly Achieved Quality and 5 - Strengthened school-level accountability and M S displayed at schools Accountability quality through information dissemination on student performance and sources/uses of funds System 5: Higher education accountability and QA through key professional licensing statures in priority fields (2.3) Number of licensing exams piloted Dropped – not Quality and 7 - Improved higher education quality monitoring N - achieved Accountability through piloting and assessment of a licensing mechanism 8 - Improved accountability of higher education N - institutions for learning outcomes N = negligible, M = modest, S = Substantial Overall Efficacy Rating M S Page 23 of 69 The World Bank Kosovo Education System Improvement Project (P149005) C. EFFICIENCY Assessment of Efficiency and Rating 65. Economic Analysis at Appraisal. The economic and financial analysis in the PAD comprised a qualitative assessment grounded on international evidence instead of a Cost-Benefit Analysis (CBA), because the original structure of the Project mainly targeted system reforms. Notwithstanding, the project restructuring allowed a refocusing of funding and efforts towards educational interventions with a higher positive effect on education quality at the school level: i) School Development Grants used to finance teacher training and learning materials, ii) learning assessment, and iii) teacher assessment. 66. Economic Analysis at Completion. Although the economic and financial analysis in the PAD did not provide baseline financial indicators, this ICR identifies and quantifies the beneficiaries of SDGs to calculate the Net Present Value (NPV) and the Internal Rate of Return (IRR) of the Project. The SDGs provided by the Project led to education quality improvement because they were used to finance School Development Plans focusing on the eligible categories of teacher training and learning materials. International evidence has identified teaching effectiveness as a key predictor of student learning,24 to such an extent that it can even offset learning deficits of disadvantaged students.25 The Project originally targeted 160 schools, from which 140 were selected using a pro-equity approach (schools in disadvantaged and rural areas) and the remaining 20 were competitively selected. After restructuring, the Project expanded SDGs to benefit 80 additional schools, all competitively selected. 67. Individuals collect the economic benefits of education in the form of wages when they access the labor market. The education quality provided to individuals during their school life affects their development, career opportunities and future accumulation of capital,26 as well as their returns to education (i.e., expected income).27 Adopting a pro-equity approach when investing in education is a cost-effective way to reduce social inequality because a higher educational attainment and better quality of education elevates the income of the poor by means of its effect on the average wages, and its influence on the distribution of income.28 68. The ex-post economic analysis estimated the economic benefits stemming from the SDGs (i.e., wage increase due to better education quality resulting from teacher practice improvement) to compute the NPV and IRR of the Project. The expected economic benefits of the learning and teacher assessment frameworks were not explored because they entail system-wide reforms making it difficult to quantify the direct beneficiaries of the project. The estimated project NPV ranges between US$6 million to US$37 million, and the IRR fluctuates between 8.9 and 20.1 percent when considering a wage increase ranging from 1 to 3 percent due to the education quality improvement promoted by the SDGs (see Annex 4 for more detail). Since this analysis considers the economic benefits resulting from SDGs exclusively, the exhibited NPV and IRR values can be considered as a lower bound of all the benefits accruing from the Project. 69. Aspects of Design and Implementation that Influenced Efficiency. KESIP was designed to support selected system strengthening in a number of pre-university and university education policy areas at the central, municipal and school 24 Evans and Popova (2015); Hanushek (2011); World Bank (2013). 25 Hanushek et al. (2005); Hanushek and Rivkin (2010); Rockoff (2004). 26 Psacharopoulos and Patrinos (2004). 27 Lazear (2003). 28 Grundlach, Navarro de Pablo and Weisert (2004); Klump and Miralles (2007). Page 24 of 69 The World Bank Kosovo Education System Improvement Project (P149005) levels. The focus of the project was aligned to the needs across various systems in education: budgeting, information systems, school-based management, teacher performance and development, and learning assessments. However, this selected systems support was still too broad to match the available institutional capacity. Following April 2019 restructuring, KESIP’s scope was reduced from the original focus on multiple education policies to three core policies: SDGs, learning assessments and teacher assessment and development. 70. In terms of implementation, several factors influenced operational efficiency of the project, including: • High turnover in MEST during the life of the project and constant PCU staff and consultant rotations, especially prior to KESIP’s first restructuring in 2019. After mid-term restructuring the stability and expertise of the Financial Management (FM), Procurement, and project activity consultants offset the short life of the Project Coordinators that continued to be rotated with each change in the administration (three governments in the last 17 months of the project). • Political instability resulting in reorganization processes and high turnover in leadership and upper management in the Ministry of Education (four changes of Ministers and three General Secretaries). • Procurement experienced serious challenges and delays, mainly due to a lack of full-time staff at the PCU during the first year of the project implementation and due to inexperienced MEST procurement bid evaluation committee members who lacked, in some cases, relevant expertise, commitment, and understanding of the World Bank procedures despite special training sessions. This contributed to three cancelled procurement packages. • Lack of communication on SDG budgets, processes and requirements between the Ministry of Finance, MEST, municipalities, and schools delayed SDG disbursements and payments. • A one-time understated project budget for 2021 and Treasury’s decision to hold discussions on the Budget Law after the closing date of KESIP left the project without access to the full amount of the credit, thus leading to the cancellation of two extra activities29 planned to be financed with remaining funds. 71. By the end of 2018, after 2.7 years of implementation, only US$0.681 million or 6.2 percent of the originally allocated US$11 million were disbursed. Two approved extensions of the project closing date resulted in the additional project implementation period of 1.5 years. 72. Many of the project’s implementation inefficiencies from the first half of the project were overcome following the first restructuring (April 2019). A period of intense focus and support from the highest levels of the MEST leadership after restructuring combined with an empowerment of MEST component leaders and the PCU that was strengthened with experienced fiduciary staff, added to the project’s improved efficiency. This strengthened capacity coupled with a simplified project design, and the Bank team’s close supervision enabled implementation and disbursements to pick up dramatically, thus leading to a successful achievement of the revised project objectives. In the last two years of implementation, 86 percent ($5.68 million) of the Project’s total financing of US$7.5 million were disbursed. While the project budget for 2021 was understated and KESIP couldn’t access the full amount of the credit, the project still managed to complete all activities successfully by end-of-project. 73. Weighing the detected operational shortcomings with the successful achievement of the PDOs, the project’s efficiency is judged to be Substantial. Economic analysis at project completion pointed to several factors that support substantial rating: i) the Project adopted a pro-equity approach, which has been identified as a cost-effective way to reduce social inequality;30 ii) the Project invested SDG resources in teacher effectiveness, namely the key predictor of student 29 This included: a consultancy for developing the student registry and update of the EMIS, and the procurement of tablets in response to COVID-19 needs to assist disadvantaged students with distance learning. 30 Ibid. Page 25 of 69 The World Bank Kosovo Education System Improvement Project (P149005) learning;31 and iii) the ex-post economic analysis showed that all the project costs can be economically justified by the benefits derived from the SDGs alone. D. JUSTIFICATION OF OVERALL OUTCOME RATING 74. The relevance of the PDO was High at appraisal and continued to remain High at project closing. The project’s overall efficacy is rated Modest before the April 2019 restructuring and Substantial after the April 2019 restructuring. The April 2019 Level 2 restructuring simplified the project, aligning the project design with the institutional capacity for implementation. This has ensured immediate satisfactory progress in project implementation and by project closing all PDO and intermediate outcome targets were achieved or exceeded, as described in section II (B) above, thus justifying the substantial rating after the restructuring. The project’s efficiency is rated Substantial at project closing. 75. Given the narrowed scope of the project, post-restructuring, with dropped activities, corresponding sub-components and PDO and intermediate results indicators, and cancellation of unused funds, a split rating methodology is being applied to arrive to the overall outcome rating for KESIP. Table 12 below summarizes the results of the split evaluation for overall outcome rating. Table 12: Summary of the Overall Outcome Rating At restructuring At completion Relevance of PDOs High Efficacy Modest Substantial Outcome 1 Negligible - Outcome 2 Modest Modest Outcome 3 Negligible - Outcome 4 Substantial Substantial Outcome 5 Modest Substantial Outcome 6 Substantial Substantial Outcome 7 Negligible - Outcome 8 Negligible - Objective 1. strengthened selected systems that contribute to Modest Substantial quality improvements in education Objective 2. strengthened selected systems that contribute to Modest Substantial accountability improvements in education Objective 3. strengthened selected systems that contribute to Modest Modest efficiency improvements in education Efficiency Substantial Outcome ratings Moderately Unsatisfactory Moderately Satisfactory Numerical value of outcome ratings 3 4 Disbursement US$0.92 million US$5.68 million Share of disbursements 0.14 (or 14%) 0.86 (or 86%) Weighted value of outcome ratings 3*0.14 = 0.42 4*0.86 = 3.44 Final outcome rating Moderately Satisfactory (0.42 + 3.44 = 3.86 rounding it to 4) 31 Evans and Popova (2015); Hanushek (2011); World Bank (2013). Page 26 of 69 The World Bank Kosovo Education System Improvement Project (P149005) 76. The weighted outcome value of 3.86 (rounded to 4) corresponds to a Moderately Satisfactory rating, on the 1 to 6 scale, for the final overall outcome. This rating is consistent with fully achieved objectives. The Moderately Satisfactory rating of the ICR for the overall project outcome is consistent, by and large, with the ratings of the achievement towards the PDO as reflected in the ISRs. E. OTHER OUTCOMES AND IMPACTS (IF ANY) Gender 77. Although there were no specific gender interventions under the KESIP, a qualitative study on girls’ transition to upper secondary education, and boy’s retention in secondary education has been undertaken under the Western Balkans Investment Facility financed education TA in 2016 and the findings of the study have been shared with all local stakeholders. The gender criterion, which was one of the eligibility criteria for selection-based grants to schools under the previous IDEP project, was removed as an indicator under the KESIP’s SDG sub-component because EMIS data showed that gender parity has been achieved at the primary level.32 78. In the Results Framework, the project reported on the number of students benefitting from direct interventions to enhance learning, disaggregated by gender, as part of the mandatory corporate results indicators. Overall, 51 percent of the 100,155 students benefitting from project interventions were female, while also exceeding the end-of-project target of female beneficiaries (37,500 female students) by 35 percent. Institutional Strengthening 79. The project’s two sub-components/activities specifically addressed institutional strengthening: (i) SDGs; and (II) Learning Assessments. With the aim of strengthening the overall school development planning system and assisting school project teams with preparation and execution of their multiple-year school development plans, KESIP’s SDG sub-component provided trainings on school development planning, financial and project management, procurement, monitoring, and school self-evaluation. In total, the project provided these trainings to school management committees (which included participation of community members/parents) in 240 schools, thus exceeding the target of 150 schools. A qualitative survey carried out in 140 beneficiary schools to assess the effectiveness of the SDGs on school-level capacities and autonomy, concluded that KESIP was highly impactful in improving the functionality of school governance mechanisms (see Figure 1) with 77 percent of respondents assessing KESIP as most effective in improving the functionality of School Governing Boards (effectiveness of teacher councils was rated highest at 85 percent). The trainings that were provided have also enhanced the capacity of SDG recipient schools to apply for future funding and grants (from local government and other donors). 80. The main activities supporting institutional strengthening of the MEST included building the capacity of the Division for Standards, Monitoring and Evaluation to design, administer and oversee sample-based and census-based national and international student assessments, analyze data and produce performance reports, and monitor educational outcomes, thus contributing to the ultimate results of improved education quality and accountability throughout Kosovo’s education system. The capacity building trainings were provided under the twinning arrangement with Cito and supported by an international consultant. Beneficiaries included DSME staff, item writers, subject specialists, statisticians, and psychometricians. Ultimately, the MEST is now much better equipped to design robust, comparable national exams (to monitor progress and evaluate trends from one year to the next) and to analyze the results of national assessments (Matura 12) and international assessments the country participates in (PISA, TIMSS, PIRLS). The 32 June 16-27, 2016 Aide Memo, World Bank. Page 27 of 69 The World Bank Kosovo Education System Improvement Project (P149005) improved abilities for data analysis also contribute to better evidence-based decision-making to channel support to teachers and bolster weak areas in student performance. The DSME also has a deeper awareness of methods to guard the security of tests. Through support of the project, the MEST was enabled to produce and approve the National Student Assessment Framework that sets the foundations for all student assessment and provides guidelines for all levels of the education system. Mobilizing Private Sector Financing 81. KESIP did not envision engagement or mobilizing of private sector financing at the time of approval, nor did such mobilization happen during the project implementation. Poverty Reduction and Shared Prosperity 82. The school development grants, through the eligibility criteria for selection, targeted rural and remote schools which are often associated with poorer educational outcomes. Schools were selected on a need-based formula that included the following three criteria and their corresponding weights: No. Criterion Number of points (weight) Minority enrolment (in% - # of minority students divided by the total # a. 1.5 of students enrolled in the school year 2016/17) Poverty level (in% - # of students under social assistance scheme b. 2 divided by the total # of students enrolled in the school year 2016/17) Grade 9 assessment test (average of school wide test results in the c. 1.5 school year 2015/16) Total points 5 83. Given that SDGs aimed to support schools in rural and remote areas with a higher proportion of socioeconomic disadvantaged groups, a higher score was given to schools with higher number of students under the social assistance scheme. Therefore, the project’s main beneficiaries under the SDG sub-component were students from disadvantaged backgrounds. The new corporate indicator introduced in the ISR#10 (March 2021) reported on “Students benefiting from direct interventions to enhance learning” – the project exceeded the end target by 33.5 percentage points with 100,155 students benefitting from enhanced learning environment in 240 schools that received and executed SDG grants. 84. In addition, options proposed for revising the financing formula for schools included the following measures: (i) equalization of the coefficient for mountainous areas with that of minorities; and (ii) creating a separate budget line for “goods and services” for children with special needs using a greater coefficient than for other groups. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION 85. Project design. KESIP’s design was ambitious aiming to support education system strengthening activities and key institutional reforms that reflected government’s priorities and were aligned to the needs of the education systems and sub-systems, including education financing, information systems/EMIS, a functioning teacher career system, Page 28 of 69 The World Bank Kosovo Education System Improvement Project (P149005) comprehensive learning assessment system, and school-based management. In addition, the project design included higher education sub-components and activities, which added to the project’s broad agenda and inclusion of many areas of investment within a single project. Even with capacity building activities built into the project, it wasn’t enough to overcome project complexity, the Borrower’s weak implementation capacity and fragile nature of MEST leadership. During the April 2019 restructuring, in recognition of project design complexities and ambitiousness, project design was more realistically framed to mainly focus on the school level service delivery (examinations, SDGs, school-based teacher professional development) with some activities aimed at capacity building and system strengthening at central level. 86. Beneficiary targeting. Given the project’s focus on selected system level changes, its beneficiary targeting was broad, stating that project’s interventions “would affect beneficiaries across all levels of the education system” (PAD, pages 15-16). However, it also appropriately targeted schools located in rural and remote areas, with the highest number of students from disadvantaged socio-economic backgrounds. At appraisal stage, the project expected to benefit approximately 45,000 students. This end target was tracked through the regular project monitoring by recording student enrollment, disaggregated by gender, in beneficiary schools. Beneficiaries and stakeholders identified, appropriately, under the SDG subcomponent included teachers, administrators, parents, and other staff employed in those schools. The planned targets and regular project monitoring served as a guidance for keeping track of impact on beneficiaries. It also helped during project simplification to know where the greatest benefits were reaching and continue to focus targeting under the SDGs, learning assessment and teacher professional development interventions. Direct beneficiaries included (i) students and teachers in SDG beneficiary schools; and (ii) DSME staff, subject specialists, psychometricians, statisticians, and item writers under the learning assessment sub-component. Indirect beneficiaries included teachers, administrators and parents. 87. Risks and mitigation measures. The overall risk rating for KESIP at appraisal was moderate which, one could argue, represents an underrating, given (candidly acknowledged) overall weak institutional capacity, fiscal constraints, and recognition that the “project would be implemented against a backdrop of a fragile country” (PAD, page 23). A couple of risks that materialized may have needed more attention at appraisal stage in terms of appropriately reflecting those in the project design. Specifically, the following two policy areas were appropriately identified as “potentially contentious areas that will require political commitment to adopt and sustain changes”: teacher career and higher education (PAD, page 23). These risks materialized, requiring Level 2 restructuring, under which the teacher career sub-component was simplified and reduced in scope, while the higher education sub-component and activities were dropped altogether. The PAD did not describe specific mitigation measures for the two abovementioned risks. The project did identify, nevertheless, other mitigation measures, described in different sections of the PAD, that sought to address technical, management and fiduciary capacities of the MEST and PCU through provision of capacity building efforts in learning assessment area, procurement, and financial management. 88. Fiduciary risk rating. At the time of appraisal, the overall financial management risk was considered Moderate even though the “risk pertaining to insufficient or untimely budgetary allocations”33 was considered Substantial (PAD, page 48). The procurement risk was rated as Substantial as well, based on procurement challenges experienced under the previous IDEP project (PAD, page 52). While those risks were clearly assessed, an overall fiduciary risk was underestimated having been rated Moderate. The risks of “insufficient or untimely budgetary allocations” as well as risks related to procurement capacity did materialize during project implementation which led to delays and influenced the dropping of activities. 33 This risk materialized during implementation as well, leading to financing gaps Page 29 of 69 The World Bank Kosovo Education System Improvement Project (P149005) 89. Readiness for implementation. Implementation readiness was ensured through the hiring of the PCU Coordinator and fiduciary staff, preparation of the procurement plan for the first 18 months, and the Project Operational Manual (POM), including an SDG Manual. The SDG sub-component included necessary implementation mechanisms and tools: the manual, implementation support firm that provided direct support to schools through trainings and capacity building and another firm to evaluate the impact of SDGs. All of that allowed for the SDG sub-component to advance more successfully than other project areas. The SDG Manual was later updated to reflect flexibilities introduced to address COVID-19 disruptions. B. KEY FACTORS DURING IMPLEMENTATION Factors subject to government and implementing entities control • Frequent changes of government MEST leadership. As an institutional reform project, KESIP heavily relied on the commitment of the government to sector priorities and policies that KESIP sought to support. Frequent changes of the government and MEST leadership meant that new leadership had to gain an understanding of the foundations being put in place, as well as an understanding of the project. This led to inconsistent policy guidance in key system- wide sectoral reforms (i.e., teacher career scheme, financing reform, student registry), and, in turn, a lack of clarity at the technical level on how to proceed with a number of activities. • Changing organizational structure at MEST. A high turnover rate of personnel in many MEST departments, with interim directors with limited knowledge of the project, led to lack of continuity, focus, policy guidance and proactivity, which further resulted in delays and inadequate implementation of the agreed steps and actions. The role of the PCU and relationship with MEST departments has been confusing at times due to lack of institutional memory with frequent leadership changes. It can also be attributed to the lack of clear organizational structure of MEST as a result of internal reorganizations that happened during the life of the project. At times this led to confusion about the MEST’s overall management roles, consultants’ technical roles, and the PCU’s fiduciary, accountability, and support roles. • Constant PCU staff rotations throughout project life. KESIP had six Project Coordinators over the course of the project life with a long period of acting personnel (e.g., from October 2017 to 2018 as recorded in the World Bank’s mid-term evaluation). Staff turn-over of fiduciary personnel (with 3 procurement specialists and 2 FM specialists) occurred in the first half of the project and during that time, while experienced and qualified, the procurement officer was part-time which prolonged procurement processing. As of January 2019, full-time FM and Procurement Specialists were hired and they remained until project closing, allowing for some stability at the PCU. 90. The above mentioned institutional, structural and management challenges resulted in significant implementation delays and low disbursements during the first years of the project. The World Bank’s December 2017 implementation support visit concluded that no component has reached the service delivery level, after 1.5 years of project implementation, and a “proactivity plan” was agreed with the MEST to facilitate progress. Despite the efforts by the counterparts and the World Bank team to improve project implementation, the mid-term evaluation (November 2018) revealed only moderate improvements with only one year left to the original project closing date (December 2019). At this time actual disbursements were less than 10 percent. The agreement was then reached between the World Bank and GoK to simplify and reduce the scope of the project. The subsequent restructuring missions agreed on the methodology and criterion for non-inclusion of specific activities and sub-components in the simplified and restructured KESIP. • Setting clear goals, empowering staff and holding them accountable for delivery. A set of milestones with agreed targets to be achieved by July 2019 provided clear goals for implementation improvement. This proved to be a Page 30 of 69 The World Bank Kosovo Education System Improvement Project (P149005) significant turning point for the project. Following the April 2019 restructuring, the Ministry appointed the Minister’s Chief of Cabinet to serve as Acting Project Coordinator. The MEST leadership on KESIP was thus considerably strengthened and communication among the PCU and with MEST Department Heads improved. Daily meetings were installed, and the discipline instilled in the team to monitor project implementation and work in a more coordinated manner with MEST departments and implementation partners proved to be vital for reaching the milestones. Per agreement with the Bank, the PCU produced bi-monthly progress reports and held regular meetings with the Bank team to discuss the progress and address implementation issues. Project implementation picked up significantly and ratings of the PDO and overall implementation progress were upgraded to Moderately Satisfactory by January 2020. Given this surge in productivity and achievement of the milestones, the project was back on track and able to make advances in project implementation and achievement of the project objectives. Other factors that affected implementation: • Change in disbursement arrangements for SDGs. At the start of the project, the Ministry of Finance had requested a change in the disbursement arrangements for the SDGs whereby the funds dedicated for school grants would be managed by the MEST and not transferred to the municipal and school level accounts, as foreseen during the design stage. A significant time was spent on clarifying such arrangements and reaching agreements between the PCU, MEST, and World Bank team and this delayed the start of implementation of the SDG component. This led to delayed time in signing of Grant Agreements with municipalities, a step not foreseen in the process as well. • Periodic non-compliance with several clauses of the Financing Agreement, including preparation and submission (not later than November 15 of each fiscal year) of an annual work plan and budget, maintaining a functional Project Steering Committee and a fully staffed PCU. • Administrative Instruction (AI) from the Ministry of Finance (MoF) on PCU salaries. The MoF instituted an AI that capped the salaries of the PCU staff and consultants. This situation resulted in the contradiction to the terms of the KESIP Financing Agreement, created uncertainties among PCU staff and caused delays in hiring consultants and renewal of the contracts, until the issue was clarified.34 Factors subject to World Bank control. 91. The World Bank team provided continuous guidance and close supervision throughout the project implementation period, including on the technical, monitoring, and fiduciary aspects. Though there were three Task Team Leaders (TTLs) from preparation to project completion stage, a proper hand-over was ensured by carrying out joint implementation support missions. Each TTL was fully committed to the project, maintained open and continuous dialogue with the MEST and PCU, was results oriented and was able to support counterparts in resolving technical and fiduciary bottlenecks. A local Human Development (HD) Specialist was hired in the World Bank’s country office in 2019, and this aided in frequent communication and further hands-on support to the government and the PCU. The World Bank also made available a local technical expert who was part of the KESIP project team from the project design stage to the very end of project completion, supporting the MEST and the PCU on technical aspects. This ensured institutional memory and a critical coordination between the Bank, the PCU and MEST. Extraneous factors. 92. Over the five years of project implementation, frequent changes in political leadership affected priority setting, decision making, and efficiency. During the life of the project there were four Ministers of Education. Beginning at the 34The MoF Legal Department issues a clarification to the MEST specifying that the signed Financing Agreement takes legal precedence. Page 31 of 69 The World Bank Kosovo Education System Improvement Project (P149005) end of 2019 through February 2021 there were three elections bringing in three new Ministers and General Secretaries at MEST. These frequent changes subsequently led to changing political agendas, conflicting directions to the PCU from the authorities within the MEST, disruption in processes, inefficiencies and slowed overall progress. In 2020, despite great efforts by the PCU and MEST to keep implementation moving forward, certain setbacks were caused by COVID-19 disruptions resulting in the extension of the project closing date by an additional six months to allow completion of project activities. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E) M&E Design 93. KESIP’s packed and ambitious policy agenda translated into a compound PDO that was broadly stated (“strengthen select systems”) and included longer-term outcomes (or “ultimate results” as stated in the PAD) that could only be expected to materialize in the long run, as correctly acknowledged in the PAD. The project’s eight expected outcomes were presented in an accompanying table included in the PAD (Table 1 in this report). The table illustrated linkages between the four PDO indicators, the sub-component activities, eight expected project outcomes and where they would contribute to the ultimate results of improved quality, accountability, and efficiency. 94. At appraisal, the Monitoring and Evaluation (M&E) system included a Results Framework with four PDO-level indicators and fifteen intermediate results indicators, including two supplemental indicators. These results indicators were generally sufficient to measure the achievement of project’s expected eight outcomes. While also found being relevant, given project’s focus on system-level reforms, the original PDO targets were probably unrealistic, given that (i) all the PDO indicator baseline values were at zero at the time of project preparation and the end-of-project targets for three indicators ranged between 80-90 percent; and (ii) the project was supposed to be implemented over a period of only four years, which was ultimately further reduced by 4.5 months, given delays with project effectiveness. 95. The changes introduced during April 2019 restructuring simplified the project design, thus aligning it with institutional capacity, and adjusted the results framework and outcome targets to better align them with the reduced scope of the project. Following the restructuring, the Results Framework included three expected outcomes, three PDO indicators and seven intermediate results indicators. In 2021, three more intermediate indicators were added, which were WB corporate indicators (ISR#10). These indicators are found to be generally sufficient to measure achievement of the three expected outcomes. However, with the dropping of subcomponent 1.1, its corresponding Outcomes 1 and 2, 35 the relevant PDO and intermediate results indicators, it was not possible for the project team to reflect the contribution of the technical assistance on the improvement of the school financing formula in the Results Framework. The TA was concluded in the last year of project implementation through funds under subcomponent 1.3. 96. The following data collection instruments were envisioned at appraisal: regular data collection processes, complemented by implementation support visits, administrative data, and semi-annual monitoring reports prepared by the PCU. The project’s Results Framework at appraisal outlined data sources and responsible entities. Separate M&E arrangements were designed for the SDG sub-component and were detailed in the SDG Manual. It described M&E responsibilities of the Implementation Support Firm, schools’ self-evaluation procedure, M&E responsibilities of 35 Improved and rationalized allocation of resources. Page 32 of 69 The World Bank Kosovo Education System Improvement Project (P149005) the municipalities and PCU, World Bank monitoring and auditing responsibilities, and the purpose of the qualitative study assessing effectiveness of SDGs to be carried out in beneficiary schools at baseline, midline, and endline points. 97. The project design also envisaged monitoring of the citizen engagement over the project implementation cycle by measuring the participation of community representatives in planning and decision-making concerning the implementation of the school grants. M&E Implementation 98. M&E implementation relied on existing data collection arrangements established at the MEST, reporting by schools benefitting from SDGs, as well as the Implementation Support Firm, contracted to support execution of SDGs. Additionally, an independent three-phase survey for the SDGs was contracted to measure the effectiveness of the school development grants in enhancing school-level capacities: composition and functionality of school governing mechanisms (SGBs, parent, student, and teacher councils) and effectiveness of school management in implementing School Development Plans. 99. Quarterly and annual progress reports were produced by the PCU. To allow for a close monitoring of implementation progress following April 2019 restructuring, the agreement was reached between the World Bank and the PCU that PCU will produce bi-monthly progress reports with a focus on outcome reporting. 100. The project team was proactive in adapting the Results Framework to implementation challenges, as part of the restructuring, by adjusting the indicators and setting realistic targets. Monitoring of the SDGs was carried out by the Implementation Support Firm who submitted regular monitoring reports to the PCU. Site visits to SDG recipient schools by the PCU and WB project team members were used to both review and provide guidance/remedies on problematic matters identified. 101. For all indicators in the revised Results Framework, the PCU was able to collect the requisite data using administrative data or project reporting. For an intermediate results indicator on the improved planning capacity of school directors, a three-phase data collection exercise (a baseline, midline and endline survey) was carried out as a means of measuring the effect (and reporting on it) of SDGs on school-level capacities. These baseline, midline and endline assessments allowed comparisons against the baseline. M&E Utilization 102. The project M&E data were appropriately used throughout the project implementation period and informed decision-making. The data were regularly used to report on the status of project implementation and results achieved and reflected in the ISRs and Aide Memos. The M&E data were also used particularly when the project was restructured, and Results Framework appropriately adjusted. 103. The School Report Cards as designed, produced and tested provided a useful tool that facilitated self-evaluation and school-level monitoring, and will be useful in initiating a dialogue with the school community on their performance. A qualitative survey carried out in 140 SDG beneficiary schools (in three waves) to study the effect of the SDGs on school-level capacity and autonomy, provided valid measurement of change over time and offered valuable feedback that was used by the project team to make necessary adjustments during the implementation of SDGs. Justification of Overall Rating of Quality of M&E Page 33 of 69 The World Bank Kosovo Education System Improvement Project (P149005) 104. Overall, M&E data on KESIP performance and results was appropriately and timely collected, adequately used to track whether the project was progressing as planned, and informed decision-making to adjust the project during implementation. While the PDO was compound and broadly stated, as described above, the PAD spelled out the specific expected project outcomes, thus allowing the assessment of each outcome. As part of the restructuring, the KESIP’s expected outcomes were reduced from eight to four, and PDO indicator targets were adjusted, reflecting the reduced scope of the project. In addition, the capacity of the DSME was enhanced in monitoring of educational outcomes and reporting standards. There were moderate shortcomings in the M&E system’s design at the project preparation stage (as described above), but the broadly stated and compound PDO was supplemented by the explicitly formulated expected project outcomes and a collection of PDO-level and intermediate results indicators was sufficient to assess the achievement of the project’s PDO. Weighing the strengths of the project’s M&E throughout implementation period with the moderate shortcomings identified at the design stage, the ICR rates the overall quality of M&E as Substantial. B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE 105. Environmental. There were no civil works foreseen under the project and the POM, as well as the School Grants Manual, included specific language regarding the ineligibility of civil works under the project components and sub- components. Give that the project design did not trigger environmental safeguards, KESIP was classified as a Category “C” project. 106. Social. The project promoted citizen engagement through participation of teachers, parents and communities in planning, implementing, and monitoring school activities financed by the school development grants. 107. Procurement. The overall risk rating was Substantial at appraisal. This was due to the procurement challenges experienced by the previous IDEP project. While the mitigation measures were adequate and stated in the PAD, they were not always complied with. The procurement risk mitigation measures suggested recruiting a full-time Procurement Specialist. It took the project more than two years to comply with this requirement, as the initially hired Procurement Specialist, while being very competent and experienced with World Bank-financed operations, was there on a part-time basis, which was followed with a few weeks of vacancy prior to the new Procurement Specialist joining the PCU team. In general, procurement practices under KESIP have been found to follow Bank procedures and no major issues were identified during the procurement reviews and implementation support missions, except minor comments related to entering complete contract information in STEP36 and other issues which typically would be resolved with the help of the Bank’s procurement specialist. Nevertheless, KESIP’s procurement was characterized by significant delays in several procurement packages due to weak capacity of MEST evaluation committees, insufficient understanding of the World Bank procurement rules by the MEST staff, lack of communication and unclear responsibilities between the PCU and MEST departments (as revealed during the mid-term evaluation review). Interviews carried out as part of this ICR preparation, with local World Bank country office staff revealed that the issue of the weak capacity of the evaluation committees at implementing agencies is a problem across the whole World Bank portfolio and to ameliorate this situation, the in-country WB Procurement Specialist provided face-to-face training and hands-on support to the members of the MEST procurement evaluation committee. The Bank’s Country Management Unit has also supported the project by allocating time from the local WB Procurement Specialist to advise on procurement activities. The procurement performance improved with the hiring of a new full-time Procurement Specialist who started in January 2019. Simplifying the project through the midterm restructuring and adding more support through a part-time, experienced procurement officer as a 36 An online system to help the World Bank and borrowers plan and track procurement activities under Bank-financed projects. Page 34 of 69 The World Bank Kosovo Education System Improvement Project (P149005) recommendation at the time of the milestone review (July 2019) also improved procurement. The ISR adequately adjusted procurement ratings37 in line with the project’s implementation progress and disbursements. 108. Financial Management (FM). The overall FM risk was considered Moderate at appraisal. The project relied extensively on various elements of Kosovo’s public FM system and the project’s annual budgets were included in the overall MEST budget. Therefore, the project’s budgeting process followed MEST’s budget preparation process and was coordinated with the MEST Budget and Finance Division (BFD). The PCU FM Specialist worked closely with BFD on planning and budgeting of project activities to ensure funds were available for financing project expenditures, on commitments and payments, as well as preparation of quarterly Interim Unaudited Financial Reports (IFRs). The MEST budget would pre-finance project expenditures to be reimbursed from the IDA credit proceeds based on quarterly financial reports that document such expenditures. Appropriate annual project budget needs had to be requested in a timely manner by MEST to MoF as part of the regular budget preparation cycle and would be included in the Budget Law. Given the characteristics of the pre-financing mechanism, the risk pertaining to insufficient or untimely budgetary allocations was identified at the appraisal stage and rated Substantial. This risk, in fact, materialized during project implementation, resulting in delays related to SDG execution during the earlier stages of project implementation and cancellation of planned activities in the last few months of project implementation, thus impeding full disbursement of the IDA credit before the closing date. Overall, the project suffered from the delays in the annual budget law, difficulty in accurately forecasting budgets for SDGs because the government budget runs on a calendar year and implementation of the grants takes place during the academic year, and at times, weak coordination among MoF, MEST and municipalities on transferring SDG funds to schools. Following the April 2019 restructuring though, the MEST, municipalities, and MoF streamlined the SDG transfers and instituted a Memorandum of Understanding that laid out an agreement between MEST and municipalities on transferring funds to schools. This made it clear that municipalities were to allocate those funds to schools and schools were to follow WB procurement rules. 109. The FM, in general, have been found to follow Bank procedures, and FM requirements were complied with throughout the project’s implementation period. The FM rating was downgraded once in ISR#6 (September 2018) from Satisfactory to Moderately Satisfactory given budget delays for school grants. 110. IFRs were submitted on time and in accordance with established Bank procedures. Annual audits included SDG audits and were submitted on time, all with unqualified opinion (clean) on financial statements, revealing no significant irregularities. C. BANK PERFORMANCE Quality at Entry 111. Quality at entry is rated Moderately Unsatisfactory. While the objectives of the project were strongly aligned and responsive to sector challenges and priorities as identified in the national development and education sector strategies, they required continuous commitment of the government to implement. KESIP was a follow on to IDEP and was designed in a fragile political context and within a system with acknowledged institutional capacity limitations that the project aimed to strengthen. The KESIP PAD clearly identified and acknowledged those constraints, as did the CPS FY12-16. The CPS Completion and Learning Review offered the following lessons: “Institutional capacity constraints, which are typical for a young and fragile democracy, should be addressed across the whole program, and taken into consideration during 37There is a discrepancy in ISR ratings related to procurement – in terms of what the actual rating shows and the main text of the ISR, which was explained by the fact that information entered/updated in PRAMS and ISR do not coincide. Page 35 of 69 The World Bank Kosovo Education System Improvement Project (P149005) preparation of new operations. Weak institutional capacity, and poor inter-institutional cooperation affected performance of operations in sectors such as health, education…”. It further suggested that “Special attention should be paid to readiness for implementation to ensure that operations design responds to clear commitment and readiness for reforms, while avoiding complex project design and complex institutional implementation arrangements.” Similar lessons on the need for less complexity in the project design and more careful assessment of the enabling environment were concluded by the ICR for IDEP and ICR review report, produced by the Independent Evaluation Group, while describing similar fragile country context and implementation challenges that were experienced by IDEP throughout its life span: high staff turn-over of key management staff in PCU, including fiduciary personnel; political instability; frequent changes in Ministry leadership; non-compliance with legal covenants; and low disbursements. So, overall, while the World Bank undertook thorough diagnosis and analysis of the sector issues and challenges during project preparation, and focused on strengthening the capacities of MEST, the project design proved to be more complex than originally assessed, given Kosovo’s fragile political environment, lack of institutional capacity, management fragilities at PCU, and time required to advance complex reforms. 112. The economic and financial analysis at the design stage comprised a qualitative assessment without a cost-benefit analysis. The technical, environmental, and social risk assessments were appropriate; however, not all identified risk mitigation measures were proposed as discussed earlier in the report. Some shortcomings identified by this ICR in the M&E design are discussed earlier as well (section IV.A). Quality of Supervision 113. The World Bank performance at supervision is rated Satisfactory. The Bank team maintained continuous and close communication with key counterpart players on the ground and worked proactively with the MEST and PCU, with a strong focus on achieving development impact. The Bank’s support was effective in overcoming techni cal, administrative and procurement challenges faced throughout project implementation. The project team consisted of the staff based in both HQ and the Kosovo country office, plus a local technical consultant who supported the project throughout its preparation and implementation period and provide institutional memory. Though there was a period of about two years when there was no local HD staff in the country office, the Bank project team relied on support provided by the CMU and the local technical consultant who sustained a continuous dialogue with the client by maintaining bi- weekly virtual progress review meetings with the PCU and TTL, which facilitated the close monitoring. The hiring of a local HD specialist in 2019 bolstered continuous hands-on support and follow-ups to address operational bottlenecks in a context of a weak institutional capacity and an overall challenging political environment. The CMU also allocated time from the local WB procurement specialist to advise MEST and PCU on procurement activities. 114. Overall, the Bank team demonstrated an impressible implementation support by holding seventeen review and implementation support missions, including a mid-term evaluation review, and two back-to-back restructuring support missions – on average every four months. An intensive support provided throughout 2018 (with five implementation support visits – every 2.4 months on average) and 2019 (four visits – every 3 months on average) was critical in turning the project around and setting it on the path towards achievement of the PDOs. The interview with the current TTL, at project closing, also revealed that close communication and support to the PCU was ensured through monthly video conferences and meetings from 2019 and onwards. High quality ISRs and Aide Memos were prepared and filed, documenting implementation progress, and reporting candidly and comprehensively on accomplishments, implementation challenges and potential risks. Critical issues were brought to the attention of the Bank management through ISRs, appropriately adjusting project ratings, which were consistent with supervision reports, downgrading to reflect implementation shortcomings and upgrading when there were improvements. The Aide Memos included summaries of agreed actions and next steps to keep the project on target. The team was also responsive to changing Page 36 of 69 The World Bank Kosovo Education System Improvement Project (P149005) circumstances and proactive in processing three restructurings to simplify project design to align it to existing institutional capacity, to adjust the Results Framework and extend the project timeline to allow the achievement of the PDOs. 115. There were other strengths in Bank supervision. An important achievement is the work done by the Bank supervision team to influence the PCU’s reporting from outputs to outcomes, which was reflected in PCU regular reports. Justification of Overall Rating of Bank Performance 116. The World Bank supervision was of high quality, in particular addressing shortcomings at project preparation stage and turning around the physical and financial implementation pace and ensuring achievement of all revised outcome and intermediate results indicators. However, given the Moderately Unsatisfactory rating at entry and Moderately Satisfactory rating for the overall project outcome, to which the ICR arrived at using the split evaluation method, the overall World Bank performance is rated as Moderately Satisfactory. V. RISK TO DEVELOPMENT OUTCOME 117. The risk to development outcome is considered substantial, given continued fragile political context, weaknesses in institutional capacity, and coordination among and within the government institutions at central, municipal and school levels. The current government is new and is in the process of drafting the next five-year education sector strategy. While the government plans to convene donor support around this strategic sector plan, it is not known yet when the draft of this document will be ready and what reforms the government will prioritize. The Bank carried out analytical work this year on early childhood development, and there is potential but no formal request yet for a possible follow-up operation that could support the government to build on the successes of the project. 118. Some continued investments will be required to sustain capacities built in learning assessments. The consulting positions financed by the project for the DSME unit (statisticians, IT specialists, and testing specialists) were not regularized and incorporated into the Ministry’s budget, as expected, so the unit continues to function with inade quate number of staff members and rely on donor support to finance positions. This situation poses significant risk to sustaining KESIP gains in strengthening the system’s capacity to monitor educational outcomes. 119. Therefore, the progress in sustaining the project’s achievements mentioned in Section II.B of this ICR will largely depend on the ability of the new Kosovo government to: (i) build consensus around next capacity building steps and formulate a sound education sector strategy rooted in data and evidence; (ii) mobilize partnerships and investments to support sector policies and strategy; (iii) ensure the MEST leadership is in the driver’s seat for the implementation of sector reforms; and (iv) further build capacity of its staff, involved in the implementation of sector reforms, to manage projects and resources and deliver services in an effective and timely manner, especially at the regional and local levels. VI. LESSONS AND RECOMMENDATIONS 120. A thorough assessment of the institutional capacity and enabling environment (clear commitment, strong ownership, human capacity, financial resources) for initiating and managing system-wide complex reforms is of primary importance in a context of a young, fragile country like Kosovo, and likely to improve the chances for success in terms of development impact. Carrying out a thorough political economy analysis, at the appraisal stage, in all project targeted areas, could have provided a better understanding of the interests and motivations of key stakeholders, especially in Page 37 of 69 The World Bank Kosovo Education System Improvement Project (P149005) potentially contentious policy areas, and aided a greater selectivity of project interventions. In the case of KESIP’s preparation, two reform areas were correctly identified as potentially sensitive ones (higher education and nation-wide teacher career development system, linking salary increases to professional development) that require strong political commitment to adopt and sustain policy changes. This recognition, however, did not translate into a more politically feasible intervention strategy in the KESIP design. An optimism about the GoK’s commitment to complex system -wide reforms and capacity to implement them resulted in a complex project design with a broad scope and unrealistic timeframe. 121. In a context of constantly changing political environment, having a stable project management structure, including an appropriately staffed, professional project implementation/coordination unit is important for the achievement of project development objectives. In the case of KESIP, the PCU with its frequent staff rotations, especially of the Project Coordinator and prior to project restructuring, became the source of fragility itself, resulting in lost institutional memory, implementation delays and adding to other challenges the project was facing. This required an extraordinary effort on the part of the Bank project team to maintain implementation momentum during extended periods of the government transition. The Bank’s recognition of the need to reinforce the PCU with supplemental part-time experienced FM and procurement staff after restructuring helped bring stability and capacity building to the permanent staff. They were kept on contract as needed when activities ramped up. 122. Government leadership, commitment, ownership and effectiveness are significant factors for project success. This needs to be exhibited in a number of areas, including strong management commitment, sustained support for dedicated project implementation team, and provision of incentives to staff. When the Ministry’s high-level leadership provides consistent guidance and there is a experienced and capable PCU Coordinator, it allows departments heads and PCU staff to be empowered, know what the goals are, and held accountable for performance. They step up and get things done. In contrast, when leadership gives mixed signals, is not consistent or absent, even with discipline and clear goals, it makes the implementers and PCU’s job more difficult. They spend a lot of time explaining procedures and less able to move implementation forward in a coordinated manner. 123. Identification of ‘champions/leaders’ among public service staff, within specific departments, those interested to move reforms ahead, is crucial to ensure institutional memory, ownership, and smooth implementation. 124. Setting short term implementation milestones agreed and noted in Aide Memoires, can be an effective tool for staying on track and making complex tasks more manageable. Milestones and regular check in meetings can be a significant tool for turning around implementation problems. The discipline to monitor and work in a coordinated manner with departments and implementation partners towards a shared goal proved to be the key to reaching KESIP’s milestones. 125. A simple and focused design, with few critical activities, is most suited in the context of complex, fragile environments characterized by weak institutional capacity. While the KESIP preparation team recognized that the project would be implemented against the backdrop of a precarious political environment and fiscal constraints and strived to strike a balance between realism and ambition, the KESIP’s original design proved to be more complex than initially assessed, with a too broad scope, covering several key reform areas in two education sub-sectors. At the closing of IDEP – precursor of KESIP, political instability was still affecting project’s implementation and ICR for IDEP concluded that the “mismatch between existing capacities and the complexity and ambitiousness of project design” caused serious delays in the early years of project implementation. It seems that this important lesson on the right “balance between the levels of complexity and ambitiousness of project design and institutional and administrative capacities” was somewhat underappreciated during the preparation of the KESIP. Page 38 of 69 The World Bank Kosovo Education System Improvement Project (P149005) 126. Formulation of a concise and clear PDO helps track progress and demonstrate achievements clearly. The composite and broad nature (“strengthen select systems”) of KESIP’s PDO that also articulated longer-term impact of “quality, accountability, and efficiency improvements in education in the Republic of Kosovo” that was beyond the purview of the project, required clarification and consultations with the TTL responsible for the project preparation, as well as Bank Guidance for the preparation of the ICR, as to how organize the assessment of the achievement of KESIP’s PDO. The approach undertaken by this ICR, to ensure a meaningful outcome evaluation, was based on the analysis of the project documentation detailing KESIP’s design (that specify eight expected outcomes as a result of project interventions), and the ICR author’s interpretation of the Bank Guidance for the preparation of the ICR that suggest in the case of compound PDOs for the assessment to be organized around each outcome separately, further advising that “it is important to assess efficacy at the level of expected outcomes and avoid just presenting data of the project’s results indicators”. Avoiding composite and broadly stated PDOs would leave little room for differing interpretations and facilitate evaluating . achievements of the project objectives. Page 39 of 69 The World Bank Kosovo Education System Improvement Project (P149005) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: Schools managing resources and quality investments to improve learning Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Beneficiary schools Percentage 0.00 80.00 60.00 99.58 disseminating report on grant budget allocation and 30-Sep-2015 30-Sep-2015 25-Apr-2019 30-Jun-2021 utilization Comments (achievements against targets): Target exceeded. Under KESIP, 240 schools were awarded and fully implemented their SDG grants. Of the 240 schools, all but one disclosed their SDG budget allocation and utilization reports. The revised end-of-project target of 60% was introduced as a result of the midterm review's project restructuring. This target was later revised upwards to 90%, based on project implementation progress, but was not formalized through restructuring. Nevertheless, both targets were exceeded. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion School report cards piloted at Percentage 0.00 0.00 20.00 20.00 Page 40 of 69 The World Bank Kosovo Education System Improvement Project (P149005) school level 30-Sep-2015 30-Sep-2015 25-Apr-2019 30-Jun-2021 Comments (achievements against targets): Target achieved. The school report cards were piloted in 160 primary schools that received school development grants, which represents about 20% of all primary schools in Kosovo. Note: There is no original target as this PDO was revised. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Reforms to Standardized 12 Yes/No No Y Yes grade Assessment Applied 01-Jan-2019 01-Jan-2019 30-Jun-2021 Comments (achievements against targets): Target achieved. The new style of test items were tested during the 2019 Grade 12 Matura exam and then used in 2020 Grade 12 Matura exam. Analysis and outcomes will continue to guide the reforms of the Grade 12 standardized school leaving assessment. A.2 Intermediate Results Indicators Component: Enhancing strategic planning, financial management, monitoring and decision-making in the education sector Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Page 41 of 69 The World Bank Kosovo Education System Improvement Project (P149005) School developing plans for Number 0.00 150.00 150.00 240.00 quality investments through school grants 30-Sep-2015 30-Sep-2015 25-Apr-2019 31-Dec-2020 Comments (achievements against targets): Target exceeded. Target is considered met after the training is provided to school management committees on the development of school improvement plans and further preparation of the school development plans, to be further financed through SDGs. By project end, 240 schools prepared school development plans for investments under KESIP, thus exceeding the end-of-project target by 60%. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of schools that Number 0.00 150.00 150.00 240.00 receive School Development Grants 30-Sep-2015 30-Sep-2015 25-Apr-2019 30-Jun-2021 Comments (achievements against targets): Target exceeded. By project end, 240 schools have received the full amount of their SDGs, thus exceeding the end-of-project target by 60%. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of beneficiary Number 0.00 150.00 150.00 240.00 schools in which parents and/or community members 30-Sep-2015 30-Sep-2015 25-Apr-2019 30-Jun-2021 were involved in grant planning or implementation Page 42 of 69 The World Bank Kosovo Education System Improvement Project (P149005) (citizen engagement indicator) Comments (achievements against targets): Target exceeded. This is the citizen engagement indicator reporting on the number of SDG-recipient schools in which parents and/or community members were involved in grant planning or implementation. Under KESIP, the school management committees were created in all 240 grant-recipient schools to lead the preparation of the school development plans and support execution of the SDGs. Those school management committees included parents and community members. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Students benefiting from Number 0.00 0.00 75,000.00 100,155.00 direct interventions to enhance learning 30-Sep-2015 30-Sep-2015 01-Jul-2020 30-Jun-2021 Students benefiting from Number 0.00 0.00 37,500.00 50,893.00 direct interventions to enhance learning - Female Comments (achievements against targets): This corporate results indicator was added in July 2020 (ISR #10) to report on the number of students benefiting from direct interventions to enhance learning. The indicator was calculated using school enrollment data for SDG-recipient schools. By project end, total of 100,155.00 students benefitted from direct interventions to enhance learning, thus exceeding the end-of-project target of 75,000 students by 33.5%, while also exceeding the target of 37,500 for girls by 36%. Component: Strengthening management capacity and accountability to enhance quality of education Page 43 of 69 The World Bank Kosovo Education System Improvement Project (P149005) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Functional teacher Text No Yes Yes, Teacher professional development Competencies system Framework Designed Framework Published 30-Sep-2015 30-Sep-2015 30-Jun-2021 Comments (achievements against targets): Target achieved. Target was met when the Teacher Professional Competencies and Development Framework was published and launched. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Functional teacher Text No Yes Yes, Pilot application performance assesssment completed system is in place. 30-Sep-2015 30-Sep-2015 30-Jun-2021 Comments (achievements against targets): Target achieved. Target was met when teacher performance assessment was piloted, and new system approved. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Grade 12 examinations Text Grade 12 exams not Grade 9 and 12 exams Grade 12 Matura Grade 12 Matura Page 44 of 69 The World Bank Kosovo Education System Improvement Project (P149005) revised and piloted in line aligned with KCF aligned with KCF Exam piloted and exam was revised and with Kosovo Curriculum evaluated piloted in Spring 2019. Framework Results were analyzed and items evaluated December 2019. 30-Sep-2015 30-Sep-2015 25-Apr-2019 30-Jun-2021 Comments (achievements against targets): Target achieved. Grade 12 Matura exam was revised in line with the Kosovo Curriculum Framework and validated through piloting in Spring of 2019. Results were further analyzed, and items evaluated in December 2019. Note: At midterm restructuring the target was adjusted by dropping Grade 9 sample-based exam. Instead the country participated in PISA 2018. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of large-scale Number 0.00 0.00 4.00 learning assessments completed at the primary or 30-Sep-2015 30-Sep-2015 30-Jun-2021 secondary levels Comments (achievements against targets): Target achieved. This is a corporate results indicator. International learning assessments included in this indicator are: PISA 2018 and TIMSS 2019. National Learning Assessment included in this indicator are: Annual Grade 12 Matura exam conducted in 2019 and 2020. Component: Implementation and Communications Support Page 45 of 69 The World Bank Kosovo Education System Improvement Project (P149005) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Feedback from school Text No Assessment Increase from midline Assessment findings 77% report SDGs directors on improved survey on Principal's capacity effectively improved planning capacity building planning capacity 30-Sep-2015 30-Sep-2015 25-Apr-2019 30-Jun-2021 Comments (achievements against targets): Target achieved. This indicator reports on the qualitative assessment of the SDG-recipient school directors’ planning capacity. The baseline, midline and endline surveys were conducted by an external firm allowing comparisons to the baseline data. The endline survey’s in-depth interviews with school directors generally revealed their high level of satisfaction with the functionality of school governing mechanisms. School directors report improvements in planning capacity to formulate school development plans and to manage and report on their grants, relative to the baseline. Page 46 of 69 The World Bank Kosovo Education System Improvement Project (P149005) B. KEY OUTPUTS BY COMPONENT Objective/Outcome 1: Improved education quality monitoring through student assessments Outcome Indicators 1. Reforms to standardized 12 grade assessment applied (Yes/No) 1. Grade 12 examination revised and piloted in line with Kosovo Curriculum Framework. Intermediate Results Indicators 2. Large-scale primary/secondary learning assessment completed (number). 1. The Kosovo National Framework for Student Assessment developed, approved, published in three languages (Albanian, Serbian and English) and disseminated. 2. New open ended and style items developed 3. Annual Matura 12 examination, containing new items, was piloted and carried out in 2019 and 2020. Key Outputs by Component 4. The report on PISA 2015 results, containing in-depth analyses with policy recommendations (linked to the achievement of the completed in 2017, published and disseminated to all relevant stakeholders. Objective/Outcome 1) 5. National PISA 2018 results report prepared in 2019, and an in-depth report with policy recommendations prepared in 2020. 6. DSME supported to participate in TIMSS/PIRLS and analyze results. 7. Trainings provided to DSME staff, item writers, subject specialists, statisticians, and psychometricians. Objective/Outcome 2: Strengthened school-level accountability and quality through information dissemination on student performance and sources/uses of funds Outcome Indicators 1. Percent of school report cards piloted at schools Intermediate Results Indicators Note: there is no intermediate results indicator linked to this outcome and PDO-level indicator 1. The concept, content and design of the School Report Card (SRC) prepared, containing Key Outputs by Component information on enrolment, teachers, students, school types (e.g., urban vs. rural, municipal average, (linked to the achievement of the national average) and student assessment/examination performance. Objective/Outcome 2) 2. SRC piloted in 160 schools (representing 20 percent of schools in the country) that are also recipients of the SDG grants. Page 47 of 69 The World Bank Kosovo Education System Improvement Project (P149005) 3. First draft of the Administrative Instruction for the SRC prepared Objective/Outcome 3: Strengthened school-level (efficient use of resources) accountability mechanisms through information dissemination on uses of funds for school level intervention Outcome Indicators 1. Percent of beneficiary schools disseminating reports on grant budget allocation and utilization 1. Schools developing plans for quality investments through school development grants. 2. Number of schools that receive School Development Grants. 3. Number of beneficiary schools in which parents and/or community members were involved in Intermediate Results Indicators grant planning or implementation. 4. Students benefitting from direct interventions to enhance learning (disaggregated by gender). 5. Feedback from school directors on improved planning capacity. 1. Total of 240 schools prepared their school development plans to be financed through SDGs. 2. First SDG round: 160 schools benefited in 26 municipalities, of which 140 were schools with disadvantaged populations selected by the MEST based on low socio-economic characteristics and low student performance. Another 20 schools were selected on a competitive basis. 3. Second SDG round: a total of 80 schools benefited in 21 municipalities; all competitively selected. Key Outputs by Component 4. A total of 100,155 students benefitted from direct interventions to enhance learning, out of (linked to the achievement of the which 50,893 were female students. Objective/Outcome 3) 5. A total of 6,896 basic education teachers benefitted from the training/professional development provided through SDG funds in support of school development plans. 6. Implementation assessment of the current school financing formula for pre-university education was carried out and simulated options for improving the formula were formulated and presented to key local stakeholders, building consensus and ownership of the reform options that could ensure that in the next budget the new school financing formula could be applied. Page 48 of 69 The World Bank Kosovo Education System Improvement Project (P149005) ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION A. TASK TEAM MEMBERS Name Role Preparation Flora Kelmendi Task Team Leader Kashmira Hoshang Daruwalla Procurement Specialist Jonida Myftiu Financial Management Specialist Andrea C. Guedes Team Member Margaret Png Counsel Jose C. Janeiro Team Member Zlatan Sabic Team Member Bekim Imeri Social Specialist James Gresham Team Member Lucas Gortazar Team Member Supervision/ICR Angela Demas Task Team Leader Arben Maho Procurement Specialist Jonida Myftiu Financial Management Specialist Mrike Aliu Team Member Kushtrim Plakolli Procurement Team Blerim Saqipi Team Member Mjellma Rrecaj Team Member Lalaina Noelinirina Rasoloharison Team Member Natasa Vetma Environmental Specialist Bekim Imeri Social Specialist Fabiola Altimari Montiel Counsel Jose C. Janeiro Team Member Lisa Lui Counsel Nadeem Mohammad Operations Advisor Joel E. Reyes Team Member, Former Task Team Leader Anne N. Ranasinghe Procurement Team Page 49 of 69 The World Bank Kosovo Education System Improvement Project (P149005) B. STAFF TIME AND COST Staff Time and Cost Stage of Project Cycle No. of staff weeks US$ (including travel and consultant costs) Preparation FY14 23.731 98,736.86 FY15 29.983 118,288.52 FY16 2.850 36,534.02 FY17 0 164.50 FY18 0 281.51 FY19 0 1,578.11 Total 56.56 255,583.52 Supervision/ICR FY15 0 0.00 FY16 1.875 34,515.51 FY17 26.998 110,338.58 FY18 19.016 159,954.15 FY19 17.326 160,775.32 FY20 40.457 197,109.52 Total 105.67 662,693.08 Page 50 of 69 The World Bank Kosovo Education System Improvement Project (P149005) ANNEX 3. PROJECT COST BY COMPONENT (original and restructured) Table A: PROJECT COST BY COMPONENT AND PERCENTAGE DISBURSED Amount at Amount at Actual at Percentage of Percentage Components Approval Restructuring Project Closing Amount at of Approval (US$M) (US$M) (US$M) Restructuring Enhancing strategic planning, financial management, monitoring 5.2 3.13 3.34 64% and decision-making in the education sector Strengthening management capacity and accountability to 5.0 3.72 2.55 51% 69% enhance quality of education Implementation and 0.8 0.66 .71 89% 107% Communications Support Total 11.00 7.52 6.60 60% 88% Table B: ORIGINAL AND RESTRUCTURED KESIP BUDGET AND ACTUAL PROJECT COSTS Original components & sub-components Restructuring status Original allocation Restructured Actual, US$ (PAD) (restructuring #1) US$ (PAD) US$ allocation US$ Component 1. Strategic planning, financial management, monitoring 5,200,000 3,126,622 3,343,609 and decision-making in education sector 1.1. Planning, financial management and Cancelled 1,100,000 43,742 98,217 decision-making capacity 1.2. Enhancing of existing EMIS Cancelled 2,100,000 22,950 32,257 1.3. School Development Grants Maintained 2,000,000 3,059,930 3,213,134 Component 2. Management capacity and accountability to enhance 5,000,000 3,716,664 2,545,984 quality of education 2.1. Improvement to teacher career system Revised/simplified 1,000,000 716,664 536,240 2.2. Monitoring education outcomes and Revised/simplified 3,000,000 3,000,000 1,969,111 assessment of student learning 2.3. Quality assurance of higher education Cancelled 1,000,000 0 40,634 Component 3. Implementation and communication support 800,000 656,714 707,415 3.1. Implementation & communication Maintained 800,000 656,714 707,415 Total 11,000,000 7,500,00038 6,597,008 38The Data Sheet indicates USD7,515,200 as a revised amount of financing at project closing on June 30, 2021. The difference of USD15,200 is due to currency exchange/fluctuations since the restructuring in April 2019. Page 51 of 69 The World Bank Kosovo Education System Improvement Project (P149005) ANNEX 4. EFFICIENCY ANALYSIS 1. The implementation of the KESIP aimed to strengthen select systems that contribute to quality, accountability, and efficiency improvements in education, by means of a loan of US$11 million that would be disbursed over a period of four years (starting in 2016 and closing in 2019).39 KESIP was a follow up to the IDEP project and as such was intended to strengthen some of the programs initiated by IDEP. 2. Initially, KESIP targeted complex reforms involving higher education financial autonomy, municipal budget reforms, enhancement of the EMIS, provision of school development grants to basic education schools, implementation of the Teacher Career System, improvement of student learning and teacher assessments, and licensing of higher education professions. 3. KESIP was restructured because of project implementation delays.40 These delays were due to insufficient institutional capacity and high turnover in the Kosovar Government. The restructuring of the project entailed: i) the cancellation of US$3.5million of credit proceeds, ii) the reduction of the scope of the project, and iii) an extension for project implementation (ending in June 2021). 4. The project restructuring allowed to refocus funding and efforts towards the educational interventions with a higher positive effect on education quality: i) SDGs used to finance teacher training and learning materials, ii) learning assessment, and iii) teacher assessment. These investments directly responded to the PDO, which targeted the improvement of quality, accountability, and efficiency of education. The benefits of education quality improvement 5. Access to quality education is essential to propel individual well-being, social welfare, and national economic growth. 6. The economic benefits of education accessed by individuals depend on their educational attainment and their learning outcomes (i.e., education quality) because these factors limit labor prospects and determine returns to education. Psacharopoulos and Patrinos (2004) found that low-quality primary education, which implies the underdevelopment of basic literacy skills, restrict development career opportunities and the future accumulation of human capital. On his part, Lazear (2003) suggests that an increase of one standard deviation in test performance, which relates to education quality, is associated with an increase of 12 percent in individual expected income. 7. Quality education for all has spillover effects that translate into social benefits. These social gains include a decrease in income inequality, poverty reduction, lower crime and violence rates, better health and nutrition, and a smoothly functioning of society as a whole.41 8. Adopting a pro-equity approach when investing in education is a cost-effective way to reduce social inequality. Education investment is more pro-poor when it targets primary and lower secondary education,42 as poor households had more children in these education levels. Grundlach, Navarro de Pablo and Weisert (2004) 39 World Bank (2015). 40 World Bank (2019). 41 Behrman and Stacye (1997); Klump and Miralles (2007). 42 Sakellariou and Patrinos (2004). Page 52 of 69 The World Bank Kosovo Education System Improvement Project (P149005) established that higher stock of education (involving educational attainment and quality of education) elevates the income of the poor, not only by means of its effect on the average income, but also by its influence on the distribution of income. For this reason, education interventions with a pro-equity (pro-poor) approach “emerges as a win-win policy that will reduce both income poverty and income inequality”.43 9. At the national level, Hanushek and Woessman (2007) have shown that the economic and social benefits of investing in education are more closely related to education quality (measured as learning performance on international benchmark tests) than they are to education spending (share of Gross Domestic Product (GDP)) and educational attainment (average years of schooling and enrollment rates). They have analyzed the data of 50 countries over 50 years and found that the countries with the higher long-run rates of economic growth are those where the students learn more. Their results suggest that an increase of one standard deviation in average performance on international benchmark test (e.g., PISA) can increase the growth of the annual long-run GDP by 1 or 2 percent. The effect of learning and teacher assessments on education quality 10. “There is a widespread recognition that evaluation and assessment frameworks are key to build stronger and fairer school systems”.44 Two major elements of the assessment framework within the education sector are: i) learning (student) assessment, and ii) teacher assessment. 11. Learning assessment is a powerful tool that allows to diagnose, plan, monitor and improve learning when it is implemented in an effective way.45 Namely, learning assessment has a positive impact on education quality when it collects data on whether and how students are learning and use this data to inform educational policy and teaching practices.46 12. Teacher assessment drives education quality improvement since teaching effectiveness has been identified as the educational feature that contributes the most to student learning.47 Teacher assessment foster education quality improvement in two ways: i) when the results are used by the teachers to reflect on their weaknesses and to adjust their teaching practice in consequence; and ii) when it is used at the system level to diagnose the strengths and opportunity areas of teachers to provide them with the training and professional development they need.48 13. The project contributed to consolidate the student and teacher assessment framework in Kosovo. Student assessment framework was strengthened through the alignment of Matura 12 examination with the curricular reform, the launch of the National Student Assessment Framework, the in-depth analysis of PISA 2018 results, and the delivery of several virtual trainings to build the capacity of the DSME; all of these aiming to continue guiding the education reforms and policy decisions made by Kosovo’s education authorities. Meanwhile, the teacher assessment framework was reinforced via the piloting and approval of the teacher performance assessment system, and the launching of the Teacher Professional Competencies and Development Framework. The effect of SDG on education quality 14. The implementation of SDGs leads to education quality improvement since it entails School Based Management (SBM), which empowers schools to make their own decisions.49 It has been ascertained that SBM have the potential to increase education access, to improve learning outcomes, and to reduce education inequality as the school 43 Klump and Miralles (2007), pp. 17. 44 OECD (2011). 45 UNESCO (2021). 46 GPE (2021). 47 Hanushek (2011); World Bank (2013). 48 OECD (2011). 49 Al-Samarri (2015). Page 53 of 69 The World Bank Kosovo Education System Improvement Project (P149005) community members are in a better position that educational authorities to identify the actual needs of their schools and to make effective decisions on aspects related to teaching and learning.50 15. The positive effect of any educational intervention on education quality depends on the educational input it is invested.51 In the case of school grants, they have a positive impact on education quality “when they are used with a specific intention, such as investing in the professional development of teachers or purchasing teaching and learning materials”.52 16. As mentioned before, teacher effectiveness has been identified as a key predictor of student learning,53 to such an extent that it can even offset learning deficits of disadvantaged students.54 In this regard, Carneiro et al. (2016) found that the schools with a greater positive impact of school grants on student learning were those that invested in teacher training. Additionally, Kremer, Brannen and Glennerster (2013) unveiled that the interventions that match teaching practices to student learning (as in the case of SDGs used for teaching training) are among the most cost- effective interventions to improve education quality. 17. Prior to project restructuring, KESIP targeted 160 basic education schools (i.e., primary and lower secondary levels) to provide them with SDGs that allow them to implement their School Development Plans (SDP). Their SDPs must include strategies to enhance the quality of teaching and learning activities (two of the crucial factors that affect education quality) in order to be candidates to receive the grant. Of this first batch of 160 beneficiary schools, 140 were selected based on a pro-equity approach (i.e., they were located in low socio-economic status communities and rural areas), while the remaining 20 were competitively selected. The project reallocation of resources resulting from the project restructuring allowed a second batch of SDG beneficiary schools, which encompassed 80 additional schools that were competitively chosen. Namely, a total of 240 basic education schools received grants, benefitting over 100,000 primary and lower secondary education students in Kosovo. Methodology 18. The ex-post economic analysis of KESIP is based on a Cost-Benefit Analysis (CBA). The purpose of the CBA is to determine if a project is globally beneficial or detrimental to society’s welfare by comparing the project’s costs to its expected benefits.55 It also allows the calculation of the Internal Rate of Return (IRR) of the project as it entails estimating all costs and benefits in monetary terms. 19. In the case KESIP, the costs comprise the total amount of the loan’s proceeds after the restructuring, the financial cost of this funding, and the recurrent administrative costs of distributing the SDGs. 20. The project’s benefits are measured as the future earnings gains due to the quality improvement induced by the project. The two main benefit sources of the KESIP are: i) the improvement of teacher practices financed by SDGs, and ii) the strengthening of the learning and teacher assessment frameworks. Notwithstanding, this analysis focuses exclusively on the benefits derived from the teacher practice enhancement supported by SDGs since the number of beneficiaries of this subcomponent is clear enough to estimate the future earnings increase promoted by the project. The expected economic benefits of the assessment framework enhancement are left aside because they are hard to quantify as they entailed system-wide reforms that will continue their implementation after project 50 Bruns, Filmer and Patrinos (2011). 51 Evans and Popova (2015). 52 Emeran (2021). 53 World Bank (2013); Evans and Popova (2015). 54 Hanushek et al. (2005); Hanushek and Rivkin (2010); Rockoff (2004). 55 Josselin and Le Maux (2017). Page 54 of 69 The World Bank Kosovo Education System Improvement Project (P149005) closure. Therefore, the project’s economic benefits computed through this analysis can be considered as a lower bound of the potential benefits resulting from KESIP’s implementation. 21. After all the cost and benefits have been identified, they are converted to the same currency (US$),56 brought to their present value using a discount rate of 5 percent,57 and exhibited in the form of three indicators: i) Benefit/Cost ratio (B/C), which measures the ratio of discounted benefits to discounted costs; ii) Net Present Value (NPV) of the total benefits of the project, which is the value of the discounted benefits once the discounted costs invested to produce them are subtracted; and iii) IRR of the project, which represents the rate of interest that equates the discounted present value of expected benefits and the present value of the costs associated to the project. 22. The analysis is based on the most reliable and most up-to-date data available on student enrollment, returns to education, and employment prospects. The data was retrieved from project documents and reports produced by the World Bank, Kosovo Agency of Statistics (KAS), World Bank Data Bank, and additional literature review. Beneficiaries 23. Quality of education is the main challenge faced by the educational system of Kosovo. The majority of children in Kosovo between the ages of 6 and 15 attend and complete basic education (i.e., primary and lower secondary).58 However, the learnings they gain are insufficient and influenced by their socio-economic level. For instance, PISA 2018 results showed that only about 20 percent of 15-year-old students in Kosovo reach a minimum level of proficiency (level 2 or higher) in all three subjects (reading, mathematics, and science), and that the students with a higher socio-economic level, outperformed disadvantaged students in reading by 40 score points.59 24. KESIP aimed to directly improve education quality by means of the provision of SDGs to support teacher practice improvement, targeting mostly disadvantaged and rural schools. The SDG were delivered to the schools between 2018 and 2021. The project originally estimated to benefit 160 schools (i.e., batch 1), from which 140 were selected using a pro-equity approach and the remaining 20 were competitively selected. Each school belonging to this first batch received a US$10,000 grant. Nevertheless, the project restructuring allowed to benefit 80 additional schools (i.e., batch 2), all of which were competitively selected and received a US$15,000 grant. Therefore, KESIP benefitted a total of 240 schools (around 25 percent of basic education public schools in Kosovo), from which the majority (58 percent) were in rural and disadvantaged regions (Table 1). Table 1. SDG beneficiary schools by classification and year Year School classification Total 2018 2019 2020 Disadvantaged 111 29 0 140 Competitively-selected 0 20 80 100 Source: KESIP data 25. Over 100,000 basic education students benefitted from the SDGs provided by KESIP. Around 46,000 beneficiary students were enrolled in disadvantage schools, while the remaining 54,000 beneficiaries were enrolled in 56 The exchange rate used in the analysis was 0.88 EUR$ per US$. It corresponds to the average annual exchange rate for year 2020 and was retrieved from the Global Economic Monitor database of the World Bank. 57 A 5 percent discount rate was chosen because it is deemed as conservative and falls within the average range outlined in the Cost-Benefit methodology. 58 According to UNICEF (2021), 98 percent and 96 percent of the children in Kosovo complete primary and lower secondary education, respectively. 59 OECD (2019). Page 55 of 69 The World Bank Kosovo Education System Improvement Project (P149005) competitively selected schools (Table 2). Less students at disadvantaged schools were benefitted because rural and remote schools tend to have lower enrollment than schools in favorable locations. However, the per capita investment attributed to SDGs was higher for disadvantaged students (US$30) than for the students in competitively selected schools (US$25). Table 2. SDG beneficiary students by classification and year Year School classification Total 2018 2019 2020 Disadvantaged 36,474 9,529 0 46,003 Competitively-selected 0 6,572 47,580 54,152 Source: Own estimations based on KESIP data. 26. As mentioned earlier, the most cost-effective way to improve education quality and reduce social inequality implies matching teaching practices to students learning60 and adopting a pro-equity approach when investing in education.61 This is exactly what KESIP has done by targeting mostly disadvantaged schools, investing more in disadvantaged students in per capita terms, and using SDG resources to train teachers according to school needs. A total of 6,896 basic education teachers benefitted from the training provided through SDGs. Of these, 3,355 teachers received training between 2018 and 2019, while the remaining 3,541 were trained between 2020 and 2021. Although it is expected that the teaching practice improvement derived from this training will affect several generations of students, this analysis estimates the project’s benefits considering only the number of students enrolled in beneficiary schools during the years they received the grant in order to provide conservative estimates. Benefits 27. Higher educational attainment and better education quality are fundamental to obtain higher wages, which not only increases the income of individuals, but also helps to reduce poverty and income inequality, alongside other positive spillover effects that benefit the society as a whole, including economic growth.62 28. The students obtaining better education quality due to better trained teachers supported through KESIP’s SDGs will collect higher earnings once inserted in the labor market and throughout all their working life. According to Kosovo’s labor law, the minimum age for entering employment is 15, while the retirement age with basic pension is 65. Participation and unemployment rates determine the number of beneficiary students that will be able to find a job in Kosovo’s labor market. The rates of participation (38.3 percent) and unemployment (25.9 percent) used in the model to estimate the project’s benefits correspond to year 2020 average.63 29. The first step to estimate the earnings increase due to education quality improvement promoted by the project entails computing the annual wage that the graduates of each educational level will obtain once inserted in the labor market. The average annual wage by educational attainment was estimated using the database of the Statistics on Income and Living Conditions (SILC) collected in Kosovo in 2018 (Table 3). 60 Kremer, Brannen and Glennerster (2013). 61 Grundlach, Navarro de Pablo and Weisert (2004). 62 Behrman and Stacey (1997); Klump and Miralles (2007). 63 Kosovo Agency for Statistics (2021). Page 56 of 69 The World Bank Kosovo Education System Improvement Project (P149005) Table 3. Annual wage (EUR$ and US$) by educational attainment Currency Educational attainment (EUR$) (US$) Primary 3,941.2 4,478.7 Lower secondary 4,205.9 4,779.4 Upper secondary 4,763.8 5,413.4 Post-secondary non-TE 6,394.0 7,265.9 Short cycle tertiary 6,688.5 7,600.6 Bachelor 11,995.6 13,631.3 Source: Own estimations based on SILC 2018 database. 30. The model runs three simulations (i.e., scenarios) to estimate the potential benefits arising from SDGS, which were used for teacher training and learning materials (Table 4). The wage increases assumed for each scenario are: i) 1 percent in the case of the most conservative scenario, ii) 2 percent for the intermediate scenario, and iii) 3 percent for the optimistic scenario.64 The estimated annual wages for each educational attainment and scenario are exhibited in Table 4. Table 4. Estimated annual wage (US$) by educational attainment and scenario Scenario Conservative Intermediate Optimistic Educational attainment (1%) (2%) (3%) (US$) (US$) (US$) Primary 4,523.5 4,568.2 4,613.0 Lower secondary 4,827.2 4,875.0 4,922.8 Upper secondary 5,467.5 5,521.7 5,575.8 Post-secondary non-TE 7,338.5 7,411.2 7,483.9 Short cycle tertiary 7,676.6 7,752.6 7,828.6 Bachelor 13,767.6 13,903.9 14,040.3 Source: Own estimations based on the model. Model estimations and economic benefits and costs Total benefits 31. The estimated economic benefits comprise the higher earnings resulting from the education quality improvement promoted by teacher practice enhancement financed by SDGs, exclusively. As explained before, the potential benefits arising from the assessment framework strengthen were left aside as this intervention entails a system-wide reform that will continue its implementation after the project’s closure and because its total number of beneficiaries is hard to estimate. Therefore, the economic benefits computed by this model represent a lower bound of the all the potential economic benefits arising from KESIP’s implementation. 32. Table 5 shows that the NPV of the estimated project benefits is US$15 million under the conservative scenario, US$31 million under the intermediate scenario, and that the NPV of the benefits increase up to US$47 million under the optimistic scenario. 64 This economic analysis assumes for each scenario a wage growth lower than what the literature review has shown in order to avoid overestimating project benefits. For instance, Ozawa et al. (2021) have done a meta-analysis of returns to cognitive ability in low- and middle-income countries that exhibits that a one-standard-deviation increase in cognitive test scores is associated with a 4.5 percent increase in wages. Also, Hanushek et al. (2015) identified that a one-standard-deviation increase in numeracy skills is associated with an 18 percent wage growth among prime-age workers. Page 57 of 69 The World Bank Kosovo Education System Improvement Project (P149005) Table 5. Higher earnings due to education quality improvement Scenario NPV (US$) Conservative 15,707,831 Intermediate 31,415,661 Optimistic 47,123,492 Source: Own estimations based on the model. Total costs 33. The main cost of KESIP’s implementation was the World Bank loan, which amounted to US$7.5 million after the project restructuring. The economic costs of the project also include: i) the financial costs of the loan, and ii) the estimated recurrent cost of managing and delivering SDGs to beneficiary schools. It was assumed that the Kosovar Government paid with its own resources for the recurrent administrative costs associated with SDGs and that those costs amounted to 1 percent of the annual expenditure that the Kosovar Government would have invested the beneficiary schools for inputs different to teacher salaries.65 34. Table 6 shows that the NPV of all the project costs is US$3.1 million. When using a 5 percent discount rate to bring to the present the World Bank loan disbursement, the NPV of the principal payment is US$2.8 million, while the NPV of the financial and the recurrent costs associated with the project add up to US$122,815 and US$164,066, respectively. Table 6. NPV of project costs (US$) by concept Concept NPV (US$) Recurrent cost 164,066.2 Financial cost 122,815.0 Principal payment 2,888,139.1 Total 3,175,020.3 Source: Own estimations based on the model. Cost-Benefit Analysis 35. The model yields a positive NPV of the whole project in all three simulations, even when only the benefits derived from SDGs were taken into account (Table 7). Under the conservative scenario, which assumed a wage increase of 1 percent due to education quality improvement, the estimated NPV of the project amounts to US$6 million, with an IRR of 8.9 percent and a B/C ratio of 1.7. In the case of the neutral scenario, where the assumed wage increase was 2 percent, the model yields a project’s NPV of US$22 million, with an IRR of 15.4 percent, and a B/C ratio of 3.4. Finally, when a 3 percent increase in wages was considered (i.e., optimistic scenario), the NPV of the project is US$37 million, with an IRR of percent 20.1, and a B/C ratio of 5.1. 65According to Government of Kosova (2018), Kosovo invested 2.4 percent of its GDP in primary and lower secondary education during 2016, from which 90.6 percent was devoted to teacher salaries. Using these figures, the estimated recurrent cost associated with SDG delivery was US$37,895 per year. Page 58 of 69 The World Bank Kosovo Education System Improvement Project (P149005) Table 7. Summary of benefit and cost indicators by scenario Scenario Indicator Conservative Intermediate Optimistic NPV of benefits (US$) 15,707,831 31,415,661 47,123,492 NPV of costs (US$) 3,175,020 3,175,020 3,175,020 NPV of loan disbursements (US$) 6,140,900 6,140,900 6,140,900 NPV of the project at 5% discount rate (US$) 6,391,910 22,099,741 37,807,571 IRR (%) 8.9 15.4 20.1 Benefit/cost ratio 1.7 3.4 5.1 Payback period (years) 21.0 14.0 11.0 Source: Own estimations based on the model. 36. It can be concluded that the project’s design and implementation was efficient, particularly after the restructuring because of three reasons: i) the project adopted a pro-equity approach, which has been identified as a cost-effective way to reduce social inequality,66 ii) it invested SDG resources in teacher effectiveness, namely the key predictor of student learning,67 and iii) this ex-post economic analysis showed that all the costs of KESIP’s can be economically justified by the benefits derived from the SDGs alone. 66 Grundlach, Navarro de Pablo and Weisert (2004); Klump and Miralles (2007). 67 Evans and Popova (2015); Hanushek (2011); World Bank (2013). Page 59 of 69 The World Bank Kosovo Education System Improvement Project (P149005) ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS The ICR was shared for comment with the Borrower’s Ministry of Education, Science, Technology, and Innovation (MESTI). The MESTI circulated the document to relevant officials including the Project Coordination Unit, and none had comments. Page 60 of 69 The World Bank Kosovo Education System Improvement Project (P149005) ANNEX 6. General structure of the Kosovo education system Figure adapted from Butler et al.2009 and update in 2015 Source: Kosovo Education Strategic Plan 2017-2021. Republic of Kosovo, Prishtina. July 2016 Adapted Page 61 of 69 The World Bank Kosovo Education System Improvement Project (P149005) ANNEX 7. SUMMARY OF THE REVISIONS TO COMPONENTS 1. Component 1 original design included three sub-components covering the national level through the MEST, the municipal level through MEDs, as well as schools and universities. The component was revised so that the planning, financial and monitoring goals of its two sub-components focused on the school level as described below. • Sub-component 1.1.: Strengthening Strategic Planning and Financial Management Capacities included pre-university and university level activities for budget formulas and financial autonomy. An initial diagnosis of the existing school financing formula was carried out in 2017 and 2018, but there was a delay in establishing an Analytical Group within the MEST and limited policy discussion to guide the financial reform. One consultancy related to strengthening financial autonomy and capacity of universities was hired as well and the assessment highlighted higher education financing system strengths and weaknesses and outlined some short- and long-term corrective actions. The work, however, lacked clear actions to support/strengthen management practices in public universities and there was no further follow up on the part of the MEST to define next steps and so, the work was not completed. Given limited overall progress, the entire sub-component and all three related IR indicators were dropped: (i) National and municipal-to-school per capita formulas officially revised and used for resource allocation (ii) Budgeting and financial management rules or bylaws for school financial autonomy approved and used (iii) Rules for full financial autonomy in universities developed and approved 2. The only activity that was supported by the project after restructuring was a TA aimed at reviewing the financing mechanisms of pre-university education and providing recommendations for revising the school financing formula and improving its implementation. • Sub-component 1.2.: Enhancing the existing EMIS. The initial analytical work and development of technical specifications to select a firm for the development of the Central Systems software & integration were initiated in 2017. However, given substantial procurement and selection delays and absence of the legal and institutional foundations guaranteeing the use of the EMIS software and hardware packages across the proposed beneficiaries, and limited implementation time left, the EMIS activities were dropped with its two intermediate indicators: (i) Education Institutions Updating Student Level-Data in EMIS Twice a Year (ii) EMIS Integrated and data exchange automated with four data bases on teacher licensing, student assessment, school map, and expenditures. 3. The preparatory work that was completed could, however, be used by MEST for possible future implementation with alternative financing sources. • Sub-component 1.3.: School Development Grants. The SDG sub-component remained unchanged, but the associated IR indicators and PDO target were revised as described earlier. While the associated PDO indicator “Beneficiary schools disseminating report on grant budget allocation and utilization (percentage)” remained unchanged the end-of-project target was revised downward (from 80 percent to 60 percent) and this change was formalized through first restructuring (April 2019). However, the ISR#7 (May 2019) that was filed following the restructuring established a more ambitious end target (90 percent) Page 62 of 69 The World Bank Kosovo Education System Improvement Project (P149005) for this PDO indicator. This end target was maintained till project closure, but not formalized through restructuring. Under the original Results Framework, the progress of this sub-component was to be measured by two IR indicators. The first restructuring added one new IR indicator: “School development plans for quality investments through school grants (number).”68 The second restructuring which extended the closing date by 12 months, allowed expansion of the SDGs to cover an additional 80 schools. 4. Component 2 was also revised so that strengthening of management capacity and accountability to enhance the education quality goals of its three sub-components focused on the school level as described below. • Sub-component 2.1.: Implementation and Improvements to the Teacher Career System. Given the lack of policy clarity, the scope of this sub-component was reduced from efforts to impact the overall Career System for Teachers to setting two core pillars leading to quality of teachers: teacher assessment and teacher professional development. Accordingly, the intermediate indicators were revised as follows: (i) “Percentage of registered teachers licensed at the Advanced Level (including the gender-differentiated indicator: “female teachers licensed”) was revised to reflect the creation of “a functional teacher assessment system”, and (ii) the “Teacher Career and Licensing System Finalized” indicator was revised to reflect the creation of a “functional teacher professional development framework”. • Sub-component 2.2.: Strengthening Capacities of Key Institutions for Monitoring Educational Outcomes and Assessment of Student Learning. The activities of this sub-component were revised to include the development of the National Learning Assessment Framework, the revision and piloting of the 12th grade Matura assessment, and support for implementation, analysis and reporting on international assessments (PISA and TIMSS). Consequently, the intermediate indicator target of “Grade 9 and 12 examinations revised in line with Kosovo Curriculum Framework and Evaluated After Pilot” was adjusted to “Grade 12 examination revised and piloted in line with Kosovo Curriculum Framework.” The two Bank core indicators related to learning assessment system were perceived as no longer achievable and were dropped as well: (i) System for Learning Assessment at the Primary Level”, and (ii) Rating Scales for Primary Level Learning Assessment. • Sub-component 2.3.: Improving accountability and quality assurance of higher education through the development of key professional licensing statues in priority economic fields. The feasibility study on regulated professions was initiated in 2017, but the work was delayed. Limited strategic policy discussion on this topic within the MEST led to no concrete plans identified for further KESIP support. All activities and the related intermediate indicator (bylaws on professional licensing developed and approved) were cancelled. 5. Component 3. There were no changes under this component. 68 While the Results Framework attached to the Restructuring Paper #1 (April 2019) states that this IR indicator was revised, it was, in fact, a new indicator that was added. Page 63 of 69 The World Bank Kosovo Education System Improvement Project (P149005) ANNEX 8. SUMMARY OF PROJECT RESTRUCTURINGS 1. The first restructuring was initiated following the mid-term evaluation for KESIP and subsequent MTR from November 12-16, 2018. The evaluation and MTR carefully assessed the project’s implementation status and constraints and proposed project simplification and a partial cancellation of US$3.5 million equivalent of the IDA credit to allow for smoother implementation while still reaching the intended PDO. The restructuring was finalized following two consecutive missions (February and March 2019) during which alternatives to simplify the project were reviewed and agreed on. The first restructuring was approved on April 25, 2019 and, ultimately, reduced the scope of the KESIP from a complex program focused on multiple policy areas to three core sub-systems: School Development Grants, Learning Assessments, and teacher assessment and professional development. Corresponding revisions were made to the Results Framework, components and costs, and disbursement estimates, while the PDO remained unchanged given that the proposed focus on three core sub-systems was in line with the original project objective of strengthening systems that contribute to improved education quality, accountability and efficiency. 2. An extension of the original closing date was planned to be considered per Government’s request, only if the project showed clear signs of progress by July 2019 against a set of agreed milestones. The World Bank mission of July 22-26, 2019 evaluated the progress towards achieving the agreed milestones and determined that significant progress had been made in the implementation and achievement of the agreed milestones. Accordingly, a 12-month extension of the original closing date (December 31, 2019) was granted in October 2019 and formalized through a second restructuring. 3. Following simplification of project design and extension of the closing date, the project demonstrated significant progress in implementation and was on the path to achieving KESIP’s development objectives by the revised closing date of December 31, 2020. However, challenges caused by COVID-19, including disruptions of established business practices and school closures, resulted in related difficulties in implementing and completing project activities, in particular the implementation of SDG because schools were closed. Consequently, a six-month extension of the closing date was requested by the GoK to satisfactorily complete the project and achieve its development objectives. The extension was granted and approved through the third and final restructuring on December 7, 2020. Page 64 of 69 The World Bank Kosovo Education System Improvement Project (P149005) ANNEX 9. ORIGINAL THEORY OF CHANGE (as interpreted from PAD, Table 1) Components Activities Outputs Outcomes / PDO Long term / High level Outcomes • Review of the national allocation formula for the education specific grant going to pre-university education Strengthened accountability • Review of local budgeting processes in pre-university education • National and municipal-to-school per capita funding formulas through transparent budget • Develop annual monitoring procedures for national & local budgeting allocation processes & resource allocation based on the revised PCF formula officially revised and used for resource allocation Improved and rationalized • Development of education spending indicators to assess & monitor allocation of resources Efficiency Component 1: education spending efficiency • Budgeting and financial management rules or bylaws for PDO 1: Percent of municipalities Improvements • Train MEDs and schools on the use of PCF formula allocating resources to schools Enhancing • Develop criteria and programs for school director licensing school financial autonomy approved & used according to revised formula strategic • Develop procedures for assessing performance of school directors • Conduct public consultations and develop school rationalization plans • Rules for full financial autonomy in planning, universities developed & approved financial • Develop criteria &programs for institutional financial autonomy Improved education data • Define accountability norms & standards through bylaws • Education institutions updating management, collection, management, and • Develop procedures for assessing performance of university leaders & student-level data in EMIS twice a utilization monitoring and management year decision-making • Establish rules and procedures for effective information management, use, • EMIS integrated & data exchange Improved education quality in the data analysis and dissemination automated with four databases on monitoring through student • Set up core central registries and detailed design of systems for integrated teacher licensing, student education assessments information management assessment, school map & sector expenditures • Develop and implement key central systems (student, teacher, schools Strengthened school-level registry; EMIS one-stop shop) • At least 150 schools received SDGs Accountability accountability and quality • Provide hardware, software and services for upgraded EMIS • In at least 150 schools parents Improvements through information • Pilot the introduction of information systems in schools and/or community members were dissemination on student • Train schools, MEDs, university managers, MEST staff on the use of all EMIS involved in grant performance and sources/uses systems planning/implementation of funds • Train & support schools to prepare and implement multiple-year SDPs • Increased percentage of school PDO 2: Percent of school report • Finance SDGs directors report improved planning cards distributed capacity and publicly displayed at schools • Develop conceptual framework for a teacher career system • Develop Competency-Based Teacher Licensing and Promotion Scheme Strengthened school-level • Develop an induction program for new teachers accountability mechanisms • Upgrade the teacher licensing database • 35 percent of registered teachers through information Component 2: • Develop procedures for accreditation, classification & QA of PD; and licensed at the advanced level dissemination on uses of funds assessment of different PD formats • Teacher career & licensing system for school level intervention Strengthening • Develop national framework for student assessment finalized & operational PDO 3: Percent of beneficiary management • Design & implement new logistical procedures & security measures for Grade • Grade 9 and 12 examinations schools disseminating reports on 9 & 12 examinations revised in line with KCF & capacity and • Establish an integrated ICT system for processing examinations and grant budget allocation and evaluated after piloting accountability utilization assessments • System for learning assessment to enhance • Train DSME staff and item writers in key subjects established at the primary level Quality Improved higher education Improvements quality of • Develop & validate new examinations • Bylaws on professional licensing quality monitoring through education • Develop, pilot, evaluate and prepare for implementation new Grade 9 &12 developed and approved piloting and assessment of a examinations I licensing mechanism • Develop, pilot & carry out a SB national assessment survey at Grade 5 Improved accountability of • Support participation in PISA higher education institutions for • Develop ‘school report cards’ based on examination outcomes learning outcomes • Develop key professional licensing statutes in priority economic fields PDO 4: Licensing exams piloted • Develop coherent nat’l strategy, including bylaws, on professional licensing Notes: PCF = per capita financing; SDPs = school development plans; SDGs = school development grants; QA = quality assurance; PD = professional development; SA = student assessment; SB = sample-based; KCF = Kosovo Curriculum Framework Page 65 of 69 The World Bank Kosovo Education System Improvement Project (P149005) ANNEX 10. UPDATED THEORY OF CHANGE (April 2019 Restructuring) Project Development Objective (PDO): to strengthen select systems that contribute to quality, accountability, and efficiency improvements in education in the Republic of Kosovo. Components Activities Outputs Long term / High Level Outcomes/PDO Outcomes Component 1: • Train school management committees • 150 school development plans prepared for investments through Strengthened school-level in SDPs development SDGs accountability mechanisms Enhancing • Disburse SDGs to finance SDPs through information strategic planning, • Review of the adequacy and equity of • 150 schools received SDGs Efficiency • In150 schools parents and/or dissemination on uses of funds & financial the existing national allocation formula community members were for school level intervention Accountability management, for the education specific grant going to monitoring and pre-university education involved in grant Improvements planning/implementation PDO 1: Beneficiary schools decision-making • Revise school financing formula disseminating report on grant in Education in the education • School financing formula revised budget allocation and utilization sector • Develop and launch a Teacher • Functional teacher professional Professional Development Framework development framework Component 2: • Develop a teacher performance designed Strengthened school-level assessment system • Functional teacher performance accountability and quality Strengthening • Develop, approve and launch the assessment system is in place through information management National Student Assessment • National Learning Assessment dissemination on student capacity and Framework Framework is in place performance and sources/uses accountability • Develop an action plan & methodology • Grade 12 examination revised & of funds to enhance for piloting Matura Grade 12 exam piloted in line with KCF quality of education • Train DSME staff and item writers • PISA-2015 and PISA-2018 carried PDO 2: School report cards Quality and • Revise and pilot Matura Grade 12 exam out and analyses conducted & piloted at school level Accountability • Design School Report Cards shared with stakeholders Improvements • Complete School Report Cards for • School report cards designed in Education schools, participating in the pilot, with • School report cards received by Improved education quality student assessment results and other schools monitoring through student data assessments Component 3: • School directors report improvements in planning PDO 3: Reforms to Carry out qualitative survey to study the capacity on a set of specific tasks standardized Grade 12 Implementation and effect of the SDGs on school-level related to school development assessment applied communication capacity and autonomy planning and implementation support of SDGs Notes: PCF = per capita financing; SDPs = school development plans; SDGs = school development grants; QA = quality assurance; PD = professional development; SA = student assessment; SB = sample-based; KCF = Kosovo Curriculum Framework Page 66 of 69 The World Bank Kosovo Education System Improvement Project (P149005) ANNEX 11. SUPPORTING DOCUMENTS & BIBLIOGRAPHY Key Project Documents • Project Information Document (PID), Appraisal Stage. Dated May 31, 2015. Report No.: PIDA27657 • Project Appraisal Document (PAD), Appraisal Stage. Dated August 19, 2015. Report No: PAD1015 • Financing Agreement for Credit 5726-XK (Closing Package) (English). Washington, D.C. : World Bank Group. • Restructuring Paper on a Proposed Project Restructuring of Kosovo Education System Improvement Project. Dated April 25, 2019. Report No: RES35753 • Restructuring Paper on a Proposed Project Restructuring of Kosovo Education System Improvement Project. Dated October 10, 2019. Report No: RES38785 • Restructuring Paper on a Proposed Project Restructuring of Kosovo Education System Improvement Project. Dated December 7, 2020. Report No: RES44443 • Project Operational Manual. Updated June 2017. Ministry of Education, Science and Technology of Kosovo. • School Development Grants Manual. Revised December 2020. • Aide Memories from World Bank Preparation, Appraisal, and Implementation Support missions: (October 2014 – May/April 2021) • Project Implementation Status & Results Reports (sequence 1-11, October 2015 – June 2021) • Borrower’s annual and quarterly reports • Kosovo Education System Improvement Project (KESIP) 2018 Mid-Term Evaluation Findings. November 12-16, 201. World Bank. • Audit Reports • Studies produced under the project: o EMIS study: Functional Specifications for the Education Information Management System. August 2016 o SDG Evaluation: Measuring the Effectiveness of ESIP School Development Grants in Enhancing School-Level Capacities: Endline Survey. December 2019 o Strategic Framework for Teacher Development in Kosovo. April 2017. Ministry of Education, Science and Technology of Kosovo. o Constraints to Girls’ Transition to Upper Secondary Education and Retention of Girls and Boys in Upper Secondary Education in Kosovo. A qualitative study final report. July 2015. o Modernizing school system in Kosovo through sustainable and coherent teacher development system. Policy analysis, teacher and school director views. June 2016. o School Map report/National Level 2016: Inventory of Educational Facilities in Kosovo. o Capital Investment Projections in Education Facilities. National and Municipal Level 2016. Key Country Documents • Country Partnership Strategy for the Republic of Kosovo for the period FY12-FY15. Dated May 1, 2012. Report No. 66877-XK • Country Partnership Framework for the Republic of Kosovo for the period FY17–FY21. Dated April 25, 2017. Report No. 112337-XK • Republic of Kosovo Systematic Country Diagnostic. Dated January 2017. Report No. 114618-XK • Country Snapshot-Kosovo. Prishtina: The World Bank Country Office in Kosovo, 2015. Page 67 of 69 The World Bank Kosovo Education System Improvement Project (P149005) • Kosovo National Development Strategy 2016-2021. Prishtina, January 2016 • Kosovo Education Strategic Plan 2011-2016 • Kosovo Education Strategic Plan 2017-2021 Bibliography World Bank. 2014. Kosovo Public Finance Review: Fiscal Policies for a Young Nation. Washington, DC. World Bank. https://openknowledge.worldbank.org/handle/10986/20756 License: CC BY 3.0 IGO. Al-Samarrai, S. (2015). Improving education through the Indonesian School Operational Assistance Program (BOS). World Bank Policy Brief 96507. Behrman, J. and Stacey, N. (1997). The Social Benefits of Education. The University of Michigan Press. Bruns, B., Filmer, D. and Patrinos, H. (2011). Making schools work: new evidence on accountability reforms. World Bank, Washington, D.C. Carneiro, P. et al. (2016). Schools Grants and Education Quality: Experimental Evidence from Senegal. World Bank Policy Research Working Paper 7624. Emeran, C. (2021). Do School Grants Improve the Quality of Teaching? Retrieved from: http://www.iiep.unesco.org/en/do-school-grants-improve-quality-teaching Evans, D. and Popova, A. (2015). What Really Works to Improve Learning in Developing Countries? An analysis of divergent findings in systematic reviews. World Bank Policy Research Paper 7203. GPE. (2021). How to Improve Education Quality? Make Learning Assessment System Better. Retrieved from: https://www.globalpartnership.org/blog/how-improve-education-quality-make-learning-assessment-systems- better Government of Kosova. (2018). Annual Statistical Report with Education Indicators 2016/17. Government of Kosova, Pristine. Grundlach, E., Navarro de Pablo, J. and Weisert, N. (2004). Education is Good for the Poor: a Note on Dollar and Kraay, in (eds) A. Shorrocks and R. Van der Hoeven, “Growth, Inequality, and Poverty”, Oxford University Press Inc., New York. Hanushek, E. et al. (2015). Returns to Skills around the World: Evidence from PIAAC. European Economic Review, Vol. 73, 103-130. Hanushek, E. (2011). The Economic Value of Higher Teacher Quality. Economics of Education, Vol. 30, 466-479. Hanushek, E. et al. (2005). The Market for Teacher Quality. NBER Working Paper 11154. Hanushek, E. and Rivkin, S. (2010). Generalizations about Using Value-Added Measures of Teacher Quality. American Economic Review, Vol. 100, 267-271. Hanushek, E. and Woessmann. (2007). The Role of Education Quality for Economic Growth. World Bank Policy Research Paper 4122. Page 68 of 69 The World Bank Kosovo Education System Improvement Project (P149005) Josselin, M. and Le Maux, B. (2017). Statistical Tools for Program Evaluation: Methods and applications to economic policy, public health, and education. Springer, Switzerland. Kosovo Agency for Statistics. (2021). ASKDATA. Retrieved from: https://askdata.rks- gov.net/PXWeb/pxweb/en/askdata/ Klump, R. and Miralles, C. (2007). How Can Education Contribute to Pro-Poor Growth? Policy implications for financial cooperation. KFW Entwicklungsbank Discusion Paper 49. Kremer, M., Brannen, C., and Glennerster, R. (2013). The Challenge of Education and Learning in the Developing World. Science, Vol. 340, 297-300. Lazear, E. (2003). Firm Specific Human Capital: a skill-weights approach. NBER Working Paper 9679. OECD. (2011). Evaluation and Assessment Frameworks for Improving School Outcomes: common policy challenges. OECD Press, Paris. OECD. (2019). Programme for International Student Assessment (PISA) – Results from PISA 2018: Country Note of Kosovo. OECD, Paris. Ozawa, S. et al. (2021). Educational and Economic Returns to Cognitive Ability in Low- and Middle-Income Countries: A systematic review. World Development 149. Psacharopoulos, G. and Patrinos, H. (2004). Returns to Investment in Education: a further update. Education Economics, Vol. 12, Issue 2, 111-134. Rockoff, J. (2004). The Impact of Individual Teachers on Student Achievement: Evidence from Panel Data. American Economic Review, Vol. 94, 247-252. Sakellariou, C. and Patrinos, H. (2004). Incidence Analysis of Public Support to the Private Education Sector in Côte d’Ivoire. World Bank Policy Research Working Paper 3231. UNESCO. (2021). Assessment for Improved Learning Outcomes. Retrieved from: https://en.unesco.org/themes/learning-assessments UNICEF. (2021). Statistical Snapshot: 2019-2020 MICS in Kosovo & 2019-2020 MICS with Roma, Ashkali and Egyptian Communities in Kosovo. UNICEF. World Bank. (2013). What Matters Most for Teacher Policies: A Framework Paper. System Approach for Better Education Results (SABER) Working Paper Series 4. World Bank. (2015). Project Appraisal Document: Education System Improvement Project (Report No. PAD1015). World Bank. (2019). Restructuring Paper on a Proposed Restructuring: Kosovo Education System Improvement Project (Report No. RES35753). World Bank (2019). World Bank Data Library – Database of Statistics on Income and Living Conditions (SILC) collected in Kosovo in 2018. World Bank. (2021). Global Economic Monitor – Exchange rate. Retrieved from: https://databank.worldbank.org/source/global-economic-monitor-(gem) Page 69 of 69