Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Education System Improvement Project (P149005) Report Number: ICRR0022925 1. Project Data Project ID Project Name P149005 Education System Improvement Project Country Practice Area(Lead) Kosovo Education L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-57260 31-Dec-2019 6,597,008.39 Bank Approval Date Closing Date (Actual) 10-Sep-2015 30-Jun-2021 IBRD/IDA (USD) Grants (USD) Original Commitment 11,000,000.00 0.00 Revised Commitment 7,515,200.00 0.00 Actual 6,597,008.39 0.00 Prepared by Reviewed by ICR Review Coordinator Group Denise A. Vaillancourt Judyth L. Twigg Eduardo Fernandez IEGHC (Unit 2) Maldonado 2. Project Objectives and Components DEVOBJ_TBL a. Objectives As stated in the November 20, 2015 Financing Agreement between the government and IDA (Schedule 1, p. 5), the project development objective (PDO) was "to strengthen selected systems that contribute to quality, accountability, and efficiency improvements in education in the Republic of Kosovo.” The design document presented the same statement (Project Appraisal Document/PAD, p. 4 and Annex 1). Page 1 of 27 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Education System Improvement Project (P149005) The PDO statement did not change throughout the project’s life. However, PDO-level indicators and targets were revised as a part of the project’s restructuring in April 2019, reflecting a substantial reduction in the overall scope and ambition of the project. Consequently, a split rating methodology is warranted. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 25-Apr-2019 c. Will a split evaluation be undertaken? Yes d. Components Original Components: Component 1: Enhancing strategic planning, financial management, monitoring and decision- making in the education sector. Original estimate: US$5.20 million; revised estimate at restructuring; US$3.13 million; actual cost: US$3.34 million (64 percent of original allocation/106 percent of revised allocation) 1.1: Strengthening strategic planning and financial management capacities to improve spending efficiency and accountability. Central-level activities for pre-university education included: (i) review of national allocation formula for education grants; (ii) review of local (municipal-to-school) budgeting processes; (iii) development of annual monitoring procedures for national and local budgeting and resource allocation processes, based on the revised per capita financing formula; and (iv) development of education spending indicators to assess and monitor spending efficiency. Municipal-level activities included training and on-the-job support for municipal education directorates and schools on the use of the formula and applications of guidelines. School-level activities included financial management capacity strengthening of school directors for expanded school financial autonomy. A few municipalities would also be given support to develop and implement school rationalization plans. At the university level, planned activities included: (i) development of criteria and programs for achieving full institutional financial autonomy; (ii) definition of norms and standards through bylaws under the new higher education law; and (ii) development of procedures for assessing performance of leaders and management. 1.2: Enhancing existing education management information system (EMIS) to facilitate the use of data analysis for sector monitoring and management. Planned activities included: (i) establishment of rules and procedures; (ii) enhancing data availability through critical registries; (iii) development of a platform to integrate existing information systems and databases; (iv) enhancing existing EMIS capabilities to expand indicators and statistical analysis; and (v) training and capacity building for schools, municipal education districts, universities, and central government on these systems’ use. Page 2 of 27 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Education System Improvement Project (P149005) 1.3: School development grants. Aimed at strengthening school capacity to plan and manage resources and promote local-level participation, planned activities included: (i) strengthening the school development planning system; (ii) training and other support to selected primary schools to prepare, implement, and evaluate multi-year school development plans (SDPs); and (iii) financing of school development grants (SDGs) for selected primary schools, ranging between $10,000 and $15,000. Component 2: Strengthening management capacity and accountability to enhance quality of education. Original estimate: US$5.00 million; revised estimate at restructuring US$3.72 million; actual cost: US$2.55 million (51 percent of original allocation/69 percent of revised allocation) 2.1: Teacher career system. Envisaged support to the Ministry of Education, Science and Technology (MEST) aimed to: (i) implement a career system for teachers’ professional development, performance assessment, and promotion; and (ii) provide policy support to strengthen, complement, and systematize the teacher career system and licensing scheme. Specific activities included: (i) development and implementation of a teacher performance assessment plan; (ii) development of a competency-based teacher licensing and promotion scheme, defining competencies for each license type and links to financial incentives; and (iii) establishment of an induction program and other supports for new teachers to improve and institutionalize performance assessment and professional development. 2.2: Strengthening capacities for monitoring educational outcomes and assessment of student learning. Aiming to revise and upgrade the current system of national student assessment and examination in grades 5, 9, and 12, planned activities included: (i) setting up a model national assessment program, aligned with international best practices; (ii) developing feedback systems on learning outcomes for decision-makers and schools; (iii) establishing an integrated information communication technology system for processing results; and (iv) participating in international student assessments and results analysis. Support also envisaged capacity strengthening of MEST’s Division for Standards, Monitoring and Evaluation (DSME) to design, administer, and process high-quality examinations and student assessment surveys. 2.3: Improving higher education accountability and quality assurance through the development of key professional licensing statutes in priority economic fields. Aimed at strengthening MEST capacity to carry out professional licensing and improved accountability mechanisms for learning outcomes, planned activities included: (i) development of a national strategy, including bylaws, on professional licensing; and (ii) piloting of licensing activities in one or two key professions to provide quality assurance to the labor market of graduates’ knowledge and skills. Component 3: Implementation and communications support. Original estimate: US$0.80 million; revised estimate at restructuring: US$0.66 million; actual cost: US$0.71 million (89 percent of original allocation; 108 percent of revised allocation). Project implementation support included the financing of training, services, and operating costs for the Project Coordination Unit (PCU) and communications support (workshops, materials, and outreach events in beneficiary communities). Revisions to Components Following its mid-term review, the project was restructured in April 2019 to scale back and simplify the original design, which was deemed too ambitious, given the country’s capacity. Page 3 of 27 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Education System Improvement Project (P149005) Under Component 1, Component 1.1 (planning, financial management, and decision-making capacity) was dropped due to limited policy and implementation progress, but the project did support a small technical assistance that recommended updating of the original per capita school financing formula, whose implementation was ultimately carried out under Component 1.3. Component 1.2 (enhancing the existing EMIS) was dropped. Component 1.3 (SDGs) was maintained and its cost increased by US$1 million (from US$2 million to approximately US$3 million). Under Component 2, two subcomponents were revised and simplified. Component 2.1 (improvement to teacher career system) limited its teacher career system to two areas: (i) a strategic framework on teacher development; and (ii) the teacher performance assessment pilot. In addition, school-based teacher training programs were supported through SDGs. The simplified scope of Component 2.2 (monitoring education outcomes and assessment of student learning) included: (i) the development and adoption of the National Student Assessment Framework; (ii) piloting of the reformed Grade 12 Matura exam; (iii) capacity building of MEST staff; and (iv) support to Program for International Student Assessment (PISA) results analysis and report writing. The project also supported the design and piloting of the school report card. Component 2.3 (quality assurance of higher education) was cancelled due to limited institutional commitment and capacity to implement higher education activities. Component 3 was maintained as originally designed. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Cost. The actual cost of the project at closing was US$6.60 million, 60 percent of the original estimate (US$11.0 million) and 88 percent of the revised estimate at the time of project restructuring (US$7.52 million). The actual cost of Component 1 was 64 percent of the original estimate; Component 2’s actual cost was half (51 percent) of the original estimate; and Component 3‘s actual cost was closest to its original estimate, at 89 percent. Financing and Borrower Contribution. The project was 100 percent financed by an IDA Credit, whose original approved amount was 7.9 million SDRs (US$11.0 million equivalent). A total of 2.5 million SDRs was cancelled at the time of restructuring, leaving an adjusted amount of 5.4 million SDRs (US$7.5 million equivalent). Actual disbursements amounted to 4.72 million SDRs, which was 60 percent of the original credit and 87 percent of the adjusted credit at restructuring. Actual disbursements were equivalent to US$6,597,008 (ICR, p. vi). No Borrower contribution was planned or provided. Key Dates. The project was approved on September 10, 2015 and became effective on May 13, 2016. On the basis of its mid-term review conducted in November 2018, the project was restructured on April 25, 2019, including the cancellation and simplification of subcomponents, changes to the results framework, a cancellation of part of the IDA Credit (2.5 million SDRs), and the reallocation of funds across disbursement categories. The project closing date was extended twice. An initial (12-month) extension was approved on October 10, 2019 and a second (six-month) extension was approved on December 7, 2020, both to allow the time needed to complete project activities, as restructured. The actual closing date was June 30, 2021, 18 months after the original closing date of December 31, 2019. Page 4 of 27 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Education System Improvement Project (P149005) 3. Relevance of Objectives Rationale The PDO (and its associated targets after restructuring) was substantially relevant to country conditions. While some gains have been made in recent years, formidable challenges to improved educational quality, accountability, and efficiency persist. There is still a mismatch between the quality and relevance of education services, on the one hand, and the needs of Kosovo’s labor market for skilled manpower, on the other. At the pre-university level, there have been some improvements in assessing learning outcomes and in using assessment results to improve quality, but much more remains to be done, including improvements to the teacher career system and the development and application of quality standards more responsive to labor market needs. At the university level, institutional commitment and capacity to improve higher education quality and relevance fall short of needs. In particular, the development of licensing standards and exams in key professions and sectors and the development of systems for tracking learning outcomes need to move forward. Notwithstanding some improvements in recent years, sector-wide systems and capacities, essential for improving accountability and efficiency, are in need of further strengthening. These include systems and capacities for strategic planning, budget allocation, and financial management, and integrated information systems for improved monitoring, evaluation, and evidence-based management and decision-making. All of these above-cited challenges would indicate a high relevance of the PDO. However, a moderate shortcoming in the relevance of the PDO was the missed opportunity to include equity among the project’s longer-term, higher-order objectives (improvements to quality, accountability, efficiency). This moderate shortcoming is identified because the issue of grave inequities is prominent in government and Bank strategic documents, both cross-sectoral and sector-specific. There was ample opportunity to make important headway in better documenting and addressing equity issues as a part of the systems- strengthening, capacity-building approach to improved sector performance. IEG’s follow-up discussion with the TTL (June 1, 2022) clarified that some elements of equity were indeed factored into systems strengthening activities (see sections on Efficacy and Lessons), but the non-inclusion of equity among the other higher-level objectives of quality, accountability, and efficiency resulted in less systematic measurement and reporting on the links between systems strengthening activities and their contributions to equity. The PDO was substantially relevant to the development priorities of the country. Kosovo’s National Development Strategy (NDS) 2016-21 (a new NDS is currently under preparation, according to the TTL) supports four thematic pillars: (1) human capital development; (2) good governance and rule of law; (3) development of competitive industries; and (4) development of infrastructure. The human capital development pillar includes interventions aimed at improving the efficiency of public spending in the education sector, increasing mechanisms and practices to strengthen sector accountabilities, strengthening the assessment of student performance and learning outcomes, and improving the quality of teaching. The PDO addressed these sector-specific elements of the human development pillar of the NDS and was also aligned with the Kosovo Education Strategic Plan (2017-2021), whose strategic objectives include improved management of the education system, enhanced quality assurance, improved teaching and learning, and strengthened quality and competitiveness of higher education. The first of its seven strategic objectives, improved participation and inclusiveness, is not captured in the PDO, even though the systems and activities targeted under the project are important vehicles for contributing to improved equity. The Page 5 of 27 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Education System Improvement Project (P149005) PDO's focus on accountability and efficiency was also highly relevant to the government's efforts to improve governance, captured in the second pillar of the NDS. The PDO is highly relevant to the Bank’s Country Partnership Framework (CPF) (FY2017- 2021) (a new CPF is currently under preparation, according to the TTL). The CPF’s support to Kosovo’s NDS is organized around three focus areas: (1) enhancing conditions for accelerated private sector growth and employment; (2) strengthening public service delivery and macro-fiscal management; and (3) promoting reliable energy and stewardship of the environment. The PDO's main thrust is highly relevant to the second focus area, particularly its education-specific objective of “strengthening selected systems to improve quality, accountability, and efficiency in the delivery of education.” The PDO's focus on education quality is also substantially relevant to the CPF’s first focus area, particularly its employment-specific objective of “enhancing employment opportunities and inclusion for youth and women.” Additionally, the PDO is relevant to the CPF's focus on the cross-cutting issue of citizen engagement, especially its focus on strengthening local and school-level management, efficiency, and accountability with and through the participation of teachers, parents, and communities in the planning, implementation, and monitoring of school activities. While the CPF includes improved equity as a health sector objective, it is not articulated as an education sector objective (CPF, p. 95). Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Strengthen select systems that contribute to quality improvements in education in Kosovo Rationale As originally designed, the project aimed to strengthen systems and capacities in eight critical areas culminating in eight outcomes that would contribute to higher-order, longer-term goals of improved sector quality, accountability, and efficiency. While many of these expected outcomes were poised to contribute to more than one of the three high-order goals, for ease of presentation they are clustered around the goal that they would most affect. The contribution to improved education quality was to be effected primarily through the achievement of two systems strengthening outcomes:  Outcome #4: Improved monitoring of pre-university education quality through student assessments, including MEST capacity building to strengthen the quality, quantity, and use of student performance and learning assessments; and Page 6 of 27 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Education System Improvement Project (P149005)  Outcome #7: Improved monitoring of higher education quality through the piloting and assessment of a licensing mechanism in priority economic fields. In addition, Outcomes #3 (improved education data collection, management, and utilization) and #5 (school- based information dissemination on student performance and sources/uses of funds), while categorized as primary contributors to accountability (see immediately below), were also expected to contribute to quality improvements. Outputs and intermediate results for Outcome #4: Improved monitoring of pre-university education quality through student assessments: With the support of a special twinning firm to help build capacity:  The Kosovo National Framework for Student Assessment was developed, approved, published in three languages (Albanian, Serbian, and English), and disseminated.  The original intermediate result to revise Grades 9 and 12 examinations in line with the Kosovo Curriculum Framework (KCF) and to evaluate them after piloting was partially achieved. The Grade 12 Matura examination was revised and piloted, and its results evaluated against KCF. The revision, piloting and evaluation of Grade 9 examination were not undertaken, because of changes introduced during restructuring.  Four large-scale student learning assessments were undertaken, fully achieving the target of four. o The report on PISA 2015 national results, containing in-depth analyses with policy recommendations, was completed in 2017 and published and disseminated to all relevant stakeholders. o The report on PISA 2018 national results was prepared in 2019, followed by the preparation in 2020 of an in-depth report with policy recommendations. These results and in-depth report were also disseminated to all relevant stakeholders. During a June 1, 2022 exchange with IEG, the TTL noted that the report assessed learning equity across the country, among other issues. o The project also supported the analysis, dissemination and use of results of the international assessment studies on math (TIMSS) and reading (PIRLS). Outputs and intermediate results for Outcome #7: Improved monitoring of higher education quality through the piloting and assessment of a licensing mechanism in priority economic fields:  No action plans were prepared and no policy discussions were held. This was due in large part to limited institutional commitment and capacity, resulting in the dropping of subcomponent 3 (quality assurance in higher education) and associated targets and indicators at restructuring.  The intermediate result indicator specifying the development and approval of bylaws on professional licensing was not achieved. Page 7 of 27 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Education System Improvement Project (P149005) Outcomes:  Outcome #4 was substantially achieved. The project succeeded in strengthening the capacity of MEST, enabling the DSME to design, administer, and analyze high-quality examinations and large- scale student assessments. The DSME is also now capable of developing testing instruments that are comparable across years, as demonstrated by the revision to the Matura Grade 12 exam, which piloted new subject items and is now aligned with competency-based curricula. These outcomes contribute substantially to the longer-term, higher-level objective of improved quality of pre-university education.  Outcome #7 was not achieved. Licensing exams were not piloted, as originally planned, following the decision to cancel university-level activities. The project, therefore did not contribute to the long-term, high-level objective of improving higher education quality. The project’s overall contribution to education quality improvements against original targets and expected outcomes is rated modest. Outcome #7 (higher education quality monitoring) was negligibly achieved; and Outcome #4 (strengthened student assessments) was substantially achieved. Two other contributing outcomes (Outcome #3, improved data collection, analysis, and use; and Outcome #5, school- based dissemination of information on student performance) were negligibly achieved, and a third contributing Outcome (#6, schools’ use of funds, detailed under accountability) was substantially achieved, as detailed below under Objective 2. The ICR does not report on any effects of SDG investments in the quality of teaching and learning. Rating Modest OBJECTIVE 1 REVISION 1 Revised Objective Strengthen select systems that contribute to quality improvements in education in Kosovo Revised Rationale The restructured design maintained the original PDO statement "to strengthen select systems that contribute to quality, accountability, and efficiency improvements in education in the Republic of Kosovo,” but its scaled- back scope and ambition focused on the achievement of only four of the eight systems and capacity-related Outcomes included in the original design. Under the simplified theory of change, the project’s critical path to improved education quality:  retained its focus on Outcome #4 (improved monitoring of pre-university education quality through student assessments), and  dropped its support to Outcome #7 (improved monitoring of higher education quality). Page 8 of 27 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Education System Improvement Project (P149005) Outputs and intermediate results for Outcome #4: With the support of a special twinning firm to help build capacity, outcomes and intermediate results are the same as those reported under the original Objective 1, with one caveat. The revised, less ambitious intermediate outcome target (the revision and piloting of the Grade 12 Matura examination and the evaluation of its results against the KCF) was fully achieved. Outcomes:  Outcome #4 was substantially achieved, as detailed under the original Objective 1. Over and above the substantial achievement of Outcome #4, two other contributing outcomes (#5, school- based dissemination of information on student performance, and #6, schools’ use of funds) were substantially achieved under the restructured project, as detailed below under Objective 2. The ICR does not report on any effects of SDG investments in the quality of teaching and learning. Revised Rating Substantial OBJECTIVE 2 Objective Strengthen select systems that contribute to accountability improvements in education in Kosovo Rationale The contribution to improved sector accountability was to be effected through the achievement of four systems strengthening outcomes:  Outcome #3: Improved education data collection, management, and utilization, including the strengthening of the EMIS and the integration of various education information systems;  Outcome #5: School-based information dissemination on student performance and sources/uses of funds;  Outcome #6: Strengthened school-level accountability mechanisms through information dissemination on the uses of funds for school level intervention; and  Outcome #8: Improved accountability of higher education institutions for learning outcomes. In addition, Outcomes #1 (transparent budget allocation) and #4 (improved monitoring of pre-university education quality through student assessments), while categorized, respectively, as primary contributors to efficiency and quality, were also expected to contribute to improved sector accountability. Page 9 of 27 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Education System Improvement Project (P149005) Outputs and intermediate outcomes for pre-university education accountability, supported by Outcomes #3, #5, and #6: Outcome #3: Improved education data collection, management and utilization  With support from Bank specialists, analytical work, guidelines for legal and institutional structures within MEST, and development of technical specifications for the core central systems supporting integrated information management were completed in 2017.  However, activities to improve education data collection, management, and utilization suffered due to failure to complete bylaws, as well as delays and lack of transparency of procurement bid evaluations. Critical, time-sensitive milestones agreed with the government (adoption of minimal legislation and completion of procurement processes for the work) were not reached, leading to the dropping of subcomponent 1.2 (enhancing EMIS) at restructuring.  After restructuring, MEST expressed interest in a scaled-back version of the EMIS upgrade, including a national student registry with links to the EMIS, and a sub-registry for Grade 12 Matura exam entrants. However, a one-time shortfall in the government's budget for the project (2021) resulted in a lack of funds to carry this out.  Two intermediate results indicators, dropped at restructuring, were not achieved: o 80 percent of schools and providers updating student-level data twice a year, up from a baseline of no student-level data in the EMIS; and o the integration of EMIS, and the automation of data exchange across four databases: teacher licensing, student assessment, school map, and expenditures. Outcome #5: Information dissemination on student performance and sources/uses of funds  With project support and under the guidance of the pre-university Director of MEST, the school report card (SRC) concept and design were completed by the SRC working group in 2020, incorporating learning assessment results, school statistics, and municipalities’ averages for comparison purposes.  Even after the restructuring dropped EMIS upgrading, MEST manually generated SRCs from the existing EMIS and piloted them during the month of March 2021 in the 160 schools that received SDGs in 2018-19. The pilot tested the template design, administrative instructions, and a user manual with indicator definitions.  Following the pilot, the Learning Assessment Unit worked on automation of the data to populate the SRCs using software. In a follow-up exchange with IEG on June 1, 2022, the TTL noted that, for the years ahead, MEST is committed to using the software program for automated production of the SRCs for all schools.  Schools’ feedback on the pilot noted its facilitation of greater collaboration within the community and among schools. They also noted that SRCs, coupled with school development plans, inform and inspire school improvement goals and schools' efforts to achieve them, with improved focus on learning outcomes and other school statistics, and enabling comparisons to municipal averages. Outcome #6: Information dissemination on uses of project funds for school-level intervention  240 schools received SDGs, exceeding the original target of at least 150. A first round of grants to 160 schools (in 26 municipalities) already exceeded the target. Of these, 140 were disadvantaged Page 10 of 27 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Education System Improvement Project (P149005) schools (selected on the basis of low socio-economic characteristics and low student performance), and 20 were competitively selected against pre-defined criteria. A second round of SDGs was provided to 80 additional schools (in 21 municipalities) on a competitive basis. Despite the timing of the second round (February 2020 during the onset of COVID-19), the PCU was effective in managing the process and delivering training and other support to enable the preparation of school development plans and use SDGs to finance improvements in teaching and learning quality.  School management committees, including parents and community members, were created in all 240 beneficiary schools, leading the preparation and implementation of school development plans and supporting SDG execution, an indication of citizens’ engagement.  By the project’s end, all 240 schools had developed plans for quality investments supported by SDGs, exceeding the target of 150 schools.  An endline survey revealed that 77 percent of school directors believed that SDGs effectively improved their planning capacity.  A total of 6,896 basic education teachers received training/professional development financed by SDGs supporting school development plans.  A total of 100,155 students benefited from direct interventions to enhance learning in beneficiary schools, of whom half (50,893) were female. Outcomes on pre-university education accountability:  Outcome #3 (information systems) was not achieved, given that most planned activities and indicators were dropped.  Outcome #5 (information dissemination on student performance and sources of funds) was not achieved. By the project’s end, SRCs were piloted for one month in 160 primary schools, or 20 percent of all primary schools in Kosovo. This fell short of the original target of 90 percent of all primary schools in Kosovo receiving and publicly displaying SRCs, produced by the updated EMIS, on a permanent basis.  Outcome #6 (information dissemination on uses of funds for school-level interventions) was fully achieved, in terms of improved accountability, but SDG investments in teaching and learning quality were not assessed for their effects on quality and equity.  Virtually all (99.6 percent) of beneficiary schools disseminated reports on grant budget allocation and utilization, exceeding the original target of 80 percent.  An independent qualitative survey conducted in a sample of 140 beneficiary schools revealed respondents’ assessment of the project as highly effective in improving the functionality of school governance mechanisms. The share of respondents assessing the project as most effective in improving the performance of various governance mechanisms were: 85 percent for teacher councils; 77 percent for school governing boards; 64 percent for student councils; and 38 percent for parent councils. The ICR did not discuss why the results were comparatively low for parent councils.  The capacity of publicly financed schools was substantially strengthened to plan and manage resources, and to promote citizens’ engagement through the participation of teachers, parents, and communities in planning, implementing and monitoring of school activities. Page 11 of 27 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Education System Improvement Project (P149005) Outputs and intermediate results for Outcome #8: Improved accountability of higher education institutions for learning outcomes:  Because there was little progress under the project in the preparation of higher-education action plans or in stimulating policy discussions, subcomponent 2.3 was dropped at restructuring. Consequently, the project did not support activities to establish a mechanism for holding higher education institutions accountable for learning outcomes. Outcomes on accountability of higher education institutions:  The project did not achieve the outcome of improved accountability of higher education institutions for learning outcomes. Overall, the project’s contribution to improvements in sector accountability and governance is rated modest. For pre-university education: Outcome #3 (improved education data collection, management and utilization) was negligibly achieved; Outcome #5 (school-based dissemination of information on student performance) was negligibly achieved; and Outcome #6 (strengthened school-level accountability mechanisms through information dissemination on the uses of funds for school-level intervention) was substantially achieved. Among other contributing outcomes, Outcome #4 (improved quality monitoring through enhanced student assessment) was substantially achieved and Outcome #1 (transparent budget allocation) was negligibly achieved, as detailed, respectively, under Objectives 1 and 3. For higher education: Outcome #8 (improved accountability of higher education institutions for learning outcomes) was negligibly achieved. Rating Modest OBJECTIVE 2 REVISION 1 Revised Objective Strengthen select systems that contribute to accountability improvements in education in Kosovo Revised Rationale The restructured project's support to improved sector accountability:  retained its focus on Outcomes #5 (school-based information dissemination on student performance and sources/uses of funds) and #6 (strengthened school-level accountability mechanisms through information dissemination on the uses of funds for school level intervention); and  dropped its support to Outcomes #3 (improved information systems and their use) and #8 (improved accountability of higher education institutions for learning outcomes). Outputs and intermediate results for Outcomes #5 and #6, along with the technical assistance on information systems, are the same as those reported under the original Objective 2, and so are not repeated here. Page 12 of 27 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Education System Improvement Project (P149005) Outcomes:  Outcome #5 (information dissemination on student performance and sources of funds) was achieved. By the project’s end, SRCs were piloted for one month in 160 primary schools, or 20 percent of all primary schools in Kosovo, fully achieving the revised target of 20 percent. (This is in contrast with the failure to achieve the much more ambitious, original target of 90 percent of all primary schools in Kosovo receiving and publicly displaying SRCs on a permanent basis.)  Outcome #6 (information dissemination on uses of funds for school-level interventions) was fully achieved, as detailed under the original Objective 2. Overall, the project’s contribution to improvements in sector accountability and governance is rated substantial, with main contributing Outcomes #5 and 6 both substantially achieved. In addition, preliminary steps to strengthen information systems were undertaken, even though Outcome #3 activities and measures were dropped. Still another contributing outcome (Outcome #4: Improved quality monitoring through enhanced student assessment), reported under Objective 1, was substantially achieved. Revised Rating Substantial OBJECTIVE 3 Objective Strengthen select systems that contribute to efficiency improvements in education in Kosovo Rationale The project's contribution to improved sector efficiency was originally to be effected through the achievement of two systems strengthening outcomes:  Outcome #1: Transparent budget allocation, through which municipalities would allocate resources to schools using a revised per capita formula; and  Outcome #2: Improved and rationalized allocation of resources. In addition, Outcome #6 (strengthened school-level accountability mechanisms through information dissemination on the uses of funds for school level intervention), while categorized as a primary contributor to enhanced accountability, was also expected to contribute to improved sector efficiency. Outcomes #1 (transparent budget allocation) and #2 (improved and rationalized allocation of resources) are assessed together, since they were supported under the same subcomponent 1.1 and tightly linked. Outputs and intermediate results: Page 13 of 27 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Education System Improvement Project (P149005)  A project-supported consultancy produced an initial diagnosis of pre-university education financing in 2018, but the dialogue on financing reform did not advance sufficiently to support improvements in budget monitoring and resource allocation based on per capita financing and spending efficiency.  A consultancy to strengthen the financial autonomy and capacity of universities was procured, but not completed, due to lack of ownership. The intermediate results target, that rules for full financial autonomy in universities would be developed and approved, was not achieved.  Although subcomponent 1.1 (strengthening strategic planning and financial management) was dropped during restructuring, technical assistance (TA) to support reforms of the per capita school financing formula remained an eligible activity. This TA ultimately supported: a review of the existing school financing formula; consultations at municipal and school levels; and policy options and recommendations for greater transparency and equity in resource allocation. Consultations on the findings, with the help of six simulations, built consensus and ownership around the proposed formula reforms. While the associated intermediate results target (revised formulas being in use by the project’s end) was not fully achieved, the TA did support movement toward this goal. In a follow-up discussion with IEG on June 1, 2022, the TTL noted that the various simulations, and the formula ultimately preferred by the government but not yet adopted or implemented, provides for extra funds for disabled students and rural schools.  Another intermediate target (also dropped at restructuring), the approval and use of budgeting and financial management rules or bylaws for school autonomy, was not achieved. Outcomes:  Both Outcomes #1 and #2 were to be measured by the same indicator: share of municipalities allocating resources to schools according to a revised allocation formula. The end-of-project target of 80 percent was not achieved.  Outcome #1 (strengthened accountability through transparent budget allocation) was not achieved, since associated activities were dropped and budget allocation formulas are not yet adopted or in use.  Outcome #2 (improved and rationalized allocation of resources) was partially achieved, given some progress made on revising the per capita financing formula and modest movement towards the improved and rationalized allocation of resources, considered a small contribution towards the ultimate result of improved efficiency. The project’s contribution to improvements in sector efficiency against original targets and expected outcomes is rated modest. Outcome #1 (transparent budget allocation) was negligibly achieved; and Outcome #2 (improved, rationalized resource allocation) was modestly achieved. Rating Modest Page 14 of 27 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Education System Improvement Project (P149005) OBJECTIVE 3 REVISION 1 Revised Objective Strengthen select systems that contribute to efficiency improvements in education in Kosovo Revised Rationale As redesigned, the project's support for improved sector efficiency was limited to some scaled-back support under Outcome #2 (improved and rationalized allocation of resources), with Outcome #1 (transparent budget allocation) dropped. Outputs and intermediate results for Outcome #2 are the same as those reported under original Objective 3, and so are presented in abridged form here:  TA to support reforms of the per capita school financing formula supported: a review of the existing school financing formula; consultations at municipal and school levels; and policy options and recommendations for greater transparency and equity in resource allocation. Consultations on the findings built consensus and ownership around proposed formula reforms. Outcomes:  Outcome #2 (improved and rationalized allocation of resources) was partially achieved, given some progress made on revising the per capita financing formula and modest movement towards the improved and rationalized allocation of resources, considered a small contribution toward the higher- level objective of improved efficiency. Outcome #2 was modestly achieved, culminating in a small contribution to the higher-order objective of improvements in sector efficiency. Revised Rating Modest OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale Overall efficacy under the original PDOs (and associated targets, indicators, and expected outcomes) was modest, given that each of the three objectives (systems strengthening contributing, respectively, to improved education quality, improved sector accountability, and improved sector efficiency) was modestly achieved. Page 15 of 27 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Education System Improvement Project (P149005) Attribution. In a follow-up exchange with IEG on June 1, 2022, the TTL noted that outcomes were strongly attributed to the project. Donors in Kosovo support different elements of the education sector. The Bank is the major donor supporting basic education. The European Union and German Technical Cooperation (GTZ) focus their support on higher education, while UNICEF is largely focused on supporting early childhood education. Counterfactual. During this same exchange, the TTL provided a compelling analysis of the counterfactual, as follows. In the absence of the project, there would have been no SDGs, and over 100,000 students and 6,800 teachers would not have benefited from investments in teaching and learning quality, in a fiscal environment where 90 percent of the budget is for the wage bill. The flexibility in the provision of goods provided a critical source of financing for distance learning needs during COVID and sanitary supplies for the return to the classroom, which may not have been provided in the absence of the project. Had there been no project, school-level management and implementation capacities would not have been strengthened, nor would MEST’s capacities for learning assessments and their use have been developed, and various assessments produced and disseminated. Moreover, without the project, the school financing formula would not have been assessed, nor would recommendations for improved implementation and options for enhanced efficiency and equity been explored. Overall Efficacy Rating Primary Reason Modest Low achievement OBJR1_TBL OVERALL EFFICACY REVISION 1 Overall Efficacy Revision 1 Rationale Overall efficacy under the revised PDOs (and associated targets, indicators, and expected outcomes) was substantial, given that the first two objectives (systems strengthening contributing to improved education quality, and systems strengthening contributing to improved sector accountability) were substantially achieved, and the third objective (systems strengthening contributing to improved sector efficiency) was modestly achieved. The above assessment of the project’s attribution and counterfactual (presented under “Overall Efficacy”) applies here as well. Overall Efficacy Revision 1 Rating Substantial 5. Efficiency Because the original project targeted system reforms, the economic analysis undertaken at appraisal was a qualitative assessment, grounded in international evidence, rather than a quantitative one. The end-of-project analysis, on the other hand, was able to compute the project’s net present value (NPV) and internal rate of return (IRR). That analysis concentrated on the restructured project’s enhanced focus on school-level quality improvements financed by SDGs, and the scaled-up coverage to 240 schools (vs. the original plan of Page 16 of 27 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Education System Improvement Project (P149005) 160). SDG-financed school development plans focused on teacher training and learning materials, reflecting international evidence about teaching effectiveness being a predictor of student learning. Of the 240 schools ultimately supported, 100 were selected on a competitive basis, and 140 disadvantaged, rural schools were selected with a view to improve equity. The ex-post economic analysis estimated the economic benefits stemming from the school grants (wage increase due to better education quality resulting from teacher practice improvement) to compute the NPV and IRR of the project. System-wide reforms were not explored, given difficulties in quantifying direct beneficiaries. The estimated NPV ranges between $6 and $37 million, and the IRR fluctuates between 9 and 20 percent, based on an estimated wage increase from 1 to 3 percent resulting from school quality improvements achieved with SDG support. Given the exclusive focus of this analysis on benefits of SDGs, the estimated NPV and IRR values represent the lower bound of benefits accruing from the project. Although the project’s original design was aligned with needs across various systems in education, it was too ambitious given limited sector capacity. The April 2019 restructuring considerably reduced the scope, strengthening its focus on a smaller number of critical systems and reforms, and enhancing quality through a strengthened focus on school-based support. Challenges to operational efficiency, concentrated mostly in the three-year period prior to restructuring, resulted in low disbursements during the project’s initial years. These challenges included: high turnover of MEST and PCU staff; political instability with high turnover in MEST leadership and upper management (four changes of Ministers and three General Secretaries); delays in procurement; delays in SDG disbursements and payments; and a one-time understated project budget for 2021, which limited project access to the full credit amount and led to the cancellation of two supplemental activities (development of a student registry and update of the EMIS, and procurement of tablets for disadvantaged students to facilitate distance learning during COVID-19). A well-conceived restructuring, stronger MEST leadership, strengthened PCU capacity, the Bank team’s close supervision, and two extensions adding 1.5 years to the project’s life all combined to accelerate implementation and disbursements, culminating in achievement of the project’s modified (scaled-back) objectives. Eighty-six percent of the project’s total financing was disbursed during the final two years of implementation, and all planned activities under the restructured project were completed by the project’s end. Overall, the project’s substantial efficiency rating is based on: its pro-equity approach, which is a cost-effective way to reduce social inequality; the focus of grant resources on teacher effectiveness, which is a key predictor of student learning; the economic analysis showing economic justification of benefits derived from the grants alone; and success in overcoming initial operational inefficiencies, as evidenced by the sound redesign of the project, and the completion of all activities and achievement of all objectives under the restructured design. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) Page 17 of 27 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Education System Improvement Project (P149005) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Under the project’s original outcome targets, the relevance of the PDO is rated substantial, efficacy in achieving each of the three objectives (systems outcomes contributing, respectively, to improved education quality, accountability, and efficiency) is rated modest, as measured against the original outcome targets, and efficiency is rated substantial. These ratings culminate in an Outcome rating of Moderately Unsatisfactory. Under the project’s revised outcome targets, the relevance of the PDO is rated substantial; the achievements against the first two objectives (systems outcomes contributing to improved education quality, and systems outcomes contributing to improved accountability), as measured against revised targets, are rated substantial, while the achievement of the third objective (systems outcomes contributing to improved sector efficiency) is rated modest; and efficiency is rated substantial. These ratings culminate in an Outcome rating of Moderately Satisfactory. Overall Outcome, weighted by disbursements made before and after restructuring, is Moderately Satisfactory. Rating Original outcome targets Revised outcome targets Relevance of PDO Substantial Efficacy Achievement of Objective 1 Modest Substantial Achievement of Objective 2 Modest Substantial Achievement of Objective 3 Modest Modest Efficiency Substantial 1. Outcome Moderately Unsatisfactory Moderately Satisfactory 2. Numerical value of outcome 3 4 ratings 3. Disbursement $0.92 million $5.7 million 4. Share of disbursements 14 percent 86 percent 5. Weighted value of the outcome 3 x 14% = 0.42 4 x 86% = 3.44 rating Moderately Satisfactory 6. Final Outcome Rating 0.42 + 3.44 = 3.86, rounded up to 4 Page 18 of 27 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Education System Improvement Project (P149005) a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome The project’s focus on systems strengthening contributed substantially to institutional capacity building. As detailed in the ICR (p. 28), project support culminated in: (a) strengthened capacity at the school level (focused on 240 beneficiary schools) in planning, management, implementation, procurement, monitoring, and evaluation of SDGs and in strengthened governance structures at the school level (school management committees and teachers' councils, among others); and (b) strengthened capacity of the DSME to design, administer, and oversee national and international student assessments, analyze data and produce performance reports, and monitor educational outcomes. Still, given the continued fragile political context, weaknesses in institutional capacity, and challenges in coordination among and within government institutions at central, municipal and school levels, there are important risks to achieved outcomes. As of the ICR's publication, the new government was in the process of drafting a new five-year education sector strategy, which has not yet been vetted with development partners and whose reform priorities are not yet known. There is potential for a follow-up operation to build on this project, but no formal request for such support has yet been issued. Continued investments will be needed to sustain capacities built in student learning assessments. The technical consulting positions for DSME financed by the project (statisticians, information technology specialists, testing specialists) were neither regularized nor incorporated into the MEST budget. This leaves the unit with fewer staff than needed and reliant on donor support to finance positions, posing a significant risk to sustaining capacity built to monitor education outcomes. The sustainability of the project’s achievements will depend on the new government’s ability to: (i) build consensus around next steps for further capacity strengthening and formulation of a sound, evidence-based sector strategy; (ii) mobilize partnerships and investments to support the implementation of sector policies and strategy; (iii) ensure that MEST’s high-level managers take the lead in implementing sector reforms; and (iv) continue capacity building of staff, including those implementing reforms, managing projects and resources, and delivering services. 8. Assessment of Bank Performance a. Quality-at-Entry The project was strategically relevant and responsive to the issues and challenges confronted by the education sector, both at design and at completion. It was appropriately focused on addressing the systems and capacity constraints that undermined, and continue to undermine, sector quality, accountability, and efficiency. However, these very issues of limited capacity were not properly factored into the project’s level of ambition, especially given the short implementation period, Kosovo’s fragile political context, and experience and lessons from the previous education operation. Indeed, much of this project's difficulty stemmed from issues that should have been foreseeable and reflected in its design, Page 19 of 27 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Education System Improvement Project (P149005) given that they already had been encountered -- and documented -- under the previous operation, including: high staff turnover within the PCU; political instability; frequent changes in MEST leadership; non-compliance with legal covenants; and low disbursements. Lessons were not taken fully into account, including the importance of: realistic assessments of institutional capacity and commitment and of country and sector readiness for reform (project preparation could have benefited from some high-quality political economy analysis, in this light); and simplicity of project design and institutional arrangements to match limited capacity. In short, even though the preparation properly diagnosed and analyzed sector issues and challenges and appropriately addressed systems and capacity issues, the scope and ambition of the project exceeded institutional capacity and constitutes a significant shortcoming to Quality at Entry. While poverty and inequities were addressed through SDG support, which provided for the selection of some beneficiary schools serving rural, disadvantaged populations, an opportunity was missed to include equity among the other higher-level objectives in the PDO, which could have infused equity concerns more systematically across all relevant areas of systems and capacity strengthening and all elements of the results framework. Environmental and fiduciary aspects were appropriately addressed during preparation. The Bank’s team undertook upstream assessments of systems and capacities for financial management and procurement, including associated risk assessments, and identified key training and other actions to mitigate most risks and to strengthen and supplement capacities for successful implementation. The overall (moderate) risk assessment was probably an underestimate. Identified risks that did materialize, particularly the need for political commitment in contentious areas (teacher career, higher education), had no mitigation measures specified in the PAD. As detailed in Section 9, M&E design was challenging in terms of defining indicators that would appropriately measure all of the eight expected systems strengthening outcomes in the initial design. M&E arrangements did clearly define indicators, roles and responsibilities for data collection, analysis, discussion, reporting, and use, and supported the decentralized school-based emphasis. Arrangements also included surveys and activities, which complemented project data. Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision The Bank’s supervision support was close, proactive, and focused on development impact. An important outcome of this supervision was its success in carrying out an evidence-based mid-term review, culminating in a well-founded project restructuring, resulting in the considerable and needed scaling back of the project’s scope and ambition, simplified implementation arrangements, additional activities to strengthen and supplement implementation capacity, and an improved results framework. Fiduciary aspects were also closely supervised by financial management (FM) and procurement specialists, guiding adherence to guidelines and addressing capacity constraints and other issues undermining the pace of implementation. Supervision inputs and processes were intensive. The Bank's team included both headquarters- and country-based staff, plus a local technical consultant, who supported preparation and implementation and provided institutional memory. Even during a two-year period of no local human development (HD) staff in the country office, the local consultant together with the Bank’s Country Management Unit ensured close monitoring through bi-weekly virtual review meetings with the PCU and TTL. The recruitment of a Page 20 of 27 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Education System Improvement Project (P149005) local HD specialist in 2019 enabled intensified, continuous hands-on support and follow-ups to address and correct operational bottlenecks in a context of weak institutional capacity and a challenging political environment. In all, the Bank undertook 17 review and implementation support missions, including a mid- term review and two back-to-back restructuring support missions, or one every four months. Support was especially intensive and critical in 2018 (with five implementation support visits) and 2019 (with four visits), enabling an in-depth project restructuring with a more realistic focus on PDO achievement. Close communication and support to the PCU was ensured through monthly video conferences and meetings from 2019 onwards. ISRs and aides-memoire were candid, comprehensive, and realistic in their assessments and ratings and focused on actions and agreements to enable PDO achievement. These supervisions and technical supports were undertaken in a difficult context, characterized by frequent changes of MEST leadership (four Ministers of Education and multiple General Secretaries), high turnover of technical staff, and constant PCU staff rotations throughout the project life and, in later years, setbacks caused by COVID-19. Bank proactivity mitigated the effects of these factors on implementation by establishing a set of milestones with agreed targets to be achieved by July 2019. These milestones proved to be a significant turning point for the project. Although there were three TTLs from preparation to completion, joint implementation missions ensured proper handovers and maintained open and continuous dialogue. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The project's original design was broad in scope, ambitious, and complex, presenting formidable challenges for M&E design. ICR’s Annex 9 presents an accurate depiction of the original theory of change, as interpreted from the PAD’s Table 1. The PDO aimed to strengthen eight selected systems that would contribute to higher-level objectives. Although specific, measurable, relevant, and time-bound, the four original PDO indicators and accompanying intermediate results indicators did not fully capture all eight capacity/systems-strengthening outcomes. Moreover, many of the targets were unrealistic for a four-year operation, given the country’s context. An analysis of Table 1 of the ICR (p. 4) linking the four original PDO- level indicators to the eight systems outcomes reveals shortcomings in the measurement of certain systems outcomes, especially: improved data collection, management, and utilization; improved education quality monitoring through student assessments; and improved accountability of higher education institutions for learning outcomes. The addition of a new PDO indicator at restructuring provided better measurement of improved education quality monitoring (with an appropriately reduced target). Arrangements, roles, and responsibilities for carrying out M&E activities were clearly laid out and appropriately embedded in the right institutions, at the right levels. The arrangements involved regular data collection through implementation support visits, administrative data, and semi-annual monitoring reports prepared by the PCU, also responsible for bring together various departments and actors in MEST and Page 21 of 27 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Education System Improvement Project (P149005) municipal education directorates to monitor and evaluate the project’s achievements. Data sources and responsible entities for their collection and reporting were specified in the design, and these were to be supported by the project’s activities to strengthen systems and capacities for policy planning, monitoring, and evaluation at the central, municipal, and school levels. Project monitoring and evaluation reports were expected to inform annual reporting under the Joint Annual Review for Education Sector. Design included three qualitative surveys (baseline, midline, endline) in a subset of beneficiary schools to study the effect of the SDGs on school-level capacity and autonomy. Citizen engagement was to be monitored over the project implementation cycle by measuring the participation of community representatives in planning and decision-making and their compliance with applicable requirements for citizen engagement, detailed in the Project Operational Manual. b. M&E Implementation M&E activities were largely implemented as planned. Data was collected, compiled, discussed, and reported on a timely basis. An independent three-phase survey implemented in a sample of beneficiary schools provided a qualitative assessment of the effectiveness of SDGs in strengthening school-level capacities, including participatory governance and management of SDPs. The PCU produced quarterly and annual progress reports. Following the 2019 restructuring, the PCU produced bi-monthly reports focused on outcome reporting, as agreed with the Bank. When the number of expected systems outcomes was substantially reduced at restructuring, the PDO-level and intermediate targets and indicators were adjusted in line with the reduced scope of the project. These adjustments addressed some of the weaknesses in M&E design related to project ambition and complexity. The Implementation Support Firm monitored the SDGs and submitted regular monitoring reports to the PCU, complemented by site visits to beneficiary schools conducted by the Bank and the PCU. The involvement of parents and other community members in the monitoring of school-based data helped build transparency, awareness, and ownership of school-level performance. Overall, the collection of PDO-level and intermediate results indicators, combined with other studies and M&E activity, enabled an adequate assessment of the project’s PDO. c. M&E Utilization Project data were regularly used to report on the status of implementation and results, reflected in internal Bank reporting and aides-memoire. Data were especially useful for inciting and informing the 2019 restructuring, with appropriate adjustments of the results framework and targets. The development and pilot testing of school report cards provided a useful tool for school-level monitoring and self- evaluation, and constituted an important element for dialogue on school performance and decision- making for improvements. The three-phase qualitative survey carried out in a sample of 140 beneficiary schools explored the effect of SDGs on school-level capacity and autonomy and was used to make necessary adjustments during implementation. The implementation and use of project M&E contributed to improvements in DSME capacity to monitor sector outcomes and in sector reporting standards. M&E Quality Rating Substantial Page 22 of 27 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Education System Improvement Project (P149005) 10. Other Issues a. Safeguards Environmental. No construction or rehabilitation activities were envisaged under the proposed project. The Project Operational Manual and School Grants Manual included specific language about the ineligibility of civil works under the project. No environmental safeguards were thus triggered under the proposed project, which was classified as Category “C." Social. The project did not trigger the Bank’s social safeguards policies, as no form of involuntary resettlement was allowed under the project. The project did support citizens engagement and school-based governance structures facilitating the participation of teachers, parents, and communities in the preparation, implementation, and monitoring of school development plans financed by SDGs. Of the 240 schools selected to receive SDGs, 140 were chosen on the basis of their rural locations, poorer socio-economic background, and lower education outcomes, with a view to improving equity. b. Fiduciary Compliance Financial Management. As part of preparation, an FM assessment was carried out to determine risks and to help establish adequate arrangements. FM responsibilities were assigned to MEST’s Budget and Finance Department, given its satisfactory performance in implementing a previous IDA-financed project. To ensure a sound fiduciary environment and the satisfaction of minimum requirements under OP 10.00, the following actions were agreed with the government: (i) the recruitment of a qualified FM specialist to be placed in the PCU, in keeping with the arrangement under the previous project; (ii) periodic, on-the-job training of relevant FM staff in Bank procedures; (iii) documentation of FM procedures in the FM Manual; and (iv) revision of the existing SDG Manual. The overall FM risk at appraisal was assessed as moderate. The MEST budget was to pre-finance project expenditures and be reimbursed by the Credit, on the basis of quarterly financial reports documenting such expenditures. Other allowed disbursement methods included direct payment and special commitment. Annual project budget needs were to be submitted by MEST to the Ministry of Finance (MoF) as part of the budget preparation cycle, and included in the Budget Law. This pre-financing mechanism ran the risk of insufficient or untimely budgetary allocations, as acknowledged at appraisal. Indeed, this risk materialized during implementation, resulting in delays in SDG execution in the early stages of implementation and in the cancellation of planned activities in the last few months of implementation, impeding full disbursement of the credit prior to closing. Delays in the annual budget law, difficulty in accurately forecasting budgets for SDGs, and occasional weak coordination among MoF, MEST, and municipalities all played a part. Following the 2019 restructuring, SDG transfers were streamlined by MEST, municipalities, and MoF, and a Memorandum of Understanding documented an agreement between MEST and municipalities, making it clear that municipalities were to allocate those funds to schools, which in turn were to follow Bank procurement rules. In general, FM was found to have followed Bank procedures, and FM requirements were complied with throughout implementation. The FM rating was downgraded only once (ISR#6, September 2018) from Satisfactory to Moderately Satisfactory, given budget delays for SDGs. Internal financial reports were Page 23 of 27 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Education System Improvement Project (P149005) submitted on time and in accordance with Bank procedures. Annual audits, including SDG audits, were submitted on time, all with unqualified opinion, revealing no significant irregularities. Procurement. The overall procurement risk rating was substantial at appraisal due to capacity constraints under previous projects implemented by MEST. The lack of procurement knowledge and experience of the multiple beneficiary schools was also a high risk factor. While mitigation measures were adequate, they were not always complied with. It took more than two years to recruit a full-time procurement specialist, since the one initially recruited was available only on a part-time basis. The ICR noted that procurement practices were found to have followed Bank procedures, and no major issues were identified during procurement reviews and implementation support missions. Minor issues encountered, including the incomplete entry of contract information in STEP (an online system to support the planning and tracking of procurement activities), were resolved with the Bank’s help. Nevertheless, several procurement packages experienced significant delays due to weak capacity of MEST’s evaluation committees, insufficient understanding of Bank procurement rules, and poor communication and unclear responsibilities between the PCU and MEST. A number of actions improved procurement performance, including: face-to-face training and hands-on support provided by the in-country Bank specialist to the MEST procurement evaluation committee; allocation of additional time of the Bank’s in-country specialist to advise on procurement activities; the hiring of a new, full-time procurement specialist, who started in January 2019; project simplification at restructuring; and the addition of more procurement support with the hiring of a part-time, experienced procurement officer in July 2019. c. Unintended impacts (Positive or Negative) None reported. d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory Satisfactory Moderately Bank Performance Moderately Satisfactory Satisfactory Quality of M&E Substantial Substantial Quality of ICR --- Substantial Page 24 of 27 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Education System Improvement Project (P149005) 12. Lessons The following lessons are derived from those presented in the ICR, reworded by IEG:  A project supporting multiple, complex, system-wide reforms in a politically fragile, low-capacity environment is not likely to succeed without undertaking, and factoring into the design, rigorous assessments of: the political economy of those reforms among key stakeholders; the commitment and ownership of the highest levels of government to reforms; the technical, managerial, and institutional capacity to implement them; and the availability of resources to ensure their proper financing and sustainability. Because these assessments were not systematically undertaken during project preparation, the initial design, scope, and outcome targets of the project were far too complex and ambitious for the country context.  Especially in the context of a constantly changing political environment with high turnover of sector leadership and key technical positions, a stable, adequately staffed PCU with low turnover is critical for ensuring smooth implementation, the achievement of results, and capacity building of ministry staff. Prior to restructuring, frequent rotation of PCU staff, including the Project Coordinator, undermined institutional capacity and memory and the pace of project implementation. Bank efforts to maintain implementation momentum during periods of government transition and to strengthen PCU staffing helped bring stability and enhance capacity, contributing to a positive turnaround in the project’s performance post-restructuring.  A timely, rigorous, candid, and evidence-based mid-term evaluation can be an effective tool for major restructuring of a non-performing project and for enabling its ultimate success. In the case of this project, the restructuring resulted in a simpler, more focused design, with fewer critical activities, much better suited to the complex, fragile environment and limited institutional capacity in Kosovo. IEG’s review offers an additional lesson:  In a country where issues of strong inequities prevail, there are multiple opportunities to address these issues both directly and indirectly. Among the 240 schools selected to benefit from SDGs, 140 were selected for their rural locations and for their service to disadvantaged populations. During the June 1, 2022 meeting with IEG, the TTL provided additional information and insight on how the project also factored equity concerns into the capacity and systems strengthening activities. Disadvantaged schools benefited from school- level management capacity building and, disproportionately so, from the school report card pilot, which enabled comparison to other schools in the municipality and nationally. The in- depth report on PISA 2018 addressed the special theme of learning equity across the country. The technical assistance exploring school financing formulas factored in the need for increased financing for disabled students and disadvantaged schools. Had equity been included as a higher-order objective, and its tracking and measurement taken into account in the choice of intermediate results and/or outcome indicators (in other words, had equity been more systematically reflected in the various components of the project’s results chain), the project’s contribution to equity may have been more fully assessed and developed. Page 25 of 27 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Education System Improvement Project (P149005) 13. Assessment Recommended? No 14. Comments on Quality of ICR Quality of evidence. The ICR’s presentation of the evidence was systematic and credible, supporting its reporting of achievements. Annex 1 presented baselines, targets (original and revised), and actual achievements, based on project data that was routinely collected and compiled, as noted in the M&E section. These data were woven into the ICR’s efficacy section, along with results from qualitative surveys on the project’s performance. Quality of analysis. The ICR undertook a credible analysis of project performance, drawing on project data and related surveys, and weaving in elements of its own direct observations and interviews of key informants during the ICR preparation process. The analysis was particularly challenging for this ICR because: (a) the project, especially in its initial stage, was extremely complex, largely focused on difficult-to-measure systems and capacity reforms, and linked to higher-order objectives; and (b) restructuring represented a substantial change in project scope and ambition. Indeed, the challenge of this ICR’s organization and analysis of efficacy is raised among the ICR’s lessons (p. 40). The decision to organize the analysis around the eight expected outcomes separately, while still assessing their links and contributions to higher-order objectives, was the right one, and it was well executed through the text and a series of tables presenting tallies of activities, intermediate results, and ratings for each outcome. Key factors affecting implementation, Bank performance, and M&E were well assessed. Quality of lessons was strong. Lessons were distilled from the evidence and analysis presented in the ICR; relevant to specific experiences for the project while also relevant to other countries and projects embarking on similar reforms; and consistent with ratings in the ICR. Results orientation. Thanks to the ICR’s organization around the eight systems/capacity outcomes and their links to higher-order objectives, its results orientation was strong and focused on what occurred as a consequence of the project. A subsequent discussion with the TTL informed two paragraph under Section 4.of this ICRR exploring attribution and the counterfactual. Internal consistency/adherence to guidelines. The various parts of the ICR were logically linked and coherent, and its reporting on safeguards and fiduciary compliance was in keeping with guidelines. The ICR correctly opted for a split rating methodology and was systematic and clear in its implementation, albeit with a small shortcoming, which did not impede IEG’s review: the ICR’s Table 12 (Summary of Overall Outcome Rating/p. 21) and accompanying text presented ratings “at restructuring” and “at completion.” The guidelines specify that a split rating should assess outcomes at the end of the project against, respectively, the original PDO and outcome targets, and the revised PDO and outcome targets. Also, the presentation of disbursements in SDRs would have provided for a more accurate calculation of disbursement rates. a. Quality of ICR Rating Page 26 of 27 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Education System Improvement Project (P149005) Substantial Page 27 of 27