Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kyrgyz Global Partnership for Education (P132490) Report Number : ICRR0021507 1. Project Data Project ID Project Name P132490 Kyrgyz Global Partnership for Education Country Practice Area(Lead) Kyrgyz Republic Education L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) TF-16201,TF-16209,TF-A4531 31-Dec-2017 12,799,458.50 Bank Approval Date Closing Date (Actual) 02-Apr-2014 30-Jun-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 12,800,000.00 12,800,000.00 Revised Commitment 12,800,000.00 12,799,458.50 Actual 12,799,458.50 12,799,458.50 Prepared by Reviewed by ICR Review Coordinator Group Judith Hahn Gaubatz Judyth L. Twigg Eduardo Fernandez IEGHC (Unit 2) Maldonado 2. Project Objectives and Components a. Objectives According to the Grant Agreement (page 5) and the Project Appraisal Document (PAD, page 8), the project objectives were as follows: • To increase equitable access to preschool education; and • To establish conditions for improving its quality. Page 1 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kyrgyz Global Partnership for Education (P132490) b. Were the project objectives/key associated outcome targets revised during implementation? No PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? No d. Components (Note: Project amounts reported below are for the Global Partnership for Education (GPE) grant only.) 1. Expansion of quality pre-primary education (Appraisal: US$ 11.6 million; Actual: US$ 11.23 million): This component aimed to increase access to preschool education, as part of a school readiness program to enter Grade 1. Activities included: development and delivery of a teacher training program for preschool education; provision of learning materials and equipment for preschool classes; training for school Deputy Directors on implementing the school readiness program in existing primary schools; establishment of community-based kindergarten programs in disadvantaged areas; and a pilot initiative for inclusive education to serve children with special education needs (SEN). 2. Improved policy, programs and system effectiveness (Appraisal: US$ 0.34 million; Actual: US$ 0.66 million): This component aimed to improve policy and programs in support of the proposed reforms in the preschool sub-sector, such as the introduction of a shift-based model, inclusive education, and preschool financing. Activities included: technical assistance on these reforms; dissemination of newly introduced Early Learning Development Standards; piloting of an Early Development Instrument (EDI) measurement tool; and classroom observation assessments of teacher performance. 3. Communications and implementation support (Appraisal: US$ 0.76 million; Actual: US$ 0.80 million): This component aimed to finance communication activities and project implementation support. Activities included: engaging stakeholders (including parents) on the importance of early childhood education; informing stakeholders on current reforms in the preschool sector; third party project monitoring; and project implementation support. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost •The appraised project cost was US$ 16.37 million. This amount included a Borrower contribution of US$ 3.57 million for teacher salaries. Not including this amount, the appraised and actual project cost was US$ 12.8 million. Page 2 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kyrgyz Global Partnership for Education (P132490) Financing •The project was financed by a GPE grant in the amount of US$ 12.8 million, which disbursed in its entirety. Borrower contribution • The appraised Borrower contribution was US$ 3.57 million to cover 100% of teacher salaries; the project team confirmed that the government fulfilled this commitment. Dates • The project closing date was extended from December 2017 to June 2018 to allow completion of additional outputs. Project savings were incurred that allowed for a greater number of teachers to be trained and preschool programs to be established, as well as a piloting of a per-capita financing model. 3. Relevance of Objectives Rationale Although considered one of the poorest economies in the Central Asia region, the Kyrgyz Republic has made some progress in the creation of a liberal market economy and integration into the global economy; however, the country is still marked by governance challenges and political instability. In terms of demographics, about one-third of the population is under the age of 15, although the specific population group aged 7-15 years declined in the period from 2007 to 2012, while the group aged 0-6 years increased. In the education sector, compulsory education includes primary and lower secondary education (grades 1-9) and coverage is high, with net enrollment in lower secondary at 92 percent. However, quality of education is mixed, as both national and international student assessment results indicate low outcomes (in 2009, two-thirds of Grade 4 students did not display mastery of essentials in language, math, and science; more than three-quarters of 15-year-olds were considered "functionally illiterate"). Only 18 percent of children aged 3-6 years were enrolled in preschool in 2010, with significant income and geographic disparities. Of these, 90 percent were in formal state-funded kindergarten programs that offer four hours of instruction plus four hours of rest and meals. These programs require costly facilities and support staff. In recent years, alternative preschool models have been introduced with support from international agencies, offering instruction on a half-day basis, requiring only basic facilities and solely teaching staff. Demand for Page 3 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kyrgyz Global Partnership for Education (P132490) these "community kindergarten" programs has increased in light of scarce public funding for the more formal full-day programs. A 2009 Law on Preschool Education set forth the goal of universal coverage with affordable and high- quality programs. The government's 2020 Education Sector Plan includes equal access to education, particularly for preschool, as a key goal, along with improving the quality and management of education. A special focus on integrating children with special needs into the regular preschool system is also included. The Bank's Country Partnership Strategy for FY14-17, in effect during project preparation, highlighted improved public service delivery as a country development goal, including in education, as a critical pre-condition for human capital formation. The Bank's current Country Partnership Framework for FY19-22 similarly highlights the development of human capital as an objective, with continued emphasis on early childhood education. Rating High 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective To increase equitable access to preschool education Rationale The theory of change was overall sound. The project's support for establishing and equipping preschool classes (including those in disadvantaged areas), training preschool teachers, and training teachers on SEN children was likely to lead to the intended outcome of increased access to preschool. Equity in access was to be achieved through expanding classes in the poorest communities (according to poverty mapping and enrollment data) as well as piloting activities for reaching SEN children. The longer-term goal was to strengthen foundational academic skills to increase employability. The shift-based approach to preschool education enabled an increased number of children to be served (an approach that eliminated the more costly facility and support staff but did not reduce instruction time). Outputs • Establishment of 2,779 preschool "school preparatory" classes in 2,204 existing primary schools, equipped with furniture and learning materials (no target identified). Page 4 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kyrgyz Global Partnership for Education (P132490) • Establishment of 120 community-based kindergarten programs (target: 100), including 16 in the hardest- to-reach mountainous regions. The ICR (page 13) noted that the project required local governments to contribute towards funding of the programs by providing the buildings and teacher remuneration for at least five years, thus contributing to a sense of ownership by local stakeholders and financial sustainability. Efficient procurement led to project savings, which enabled the establishment of 20 programs beyond the targeted 100. • Training of 5,210 teachers (target: 3,300) in preschool education, primarily selected from the existing pool of primary education teachers. • Provision of learning materials in Kyrgyz, Russian, and Uzbek languages to all project schools. • Support to 33 schools in one pilot district to serve SEN children. This included the provision of SEN learning materials to all pilot schools and training of 244 teachers and school social workers on the SEN model (target: 150). This inclusive education model was evaluated and recommended for implementation nationwide. • Information and communication activities on the importance of early childhood education. Outcomes • Enrollment in pre-school "school preparatory" classes was 116,985 by Year 3, surpassing the target of 80,000. In Year 1, enrollment was 114,451; in Year 2, enrollment was 137,886. The total number of children participating in the program over the three year period was 369,322. The ICR (page 13) reported that the total number of six-year-old children in project years 1-3 was 395,842; therefore, 93% coverage was achieved. • Enrollment in community-based kindergartens in the targeted communities was 10,000, achieving the target of 10,000. The community-based programs were implemented in poor, hard-to-reach mountainous regions. Hence, services were provided to disadvantaged children, in their mother tongues. In the three regions with the highest poverty rate, the number of children from poor households enrolled increased from 16,195 in 2015-16 to 17,660 in 2016-17, dropping to 15,236 in 2017-18. • Enrollment of SEN children in mainstream schools was 50 (no target identified). • The share of groups in community kindergarten programs that operate in the Uzbek language was 10.5%, surpassing the target of 9.0%. The share of groups that operate in the Kyrgyz language was 87.7%, surpassing the target of 62.0%. • The ICR noted that as gender parity in gross enrollment rate had already been achieved for the country, the project intended to simply monitor this indicator to ensure that gender parity was maintained. The gender parity index for gross enrollment rate for pre-primary school was maintained at 1.0. Page 5 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kyrgyz Global Partnership for Education (P132490) Due to the surpassing of targets for preschool enrollment (attributable to both expanded supply due to procurement savings and increased demand due to outreach), along with evidence of having reached disadvantaged populations, achievement is rated High. Rating High PHREVDELTBL PHEFFICACYTBL Objective 2 Objective To establish conditions for improving its quality Rationale The theory of change was overall sound, with the project's support for teacher training (including implementation of teacher performance assessment tools), learning materials, strengthened legal and regulatory framework, and measurements of student readiness likely to contribute to conditions for improved quality. Outputs • Provision of a package of essential teaching-learning materials to all 2,779 preschool "school preparatory" classrooms (target: 100% coverage). The ICR (page 18) reported that the storybooks in Russian and Kyrgyz languages were submitted to the XIV CIS Member States International Competition among publishers and won first place for quality and content. • Training of 5,210 teachers (target: 3,300) for the preschool "school preparatory" program, most of whom were selected from the pool of existing primary education teachers. • Training of 310 teachers (target: 250) for the community-based kindergarten programs. • Training of 2,930 deputy directors on the school preparatory program (target:2,200). • Training of 210 methodologists to provide one-on-one mentoring support to teachers. • Introduction of the EDI, which was used to monitor school readiness. • Dissemination of early learning standards and subsequent revision of the preschool curriculum. • Piloting of per capita financing for the education sector. Outcomes Page 6 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kyrgyz Global Partnership for Education (P132490) • 100% of the teachers trained in the preschool "school preparatory" program were certified to teach, surpassing the target of 90%. • The quality score of teacher pedagogical practices improved by 10%, surpassing the target of 5% improvement. Two sample-based surveys were conducted in 2015 and 2017. Emotional support scores improved by 9.81%; classroom organization scores improved by 10.56%; and instructional support scores improved by 8.61%. The project team clarified that an improvement in scores beyond 5% is considered statistically significant in terms of education outcomes. • The EDI was tested in two rounds and then adopted by the Ministry of Education and Science (MoES) for nationwide use. In addition, the following results from the EDI indicate increased learning attainment among those enrolled: • Improved language and cognitive development skills, physical well-being, social competence, emotional maturity, communications, and general knowledge ("effect size" greater than 2.0, signifying statistical significance). • Decreased disparity among children from the lowest wealth group in the areas of language and cognitive development and social competence. Due to the provision of quality learning materials and surpassing of targets for improved teacher performance, along with improved student learning outcomes, achievement is rated High. Rating High PHREVDELTBL PHOVRLEFFRATTBL Rationale Achievement of the objective to increase equitable access to preschool education is rated High, due to the surpassing of targets for preschool enrollment, along with evidence of having reached disadvantaged populations. Achievement of the objective to establish conditions for improving preschool education quality is rated High due to the provision of quality learning materials and surpassing of targets for improved teacher performance, along with improved student learning outcomes. The overall Efficacy rating is High. Overall Efficacy Rating High Page 7 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kyrgyz Global Partnership for Education (P132490) 5. Efficiency During project appraisal, there was no cost-benefit analysis conducted, although the PAD's economic and financial analysis (Annex 6) cited generally accepted international evidence on the benefits and cost- effectiveness of preschool education. In addition, the PAD drew upon data and analysis from sector studies to support the conclusion that support to the preschool sector was cost-effective: the 2009 Program on International Student Assessment; the 2012 Public Expenditure Review, which concluded that "preschool attendance is a more cost-effective policy for improving student achievement (measured by test scores in reading) in students tested at 15 years of age compared to other interventions such as teacher training or pupil:teacher ratios"; and the 2014 Public Expenditure Review, which concluded that coverage of preschool services could increase four-fold without increasing costs if the sector offered shift-based vs. full-day programs. The ICR (Annex 4) provided similar cost-effectiveness data. An analysis of unit costs (desks, chalkboards, cabinets, teacher desks) with the two previous Fast-Track Initiative (FTI) projects pointed to comparable unit costs. An additional analysis of cost per hour for the shift-based versus full-day program indicated the following: • In rural areas, the cost per hour for the project's "school preparatory program" was 1,478 Kyrgyz som, and the cost per hour for the project's community-based kindergartens was 3,158 Kyrgyz som. These compared favorably to the cost per hour for traditional state-funded kindergarten, which was 4,727 Kyrgyz som. • In urban areas, the cost per hour for the project's "school preparatory" program was 1,338 Kyrgyz som, and the cost per hour for the project's community-based kindergartens was 2,553 Kyrgyz som. These compared favorably to the cost per hour for traditional state-funded kindergarten, which was 4,160 Kyrgyz som. Despite the shortened school day (three-hour program with no rest or meal time), the quality of learning appears to have not been affected; in fact, EDI scores indicate learning improvements. Additionally, the project accumulated savings due to efficient procurement, which allowed the exceeding of targets: additional preparatory classes at existing primary schools and newly-established community kindergartens, and training of methdologists. The project did experience a six-month closing date extension, although the ICR reported that this was due to the project's savings, which were being used for additional outputs to increase project impact. There were no major implementation problems reported that would have caused other project inefficiencies. Page 8 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kyrgyz Global Partnership for Education (P132490) Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Point value (%) *Coverage/Scope (%) Available? 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of the objectives is rated High. Achievement of the objective to increase equitable access to preschool education is rated High due to surpassing of targets for preschool enrollment, along with evidence of reaching disadvantaged populations. Achievement of the objective to establish conditions for improving its quality is rated High due to the provision of quality learning materials and surpassing of targets for improved teacher performance, along with improved student learning outcomes. Efficiency is rated Substantial due to unit cost analyses indicating efficient use of project resources. These ratings indicate no shortcomings in the project's preparation and implementation, leading to an Outcome rating of Highly Satisfactory. a. Outcome Rating Highly Satisfactory 7. Risk to Development Outcome Technical capacity, with regards to preschool curriculum and assessment tools, was well established through project support. Political support for expanding access to preschool is likely to be sustained, as reflected in the Law on Preschool Education that has been in effect since 2009; in addition, the pilot program on the inclusive education model was included in the National Concept on Inclusive Education, which is planned to be implemented nationwide. For the community-based kindergartens, local governments were required to contribute funding, specifically by providing the facility and funding for teacher salaries for at least the first five years, thereby contributing to financial sustainability. However, the country is still vulnerable to external shocks, and there will likely continue to be some volatility in the public budget for education. Page 9 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kyrgyz Global Partnership for Education (P132490) 8. Assessment of Bank Performance a. Quality-at-Entry The project built upon two previous GPE/FTI grants, the second of which had included technical assistance for developing a 3-4 month "school readiness" preschool program. Due to unexpectedly high demand for the preschool services, including in rural areas, the government made the decision to offer a full-year "school readiness" program using existing primary school facilities, but it needed further donor support to update learning materials and teacher capacity. The project design responded to these needs, while also including targeting strategies to increase coverage, for example using poverty maps and enrollment data to identify disadvantaged groups and also piloting an inclusive education program to identify and benefit SEN children. Given the inadequacies of the existing government structure, such as the tendency of sectors to operate in "silos," the project design focused on early childhood education, as an intermediate step towards developing a broader early childhood development approach for the country. The risk assessment identified substantial risks in the areas of political commitment, implementation capacity, and fiduciary management. The monitoring and evaluation (M&E) framework was strong with clear baseline and target data, relevant and measurable indicators, and timely evaluative studies, though a more explicit focus on equity outcome indicators at the PDO level could have been included. Quality-at-Entry Rating Satisfactory b. Quality of supervision There were regular supervision missions carried out during the project period, including a midterm review mission. The Bank team drew upon the existing strong relationship with the MoES, due to regular communication and past history of operational support, to ensure timely responses to implementation issues. M&E implementation was strong, with data collected as planned and evaluative studies conducted, the findings of which were used to inform project adjustments and demonstrate achievements. Savings due to efficient procurement were used to increase project impact. The fiduciary team was based in the region and provided regular support, with no major issues reported for either financial management or procurement. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Page 10 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kyrgyz Global Partnership for Education (P132490) Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The M&E design was strong, with both the results framework and M&E arrangements well considered at the project's outset. The results framework monitored key outputs as well as outcomes using measurable and relevant indicators that covered key aspects of the project objectives (access and quality); however, there was no explicit PDO-level indicator for the aspect of equity, other than "number enrolled in community-based kindergartens" (presumably reaching the poor) and intermediate outcome indicators (number of special education needs children supported). The project also introduced internationally recognized measurement tools for teacher performance and early childhood development, both of which were to be implemented in two rounds during the project period. b. M&E Implementation Indicators were tracked regularly as planned. Evaluative assessments of key activities - teacher training and school readiness - were conducted, not only providing robust evidence of project impact but also informing adjustments to project design during the project period. Both assessments included two rounds of sample- based classroom observations/testing to provide baseline and final results commensurate with the project period. MoES staff were trained on the implementation of the assessment tools, and M&E functions have been integrated into existing MoES routines to support capacity development. c. M&E Utilization As noted above, evaluative activities were sequenced appropriately and thus findings were used to inform project design adjustments. For example, results from the classroom observations of teachers in the first assessment round led to adjustments in the training modules to address gaps in key domains and also to introduce one-on-one teacher mentoring. Both assessment tools were also approved by the MoES for further use nationwide. The ICR (page 21) also reported that data were used to demonstrate progress to the public and key stakeholders. M&E Quality Rating Substantial Page 11 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kyrgyz Global Partnership for Education (P132490) 10. Other Issues a. Safeguards The project was classified as an Environmental Category "B" project due to minimal school construction activities. The safeguard policy on Environmental Assessment (OP/BP 4.01) was triggered, and an Environmental Management Framework was prepared. The Framework required the use of site-specific checklists and Environmental Management Plans for each school construction site prior to initiating work. A regional safeguard team conducted reviews of project sites and confirmed that all construction and waste treatment was in compliance with safeguards. b. Fiduciary Compliance Financial management: Regular reviews of financial management were conducted, and no major issues with compliance were reported. There was no information reported in the ICR on whether audits were conducted or whether there were any qualified opinions. The project team reported that it was not aware of any qualifications on audits. Procurement: Regular reviews of procurement, including post-procurement reviews on a sample basis, were conducted, and no major issues with compliance were reported except for minor delays at the project start. Efficiencies in procurement led to project savings, which were then used to deliver additional project outputs. c. Unintended impacts (Positive or Negative) None reported. d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Highly Satisfactory Highly Satisfactory --- Bank Performance Satisfactory Satisfactory --- Page 12 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kyrgyz Global Partnership for Education (P132490) The results framework could Quality of M&E High Substantial have focused on "equity" more explicitly Quality of ICR Substantial --- 12. Lessons Lessons drawn by the ICR, adapted by IEG: • A strong and clearly laid out results chain, translated into a strong results framework and M&E design, can aid project performance and reporting on achievements. In the case of this project, the well-focused results chain with attributable outputs and outcomes, and relevant and measurable indicators on access and quality, led to effective monitoring and feedback loops. • Community-based kindergartens can be a low-cost alternative to preschool education when the goal is to expand access in remote areas. In the case of this project, local contributions via facilities and teacher salaries helped to lower costs, and assessments showed that learning outcomes improved in poor areas with this alternative model. 13. Assessment Recommended? Yes Please explain To verify outcomes, particularly the transition rates from pre-primary to primary schools and learning outcomes, and to learn lessons from the significant achievements in Efficacy and M&E. 14. Comments on Quality of ICR The ICR had a strong focus on results and a clearly articulated results chain. It was notable for its use of M&E data and findings to substantiate achievements. The quality of the analysis was strong, as the results chain presented in the ICR clearly linked project outputs to outcomes. Although the results framework did not include sufficient indicators on equity, the ICR included additional data and analysis (Annex 6) to supplement. However, particularly given the weak governance and low fiduciary capacity noted in the PAD, Page 13 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kyrgyz Global Partnership for Education (P132490) there was very limited information reported in the ICR on fiduciary performance, other than that there were no problems with compliance with fiduciary standards, and there was no information on audit results. a. Quality of ICR Rating Substantial Page 14 of 14