Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Global Partnership for Education (GPE)-4 (P131441) Report Number : ICRR0021236 1. Project Data Project ID Project Name P131441 Global Partnership for Education (GPE)-4 Country Practice Area(Lead) Tajikistan Education L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) TF-15134 30-Sep-2016 16,200,000.00 Bank Approval Date Closing Date (Actual) 17-Sep-2013 30-Sep-2017 IBRD/IDA (USD) Grants (USD) Original Commitment 16,200,000.00 16,200,000.00 Revised Commitment 16,200,000.00 16,200,000.00 Actual 16,200,000.00 16,200,000.00 Prepared by Reviewed by ICR Review Coordinator Group Judith Hahn Gaubatz Judyth L. Twigg Joy Behrens IEGHC (Unit 2) 2. Project Objectives and Components a. Objectives According to the Grant Agreement (page 8) and the Project Appraisal Document (PAD, page 17), the project objective was as follows: • To contribute to improving the learning conditions in preschool and general education. Page 1 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Global Partnership for Education (GPE)-4 (P131441) b. Were the project objectives/key associated outcome targets revised during implementation? No PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? No d. Components (Note: The appraised amounts below reflect the amounts officially approved for the project, which differ from the amounts requested in the original project proposal.) 1. Increasing access to quality early childhood education (ECE) program (Appraisal: US$ 2.55 million; Actual: US$ 2.2 million): This component aimed to increase access to affordable and quality ECE programs. Activities included: an analysis of the preschool sector; and reinforcement of state kindergartens and early learning centers (ELCs) through provision of teaching and learning materials and teacher training. 2. Enhancing quality of education (Appraisal: US$ 3.25 million; Actual: US$ 3.6 million): This component aimed to help shift the curriculum from knowledge-based to competency-based, and to broaden the teaching-learning process with active learning. Activities included: communications campaign about the revised curriculum; teacher training, training for school directors, and provision of teaching and learning materials; and a pilot program to introduce the new curriculum. 3. Improving child-friendly learning environments (Appraisal: US$ 8.0 million; Actual: US$ 7.9 million): This component aimed to increase access to improved learning environments in general secondary education. Activities included: construction or rehabilitation of approximately 40 schools; and equipping of those schools with furniture. 4. Strengthening system capacity (Appraisal: US$ 2.4 million; Actual: US$ 2.5 million): This component aimed to strengthen capacities at the central and local levels to manage the education sector. Activities included: training on fiduciary management, monitoring and evaluation (M&E), and school leadership; training on per-capital financing to include preschools; and development of new modules for the Education Management Information System (EMIS). e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost • The appraised project cost was US$ 16.2 million, and the actual project cost was also US$ 16.2 million. • Due to significant depreciation of the local currency against the US Dollar, plus efficient procurement, there was US$ 1.3 million in savings. This savings was used for civil works and expansion of teacher Page 2 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Global Partnership for Education (GPE)-4 (P131441) training and curriculum distribution. Financing • The project was financed entirely by a Global Program for Education (GPE) Grant of US$ 16.2 million, of which 100% disbursed. Borrower contribution • There was no expected Borrower contribution, beyond local contributions to minor civil works such as fencing, utility connections, etc. Dates • May 2016: The project design was modified such that the upgraded curriculum in Math and Tajik would be piloted only in grades 5 and 6, rather than in grades 5-11. This did not affect the key project indicators or targets. The project closing date was also extended from September 2016 to September 2017 to allow time to conduct the above-mentioned pilot before national roll-out. 3. Relevance of Objectives Rationale Tajikistan is a small, low-income country, having achieved independence from the Soviet Union in 1991. It has gone through various periods of civil unrest and economic contraction, though its economy is now achieving positive growth. The population is generally young (37% under the age of 14 years) and rural (73%). Job opportunities are far below the needs, given the number of secondary school graduates, and the skills profile of workers is generally low. At appraisal, the preschool enrollment rate was among the lowest in the region (net enrollment rate at 8.9% in 2012), with 80% of those enrolled in state-run kindergartens. Preschool education financed partly by private households -- in the form of ELCs -- was only recently developing in the sector. The main constraints in the preschool sector were the shortage of qualified teachers, lack of quality teaching and learning materials, and lack of adequate facilities. Basic education (primary grades and lower secondary grades) was marked by low quality outcomes (88% graduation rate for 9th grade girls, low reading fluency for 2nd and 4th grade, low critical thinking skills). The main constraints to the basic education sector were low quality of the curriculum and pedagogy, lack of supplementary teaching materials, overcrowded facilities, and lack of data and information for strategic decision making. Given these factors, the project objective was Page 3 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Global Partnership for Education (GPE)-4 (P131441) highly relevant to the country conditions. The project objective was also highly relevant to the government’s National Strategy for Education Development for 2012-2020, which had as its main goal to create the conditions to ensure universal access to relevant and quality education. It was less relevant to the Bank’s Country Partnership Strategy for FY2015-18, which recognized the importance of early childhood education but prioritized higher education for financing. As noted by the ICR (page 13), this is due to the government’s recent progress on debt relief such that Bank assistance to the country was changed from grants to partial grant/partial credit. Therefore, the government had shifted to seeking other donor assistance – on more favorable financing terms – for preschool and general education, while seeking Bank assistance for higher education only. Rating Substantial 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective To contribute to improving the learning conditions in pre-school and general education. Rationale The theory of change was overall sound. As noted in project documentation, "learning conditions" were understood to include education program standards and content, teaching-learning materials and practices, and/or the physical environment. Project outputs supported improvements in curriculum, the supply of teaching and learning materials, teacher training to improve pedagogical methods, and construction/rehabilitation of classrooms. These interventions were likely to lead to improved conditions for learning, particularly for underserved groups, as the selection criteria included the level of subsidization from central budget (indicator of relative poverty of the district) and the targeting of marginalized populations (including in rural areas for preschools). Additional outputs to improve education sector management included development of information systems and budget allocation systems to improve decision making. Preschool The preschool activities were implemented in 32 rayons in three provinces, covering 450 ELCs and 450 State Kindergartens that serve 23,610 preschool children in total. A new preschool curriculum was developed by other donors (including the Aga Khan Foundation and UNICEF), while the Bank, through this project, financed the training of teachers on the new curriculum, conducted monitoring, and provided mentoring support to trained teachers. Outputs Page 4 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Global Partnership for Education (GPE)-4 (P131441) Education program standards • Training of 24 trainers on the preschool curriculum. • Training of 1,182 preschool teachers on the new preschool curriculum (target: 900), of whom 1,034 were deemed qualified to teach. • Training of 483 preschool directors on school management. • Conducting of an ECE sector review including an institutional census, a legal-regulatory analysis, a financing analysis, and a services and facilities assessment. 73 preschools were assessed in terms of their physical space and hygiene, interaction, language development, and other education activities. According to the ICR (page 14), this assessment laid the basis for establishing a preschool services assessment system and self-assessment for preschool institutions. Teaching-Learning materials • Provision of teaching and learning materials at 900 preschools, benefitting 38% of all preschool students in the country (target: 28%). Physical environment • Provision of furniture for 450 ELCs. The ICR noted that the project supported existing preschools, rather than constructing new ones. Outcomes • 18,000 children were enrolled in preschools supplied with teaching-learning materials and trained teachers, achieving the target of 18,000. General Education The project activities contributed to the broader shift from knowledge-based to competency-based curriculum in general education. The United States Agency for International Development (USAID) supported work on primary grade reading, while UNICEF supported integration of a Life Skills curriculum into the new curriculum. The Bank complemented these curriculum upgrades through the project outputs listed below. Outputs Education program standards Page 5 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Global Partnership for Education (GPE)-4 (P131441) • Development of new curriculum package for all subjects (but Tajik language) in primary grades; consolidation of inputs from other partners into one primary grades package for national implementation from September 2016. • Development of new curriculum package for two subjects (Math and Tajik) for grades 5-11. Curriculum for grades 5-6 for those two subjects was implemented as a pilot program through project support (original target: implementation for grades 5-11), and then introduced nationwide using state budget resources in September 2017. The ICR (page 13) also reported that the pilot program attracted funds of other donors to support secondary curriculum upgrades for other subjects. • Training of 5,395 primary grade teachers (target: 2,600). The PAD (page 20) noted that 2,000 primary teachers represented about 7.4% of all primary teachers. • Training of 130 trainers, and 187 methodologists and university teachers. • Training of 2,418 school directors on school management. Teaching-Learning materials • Provision of supplemental reading books for primary grades to all schools nationwide. Physical environment •Construction of 24 school buildings and 2 district education offices, and rehabilitation of 14 school buildings. This included 202 new classrooms and 62 rehabilitated classrooms. 20 schools were also adapted for children with disabilities, benefitting 766 students. Outcomes • 160,000 primary students enrolled in schools with improved learning conditions, surpassing the target of 100,000. • 18,978 students benefitted from physical infrastructure upgrades at schools, surpassing the target of 7,900. The ICR (Annex 1) reported that the overachievement was due to additional works funded out of project savings. • 52% of trained teachers follow the revised curriculum and apply improved teaching-learning practices in Mother-tongue language and math, achieving the target of 50%. This was based on an impact evaluation of 150 randomly selected trained primary school teachers (classroom observation) and 89 school directors (interviews). Page 6 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Global Partnership for Education (GPE)-4 (P131441) • An impact evaluation of the school infrastructure activities reported the following outcomes: (i) improved school attendance by students (for boys and girls proportionally); (ii) decreased drop outs among secondary education students; (iii) decreased rate of illness among students; and (iv) reduced teacher absenteeism. No specific data is provided, nor is attribution to the project activities clear. Other sector outcomes • Due to project-supported training on per capita financing, this approach is now being implemented nationwide for preschools. The ICR (page 18) reported that the increased autonomy in budget distribution has helped to: (i) increase transparency as budgets are posted in schools; (ii) mobilize additional (parental and/or community) contributions that are integrated into a single estimate of resources and expenditures; and (iii) allocate resources to the specific school's priorities. • Due to the project-supported development of additional EMIS modules on preschool, primary vocational education and training, secondary vocational education and training, and higher education, sector management has improved. The EMIS is now fully functional and the ICR (page 14) reported that it is now the main source of education sector statistics. Education statistical editions for academic years 2013/2014 to 2016/2017 have been published in three languages. The ICR (page 16) suggested that with data on institutions, students, teachers, and materials, the Ministry of Education has increased its capacity to identify gaps and determine future priorities. Rating High PHREVDELTBL PHOVRLEFFRATTBL Rationale Due to the achieving and/or surpassing of targets in providing preschool and primary/secondary school children with improved learning conditions, achievement is rated High. Targets were achieved for all project indicators, and significantly surpassed for two. Overall Efficacy Rating High 5. Efficiency The project design featured cost-effective interventions, such as the provision of preschool education (as Page 7 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Global Partnership for Education (GPE)-4 (P131441) early development of cognitive and non-cognitive skills leads to better future learning and labor market success) through a less expensive model of half-day preschool programs, and support to a per-capita funding mechanism to ensure efficient costing methodology. The economic analysis at appraisal (PAD, Annex 6) referred to these aspects of project design in its discussion of the rationale for selection of activities; however, there was no specific analysis such as cost-benefit ratio or economic rate of return for these specific interventions or for the project as a whole. The economic analysis at closing (ICR, Annex 4) reported the varying unit costs for teacher training (depending on the geographic location, number of training days, preschool vs. primary school training), which ranges from TJS 919 for seven-day training (preschool teachers) to TJS 1,279 for ten-day training (primary teachers). This compared favorably to the TJS 130 per trainee for per day cost under teacher training courses funded by the state budget. The Annex noted that unit costs for teacher training under other donor- supported projects is higher, but does not provide specific data to verify the comparison. The unit cost comparison of classroom construction was also provided. For a school with 8 classrooms: the project-financed school cost was US$ 347/ sq. meter; the Ministry of Education-financed school cost was US$ 336/ sq. meter; and the “other” donor-financed school cost was US$ 463/ sq. meter. Averaging the unit cost for all schools (ranging from 6-12 classrooms), the project-financed school cost was US$ 331/ sq. meter; the Ministry of Education-financed school cost was US$ 352/ sq. meter; and the “other” donor-financed school cost was US$ 408/ sq. meter. The unit cost of furniture and equipment for this project was higher compared to a prior Bank education project in the country (US$ 420 vs. US$475) due to modified technical specifications to improve quality, and the limited number of suppliers and qualified manufacturers in the domestic market. However, procurement savings enabled a greater number of schools than originally planned to be furnished. There were some minor shortcomings in the initial implementation period that may have affected efficiency, including a six-month delay in effectiveness, an internal restructuring of the Ministry of Education, longer than expected time to hire international technical assistance, extension of the project period by one year, and high turnover in preschool teachers. However, the project had a catalytic effect in mobilizing additional budget resources, as evidenced by the national roll-out of the new curricula and nation-wide implementation of the per-capita funding mechanism, using state budget resources. The effective coordination and sequencing of donor activities also likely contributed to efficiency. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) Page 8 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Global Partnership for Education (GPE)-4 (P131441) 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of the project objective is rated Substantial, due to high relevance to country conditions and the country education strategy, with less relevance to the Bank's current country strategy. Achievement of the objective to contribute to improving the learning conditions in preschool and general education is rated High due to evidence of outcomes in the number of preschool and primary/secondary school students enrolled in schools with improved learning conditions, some of which surpassed targets. Efficiency is rated Substantial due to favorable unit costs of project outputs compared with government and other donor-financed outputs. These ratings indicate only minor shortcomings in the project's preparation and implementation, and therefore Outcome is rated Satisfactory. a. Outcome Rating Satisfactory 7. Risk to Development Outcome The strong alignment of the project objectives and design with the Ministry's sector strategy will likely sustain government commitment. The embedding of project implementation arrangements within the Ministry has also helped to develop capacity and ensure sustainability of institutional learning, alongside explicit institutional strengthening activities in financial management and pedagogical leadership. As noted by the ICR (page 29), an ongoing risk is financial support for continually expanding initiatives and new investments in the education sector, given that most of the project activities (aside from those being implemented nationwide) are currently relying on donor support. Other non-donor-funded activities such as the ELCs are alternatively funded by parental contributions, which also leaves them financially vulnerable. Other key donors are continuing support to the education sector. 8. Assessment of Bank Performance a. Quality-at-Entry The theory of change was clear, with a well-defined project objective and results framework. Actual learning outcomes were not included in the results framework, as they were unlikely to materialize within the relatively brief three-year project period. The project built upon three prior Education For All (EFA) operations, as well Page 9 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Global Partnership for Education (GPE)-4 (P131441) as the work of other donors including UNICEF, USAID and Aga Khan Foundation. The complementarity of donor support was ensured through a proactive Local Education donor group. Lessons learned from prior experience, such as the importance of providing supplementary learning materials, ensuring the quality of teacher training, and the need for technical support for the per capita financing model at decentralized levels, were incorporated into the project design, while also introducing new areas of support such as ECE. The implementation arrangements were embedded in the Ministry of Education, thereby ensuring alignment with government strategy (and strong government commitment) and contributing to sustainability. The risk assessment, which rated the overall risk as Substantial due to inadequate implementation capacity, identified mitigation measures such as the use of the same fiduciary arrangements from the prior GPE operations, and an evaluation of ECE models prior to national roll-out. The M&E design is notable for the clarity in indicators and provision of technical assistance to ensure M&E implementation. Quality-at-Entry Rating Satisfactory b. Quality of supervision Project implementation was initially slow due to the following factors: a six-month delay in effectiveness; an internal restructuring of the Ministry of Education that eliminated the Education and Policy Analysis Unit responsible for project M&E; and longer than expected time to hire international technical assistance for upgrading the primary curriculum and to reach consensus with key stakeholders on the concept of the new competency-based curriculum. The latter issue also led to the decision to introduce the new primary curriculum with a one year academic delay, following implementation of the pilot. Supervision ratings were candid, reflecting these delays by downgrading the ratings until the issues were resolved. The project implementation period is notable for active donor coordination, given the need to sequence certain activities. This included donor participation in Bank supervision missions. The Bank team also sustained the strong focus on M&E, as evidenced by the robust data produced by the EMIS, and the overall contribution to developing a culture of M&E in the Ministry (ICR, page 28). Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The results framework was well considered. Measurable indicators were clearly linked to the project activities, with baseline and target figures. The ICR noted that there were limited outcome-oriented key Page 10 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Global Partnership for Education (GPE)-4 (P131441) project indicators, but justified this on the basis of the short implementation period and therefore limited time for longer-term outcomes to become measurable. M&E arrangements included project support to implementing monitoring activities and carrying out evaluations. The development of an EMIS was also a key aspect of project design, signaling the strong focus on establishing reliable data sources and a culture of decision making. b. M&E Implementation The project provided support to the unit within the Ministry of Education that was responsible for M&E. The ICR (page 24) reported that project monitoring reports were compiled and submitted in a timely manner. The EMIS modules developed through project support became functional and were producing data on the preschool and primary school levels. Evaluative activities were carried out as planned, including on teacher training and the impact of civil works activities. c. M&E Utilization The ICR (page 24-25) reported the following examples of the use of M&E: training of teachers and administrators was monitored through observation, and the results led to the replacement of trainers in certain cases; classroom observation by an independent team contributed to the design of a follow-up operation funded by the Russian government; the evaluation of key education indicators at schools benefitting from civil works confirmed the validity of the project's investment in civil works and provided reliable information and comparative analysis of behavior and trends of beneficiaries; and strengthened monitoring of civil works improved the quality of civil works and informed the design standards for subsequent operations. M&E Quality Rating High 10. Other Issues a. Safeguards The project was classified as an Environmental Category "B" project due to civil works and potential land acquisition. Safeguard policies on Environmental Assessment (OP/BP 4.01) and Involuntary Resettlement (OP/BP 4.12) were triggered. The Environmental Management Framework that had been in effect for the prior GPE project was updated to include the process for preparing site-specific Environmental Management Plans and a negative list of sites that would not be allowed as locations for new schools. Similarly, the Resettlement Policy Framework in effect for the prior project was to be used as a screening device for all activities involving temporary or permanent land acquisition. In the event of land acquisition, any affected persons would be consulted and informed, and Resettlement Action Plan (RAP) developed. Page 11 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Global Partnership for Education (GPE)-4 (P131441) A review of environmental safeguards during the Mid-Term Review found that civil works complied with the Bank's environmental requirements and produced no notable impact to the environment. There is no statement of compliance for the remaining project period following mid-term. The project team subsequently confirmed that compliance with the environmental safeguards was checked by the follow-up technical reviews. Regarding the safeguard on Involuntary Resettlement, the ICR (page 26) reported one incident in which a land user was relocated without the preparation of the RAP. After a retroactive review, the project team identified mitigation measures to bring compensation up to OP 4.12 standards and detailed the measures in an Abbreviated Resettlement Action Plan (ARAP). Another incident involved a construction site that resulted in the involuntary acquisition of land. As a response to the incident, the Bank advised the project team to halt any acquisition activities and to prepare an ARAP, while continuing to review closely any construction sites that could require land acquisition. While the matter for these two schools was being clarified, disbursement for expenditures concerning those sites was suspended. The ICR reported that the cases were resolved satisfactorily and disbursements resumed. b. Fiduciary Compliance Financial management (FM): Financial management arrangements included measures to mitigate potential risks, including updating the FM Operational Manual, recruiting a full time paid FM consultant to support the Head of the Department for Accounting and Financial Reporting, and signing a contract to upgrade the automated accounting system to be used for project accounting, budgeting, and reporting. The ICR (page 27) stated that financial reports and audit reports were submitted in a timely manner, the latter with no qualifications, and project payments were made on time . An early concern about the need to complete the adaptation process for the financial software and make it fully functional was addressed. The issues highlighted by the auditor’s management letter on the accounting and internal control systems were also addressed. Procurement: The project's fiduciary arrangements included measures to mitigate potential procurement risk, including that procurement packages would be consolidated to maximize interest from reputable bidders, advertising policy would be established, a third party quality assurance and control consultant would be hired to ensure quality of civil works, regular physical inspection missions would be conducted by the Bank, and the public disclosure and transparency provision of the Bank’s guidelines would be enforced. The Mid-Term Review found the project’s procurement processes and contract administration generally of good quality and performance of procurement staff at a satisfactory level. There were no major problems in procurement reported. c. Unintended impacts (Positive or Negative) Page 12 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Global Partnership for Education (GPE)-4 (P131441) None reported. d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory --- Bank Performance Satisfactory Satisfactory --- Quality of M&E High High --- Quality of ICR High --- 12. Lessons Lessons drawn from the ICR, adapted by IEG: • Early learning centers, or other forms of community-based early childhood education programs, can be a viable model for expanding preschool education enrollment. The use of existing facilities, topped up with preschool-focused materials and trained teachers, can lead to enrollment levels comparable to public preschools. • Incorporating M&E implementation support in the project design can be an effective way to demonstrate the value and use of M&E data in decision making. In the case of this project, the active utilization of M&E systems left in place a higher quality system but also a culture of M&E in the Ministry. • A global donor assistance program can provide an effective umbrella framework through which external donor support can be coordinated. In addition, the sustained external funding and technical assistance can lead to significant achievements as well as capacity development. In the case of this project, the careful definition of donor activities, and the regular meetings of the joint donor group, helped to ensure achievement of outcomes. 13. Assessment Recommended? Yes Page 13 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Global Partnership for Education (GPE)-4 (P131441) Please explain To verify the sustainability of the preschool interventions, the primary grade curriculum upgrades, and the sector management tools (per capita funding and the EMIS). Also, to assess whether learning outcomes have improved as a result of the improved learning conditions. 14. Comments on Quality of ICR The ICR quality was High. There was a substantial amount of evidence reported to verify achievements, alongside a clear analysis of the results chain. It was attentive to the project's theory of change throughout. The ICR was notable for its consideration of attribution issues, given the extent of donor involvement in the sector. The Efficiency section was notable for providing some quantitative analysis, even though the PAD did not do so. Lessons were well drawn from project experience. The assessment of safeguards performance did not provide information on environmental compliance for the entire project period (only for the period up until the Mid-Term Review). This project was a notable example of a strong focus on M&E in project design, both in developing monitoring systems and evaluating activities to understand their impact, which was then reflected in the quality of the ICR. a. Quality of ICR Rating High Page 14 of 14