Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SV Education Quality Improvement (P126364) Report Number: ICRR0021815 1. Project Data Project ID Project Name P126364 SV Education Quality Improvement Country Practice Area(Lead) El Salvador Education L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-81100,TF-10361,TF-14166 31-Dec-2017 59,716,220.99 Bank Approval Date Closing Date (Actual) 13-Dec-2011 31-Dec-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 60,000,000.00 228,680.00 Revised Commitment 59,716,220.99 128,817.40 Actual 59,716,220.99 128,817.40 Prepared by Reviewed by ICR Review Coordinator Group Judith Hahn Gaubatz Salim J. Habayeb Joy Behrens IEGHC (Unit 2) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives According to the Loan Agreement (page 4), the project objectives were as follows: Improve access, retention, and graduation rates for students in the Lower Secondary Education and Upper Secondary Education of the Borrower's public schools adopting the Inclusive Full Time School (IFTS) Model. The statement of objectives in the Project Appraisal Document (PAD, page 6) and the Implementation Completion and Results Report (ICR, page 7) were identical. Page 1 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SV Education Quality Improvement (P126364) Although the project objectives remained the same throughout the project period, key outcome targets were revised downward and therefore a split ratings approach is applied in this review. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? No c. Will a split evaluation be undertaken? Yes d. Components 1. Adoption of the IFTS Model (Appraisal: US$65.25 million; Actual: US$ 61.60 million): This component aimed to support implementation of five elements of the Inclusive Full-Time School (IFTS) Model, including teacher and school director training on the IFTS pedagogical model, provision of educational materials, rehabilitation and furnishing of educational facilities, teacher salary payments to cover extension of the school week from 30 to 50 hours per week, and supplemental funds for extracurricular activities, transportation, and minor rehabilitation. The project restructuring in July 2015 resulted in the following changes to the component: teacher training was shifted from direct implementation by the Ministry of Education to a cascade training-of-trainers model; teacher compensation and transportation modalities were revised; school feeding added; and use of extra time payments to teachers to expand the supply of upper secondary teachers. 2. Improvement of MINED's Institutional Capacity and the Schooling System's Governance (Appraisal: US$ 4.0 million; Actual: US$ 4.5 million): This component aimed to strengthen the Ministry of Education's (MINED) policy-making, planning, implementation, and monitoring capacity. Institutional support was provided in the following areas: teacher recruitment and evaluation, education standards, student assessment, monitoring and evaluation, communications to increase involvement of local communities, piloting a new governance model based on clusters of schools, and an impact evaluation. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost  The project cost was appraised at US$ 70.6 million, later revised to US$ 64.8 million. The actual cost was US$ 66.3 million. Financing Page 2 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SV Education Quality Improvement (P126364)  The project was financed by an IBRD Loan of US$60.0 million and a Trust Fund for Statistical Capacity Building grant of $130,000, both aggregating at US$60.1 million, out of which US$59.7 million were disbursed. Borrower contribution  The planned Borrower contribution was US$ 10.4 million, later revised to US$ 4.7 million. The actual Borrower contribution was US$ 6.6 million. Dates  December 2011: The project was approved by the Board.  May 2012: The project became effective.  January 2013: The project implementation arrangements were modified, due to an internal reorganization in the Ministry of Education. A separate project implementation unit was established, supported by project funds.  July 2015: The project was restructured to reflect changes in the project design and results framework. The number of school clusters targeted for full IFTS implementation was decreased from 101 to 40, although the remaining 61 clusters still benefitted from in-service teacher training, educational materials, minor civil works, and capacity building, in preparation for full IFTS implementation at a later stage. The key project indicators, which originally captured results only from lower secondary, were revised to capture results from both lower secondary and upper secondary. The targets for these indicators were also revised downward to reflect the narrowed scope of schools.  October 2016: The Mid-Term Review was conducted.  June 2017: The project closing date was extended from December 2017 to December 2018, due to implementation delays.  December 2018: The project closed. 3. Relevance of Objectives Rationale El Salvador was experiencing slow recovery from recent external shocks and sluggish economic growth, which had contributed to high levels of unemployment and insecurity, particularly among youth. This vulnerable segment of the population was continually confronting poverty and organized crime and violence, and therefore, improving the quality of education and increasing access to post-primary education were high priorities for the government. Coverage of primary education was high, but dropout rates accelerated before students finished lower secondary education (Grade 9). According to the PAD (page 1-2), of the students that start in grade 1, only 39 percent finished basic education, 28 percent enrolled in upper secondary education, 22 percent completed upper secondary education, and only 12 percent enrolled in tertiary education. Among the poorest quintiles, only 60 percent of 16 year olds and 20 percent of 18 year olds were attending school. Moreover, education outcomes in secondary education were poor, attributable to under-resourced small schools (lacking specialized teachers, equipped Page 3 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SV Education Quality Improvement (P126364) classrooms), two-shift classrooms that limit learning time, 25 hour teacher work week, and lack of teacher support. The government's ongoing education strategy included launching the Inclusive Full-Time School (IFTS) Model as the first step towards a more comprehensive reform of the secondary school system. This model was based on implementation of a similar model in other Latin American countries. The IFTS Model introduced pedagogical reform as well as a new school governance model through creating clusters of schools. The project thus responded to the need for improved access to and quality of post-primary education, particularly by supporting the government's reform efforts in implementing the IFTS model. The Bank's Country Partnership Framework for FY16-19 also identified attainment of secondary school education as a key objective, referring to poor educational outcomes and high rates of attrition as key concerns. The Country Partnership Framework indicators included improved transition rate between lower and upper secondary education and the number of school systems that adopt the IFTS model, both of which are included as indicators for this project. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Improve access rates for students in the Lower Secondary Education and Upper Secondary Education of the Borrower's public schools adopting the Inclusive Full Time School (IFTS) Model. Rationale The premise underlying this project was that improvements in school infrastructure, increased availability of education materials, extending the school day, training teachers and school directors, and providing transportation stipends -- all elements of the Inclusive Full Time School (IFTS) model – were expected to improve access to and retention in school, as well as to improve graduation rates. Support to strengthen MINED's institutional capacity was also likely to help achieve improvements in access, retention, and graduation rates. This theory of change was overall reasonable and plausible, although the scope of the project, in terms of number of schools and support for the full spectrum from access to lower secondary to graduation from upper secondary, was somewhat ambitious given the timeline and extent of changes in the educational approach. "Improved access" was defined as the number of students enrolled in 7th grade (lower secondary) or 10th grade (upper secondary) as a proportion of the total number of 6th grade and 9th grade students, respectively, promoted the previous year. The calculation of the indicator was later slightly modified to align with UNESCO's definition of effective transition rate. Page 4 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SV Education Quality Improvement (P126364) Outputs In the original project design, 29 municipalities were selected based on needs and feasibility. Within these municipalities, 270 schools that offer lower and upper secondary education were reorganized into 201 schools that would implement the full IFTS model. An estimated 34,017 lower secondary and 7,038 upper secondary students attending these schools were expected to directly benefit. During actual implementation, 163 schools fully adopted the IFTS Model (revised target: 213), whereby at least four of the defined elements of the IFTS Model were met. These schools were then grouped into 101 "school clusters," as part of a new approach to governance and consolidation of resources. As mentioned above (section 2, Dates), at the July 2015 restructuring, the number of school clusters targeted for full IFTS implementation was decreased from 101 to 40, although the remaining 61 clusters still benefited from some services.  Extensive renovation of 36 "full" IFTS schools with fully integrated system infrastructure. In addition, "part" IFTS schools (schools that implemented the IFTS model in part, but not fully) improvements included: rehabilitation/construction of 718 auxiliary structures such as kitchens, libraries, bathrooms; furnishing of 441 schools, and minor repairs in 230 schools. However, this activity was not fully completed until close to the project end.  Provision of education materials to 195 schools, including computers, projectors, library books, sports equipment, and art supplies.  Financing of extra time payments to teachers, which resulted in additional hours of class time for lower secondary education. The number of students in 7-9th grade that have 30 or more weekly hours of pedagogical activities increased from 4,410 at the first year of project period to 9,842 by project closing.  Provision of transportation stipends. The number of students receiving transportation stipends increased from 4,161 at the first year to 9,759 by project closing.  Training and certification of 1150 master trainers and 1,169 teachers on pedagogical reforms.  Training and certification of 436 school directors on the IFTS model.  Design and implementation of a communications strategy for the IFTS model, including creation of communication portals for each school cluster to share achievements and provide information updates.  Design of information technology platform to support MINED's management of its education programs. Outcomes *Note: After the July 2015 restructuring, when the project scope was reduced from targeting 101 school clusters for full IFTS implementation to targeting 40 school clusters, baselines for key indicators were revised to include only the 40 school clusters; in this revision, the revised baseline figures were given for 2015, rather than 2011. Keeping in mind that the Bank’s split rating methodology calls for achievements across the entire project period to be compared to the original targets as well as to the revised targets, a baseline figure from the project start in 2011 for the 40 school clusters (as a sub-set of the original 101 school clusters) would have enabled a more accurate assessment of achievement of objectives under the revised targets. In assessing the achievement of objectives, therefore, this ICR review looks to the results for both sets of indicators reported in the ICR. Page 5 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SV Education Quality Improvement (P126364)  The access rate for lower secondary education for the original scope of 101 school clusters decreased from 96.3% in 2011 to 95.1% in 2017, falling short of the target of 98.8%. The access rate for the revised scope of 40 school clusters marginally increased from 96.6% in 2015 to 96.7% in 2017.  The access rate for upper secondary education for the revised scope of 40 school clusters increased from 78.0% in 2015 to 87% in 2017, surpassing the target of 78.0%. Achievement is rated Negligible due to a decrease in access rate for lower secondary education for the original scope of 101 school clusters. Rating Negligible OBJECTIVE 1 REVISION 1 Revised Objective Improve access rates for students in the Lower Secondary Education and Upper Secondary Education of the Borrower's public schools adopting the Inclusive Full Time School (IFTS) Model. (REVISED TARGET) Revised Rationale The same theory of change described above under the original Objective 1 also applies here. Outputs See outputs reported above, under Objective 1. Outcomes See "Note" above under Objective 1 Outcomes.  The access rate for lower secondary education marginally increased from 96.6% in 2015 to 96.7% in 2017, thus achieving the revised target of 96.6%. Due to the declining trend of this indicator at the national level (as stated in the ICR (page 11) although no specific data are cited), maintaining the baseline level was assessed as a positive development.  The access rate for upper secondary education increased from 78.0% in 2015 to 87.0% in 2017, surpassing the target of 78.0%. An important element of context in interpreting achievement of this objective is that infrastructure development -- one IFTI elements envisioned to contribute to improved access – was not completed until the end of the project. This timing weakens the case for attribution or contribution of the project’s activities to the outcomes observed. However, even with this caveat in mind, achievement is rated Substantial based on the outcomes noted above. Revised Rating Page 6 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SV Education Quality Improvement (P126364) Substantial OBJECTIVE 2 Objective Improve retention rates for students in the Lower Secondary Education and Upper Secondary Education of the Borrower's public schools adopting the Inclusive Full Time School (IFTS) Model. Rationale The theory of change described above under the original Objective 1 also applies here. "Improved retention" was defined as the proportion of students in grades 7-9 (lower secondary) and grades 10-11 (upper secondary) that remain in school for the entire academic year. Outputs See outputs reported above under Objective 1. Outcomes See "Note" above under Objective 1 Outcomes.  The retention rate for lower secondary education for the original scope of 101 school clusters declined from 95.1% in 2011 to 93.7% in 2017, falling short of the target of 98.4%. The retention rate for the revised scope of 40 school clusters increased from 91.0% in 2015 to 93.8% in 2017, achieving the target to maintain the rate at 91.0%.  The retention rate for upper secondary education marginally increased from a baseline of 92.5% in 2015 to 92.8% by 2017, achieving the target to maintain the rate at 92.5%. Achievement is rated Negligible due to a decrease in retention rate for lower secondary education for the original scope of 101 school clusters. Rating Negligible OBJECTIVE 2 REVISION 1 Revised Objective Improve retention rates for students in the Lower Secondary Education and Upper Secondary Education of the Borrower's public schools adopting the Inclusive Full Time School (IFTS) Model. (REVISED TARGET) Revised Rationale The same rationale described above under the original Objective 1 also applies here. Page 7 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SV Education Quality Improvement (P126364) Outputs See outputs reported above under Objective 1. Outcomes See "Note" above under Objective 1 Outcomes.  The retention rate for lower secondary education for the revised scope of 40 school clusters increased from a baseline of 91% in 2015 to 93.8% in 2017, achieving the revised target to maintain the rate at 91%. This translated into 529 additional children per year that remain in the school system. The ICR (page 13) suggested that this achievement in lower secondary can be attributable to the project, as the impact evaluation data indicated that the project schools (sub-sample of 174 schools) experienced a decrease in dropout rate of 5.8%, compared to 8.1% in control schools.  The retention rate for upper secondary education marginally increased from a baseline of 92.5% in 2015 to 92.8% by 2017, achieving the target to maintain the rate at 92.5%. Achievement is rated Substantial based on increased retention rates for lower secondary and upper secondary. Revised Rating Substantial OBJECTIVE 3 Objective Improve graduation rates for students in the Lower Secondary Education and Upper Secondary Education of the Borrower's public schools adopting the Inclusive Full Time School (IFTS) Model. Rationale The same rationale described above under the original Objective 1 also applies here. "Improved graduation rates" was defined as the proportion of 11th grade students that graduate at the end of school year. Outputs See outputs reported above under Objective 1. Outcomes See "Note" above under Objective 1 Outcomes. Page 8 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SV Education Quality Improvement (P126364)  The graduation rate for lower secondary education for the revised scope of 40 school clusters increased from 69.0% in 2015 to 71.4% in 2017, achieving the target to maintain the rate at 69%. This translated into 597 additional children per year that successfully completed their school year. The findings from the impact evaluation also indicated that the project contributed to graduation rates in lower secondary in project schools (sub-sample of 174 schools) of 90.2%, compared to the graduation rate in non-project schools of 87.7%.  The graduation rate for upper secondary education for the original scope of 101 school clusters decreased from 75.0% in 2011 to 73.8% in 2017, falling far short of the target of 91.7%. Achievement is rated Negligible due to a decrease in graduation rate for upper secondary education for the original scope of 101 school clusters. Rating Negligible OBJECTIVE 3 REVISION 1 Revised Objective Improve graduation rates for students in the Lower Secondary Education and Upper Secondary Education of the Borrower's public schools adopting the Inclusive Full Time School (IFTS) Model. (REVISED TARGET) Revised Rationale Outputs See outputs reported above under Objective 1. Outcomes See "Note" above under Objective 1 Outcomes.  The graduation rate for lower secondary education increased from 69.0% in 2015 to 71.4% in 2017, achieving the target to maintain the rate at 69%. This translated into 597 additional children per year that successfully completed their school year. The findings from the impact evaluation also indicated that the project contributed to graduation rates in lower secondary in project schools (sub- sample of 174 schools) of 90.2%, compared to the graduation rate in control schools of 87.7%.  The graduation rate for upper secondary education for the revised scope of 40 school clusters increased from 71.5% in 2015 to 75.9% by 2017, surpassing the target to maintain the rate at 71.5%. There were inconsistencies with findings of an experimental impact evaluation which found that graduation rates for upper secondary in project schools were lower than in control schools, although M&E Section 9 below discusses the questionable quality of the impact evaluation. Achievement is rated Substantial due to evidence of maintained graduation rates for lower and upper secondary education. Page 9 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SV Education Quality Improvement (P126364) Revised Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale Overall efficacy under the original targets is rated Negligible due to decreases in access and retention rates in lower secondary and decrease in graduation rate in upper secondary for the original scope of 101 school clusters. Overall Efficacy Rating Primary Reason Negligible Low achievement OBJR1_TBL OVERALL EFFICACY REVISION 1 Overall Efficacy Revision 1 Rationale Overall efficacy under the revised targets is rated Substantial due to three almost fully achieved objectives. Overall Efficacy Revision 1 Rating Substantial 5. Efficiency The economic analysis at appraisal (PAD, Annex 6) provided an internal rate of return estimate and a cost benefit analysis for the project. Costs were calculated as the total project costs. Benefits were calculated according to the benefits accrued to the students who stayed in the school system, rather than dropping out, thereby completing lower and/or upper secondary education and receiving higher wages. The internal rate of return was estimated at 13% for the low case scenario, and the net present value at US$ 8.4 million, again under the most conservative salary differential estimate. The economic analysis in the ICR (Annex 4) more narrowly focused on the interventions in Component 1 (implementation of IFTS model) and did not include Ministry-level capacity building activities in Component 2 (according to the ICR (Annex 4), this was due to the difficulty in measuring the economic benefits of the latter interventions). Benefits were similarly calculated according to additional years of education that lead to higher productivity and wages. Costs were calculated as the project costs for Component 1 and recurrent costs for maintaining school infrastructure. As the interventions were implemented at different points in time over the project period and to a varying extent by the participating schools, beneficiaries were grouped according to the interventions by year. For example, educational materials and minor rehabilitation were implemented early on, but teacher training and renovations did not take place until year four. Also, some schools only implemented one or two elements of the IFTS Model, while others implemented all elements. These beneficiary groups thus Page 10 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SV Education Quality Improvement (P126364) ranged from 5,899 students who benefitted from all elements in 2018, to 41,068 students who benefitted from teacher training only. The estimated impact - the additional years of education gained through an intervention - was also noted according to the specific element of the IFTS model. Based on the above, the net present value was calculated as US$64.1 million and the internal rate of return as 8.9%. A sensitivity analysis was also conducted. There were moderate shortcomings in the efficiency of implementation due to implementation delays stemming from Presidential elections that led to leadership changes in the Ministry of Education, the Ministry's change in strategic vision, security issues, land titling issues, changes in specifications for infrastructure, long turnaround times in procurement, and fiscal constraints and bureaucracy that led to delays in providing counterpart funds. Based on favorable returns, but with moderate shortcomings in the efficiency of implementation, efficiency is rated Substantial. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 100.00 Appraisal  13.00  Not Applicable 92.90 ICR Estimate  8.90  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of the project objectives is rated High due to strong alignment with government strategy and Bank strategy. Efficacy is rated Negligible under the original objectives based on decreases in access and retention rates in lower secondary and decrease in graduation rate in upper secondary for the original scope of 101 school clusters.. Efficacy is rated Substantial under the revised objectives due to evidence demonstrating that outcomes were almost fully achieved in access, retention and graduation rates. Efficiency is rated Substantial based on favorable returns, but with moderate shortcomings in the efficiency of implementation, Project under original targets Overall outcome is rated Unsatisfactory due to High Relevance of Objectives, Negligible Efficacy, and Substantial Efficiency. Page 11 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SV Education Quality Improvement (P126364) Project under revised targets Overall outcome is rated Satisfactory due High Relevance of Objectives, Substantial Efficacy, and Substantial Efficiency. According to harmonized OPCS/IEG guidelines on restructured projects, the overall outcome is weighted according to the amount of the Loan that disbursed before and after the restructuring. At the time of restructuring in July 2015, US$13.86 million, or 20.9%, of the Loan had disbursed. Therefore, the original project numerical rating is 0.418 (0.209 x 2 (Unsatisfactory)) and the revised project numerical rating is 3.955 (0.791 x 5 (Satisfactory)), therefore the combined rating is 4.373, which rounds down to Moderately Satisfactory. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome Financing was received from the Millennium Challenge Corporation to further expand the IFTS model to an additional 45 clusters (349 schools), which is providing not only financial support but continuing critical institutional capacity development. Despite a change in presidential leadership, the IFTS model remains a high development priority for the government. However, there was uncertainty about whether students successfully transitioning from lower secondary to upper secondary were provided adequate sustained support through to graduation. 8. Assessment of Bank Performance a. Quality-at-Entry The project objectives and design were strongly aligned with country conditions and strategy at project appraisal. The approach to the project design - to draw upon success with the IFTS Model in other Latin American countries and pilot the Model in El Salvador with the aim to scale up nationally - was appropriate given the extensive reforms in the school system introduced by this Model. Also drawing upon experience with prior Bank education operations in the country, in which several prior projects failed to reach effectiveness due to insufficient political support, the project team prepared the project quickly (six months from concept to approval) in order to capitalize on the existing high level of support from the then-Minister of Education. However, the quick preparation also set the stage for some shortcomings in project readiness, as a thorough on-the-ground assessment of the infrastructure needs of schools did not take place, leading to an underestimation of renovation requirements and costs for this activity. The risk of violence (and its effect on infrastructure development and access to schools) was appropriately recognized as a critical risk, though it is unclear whether more effective mitigation measures could have been identified given the exogenous nature of the risk. The M&E design was outcome-oriented (as Page 12 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SV Education Quality Improvement (P126364) reflected in the key indicators that measured actual outcomes rather than outputs), although there were some shortcomings in the choice of indicators (see Section 9). Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision Given the pilot nature of the project approach, the Bank team paid close attention to learning lessons from implementation during the early project period, including both strategic issues and operational bottlenecks: shifting from MINED-led teacher training to a cascade model, modifying the teacher compensation and transportation stipend modalities, and adding school feeding and teacher extra time payments. Project restructurings were well-utilized to reflect these changes, including in the results framework. There were some implementation delays due to the following issues: change in leadership at MINED, delays in confirmation of land titles, revisions to infrastructure plans due to underestimation of needs, inefficiencies in procurements, and delays in provision of counterpart financing for certain activities due to national fiscal constraints and bureaucracy. The project was also affected by a sharp increase in violence and crime, due to the ending of the gang ceasefire agreement. Of note, the Bank team placed a strong emphasis on effective M&E, including raising additional funds to complement existing M&E data and provide additional evaluative assessments. This included a grant from the Spanish Fund for Latin America and the Caribbean (SFLAC), executed by MINED, to finance five studies that would inform implementation; a Trust Fund Statistical Capacity Building grant to develop educational indicators and build capacity. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The objectives were clearly stated. The original M&E design had a clear outcome-orientation with both rigorous monitoring arrangements and an evaluative component. Baseline data was established at the project outset, with appropriate revisions during project restructuring. However, there were some shortcomings in the original results framework. Although the project objectives encompassed both lower secondary and upper secondary education, the key project indicators were not disaggregated to reflect both sub-sectors. Targets were also later deemed unrealistic, due to actual declining trends for those indicators at the national level due to macro factors such as deteriorating economic conditions, migration and increased crime and violence. The methodology to calculate access rate was also later redefined to align it with the UNESCO definition of effective "transition" rate. Page 13 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SV Education Quality Improvement (P126364) The M&E design also included an evaluative assessment, through an experimental impact evaluation; however, capacity to undertake this evaluation was overestimated and the robustness of the findings to fully inform project performance, given it was not a full-fledged impact evaluation, is questionable. Also, an institutional culture that did not give importance to rigorous measurement of impact contributed to limited ownership of this evaluation from MINED. The ICR (page 19) suggested that an evaluation of the pilot program through a rigorous process evaluation may have been a more appropriate approach. b. M&E Implementation M&E data collection was effectively supervised by the project team to emphasize the importance of reliable and good quality data. As reported above, the results framework was modified during the project period to more accurately reflect project achievements and ensure realistic targets. Although the full experimental impact evaluation could not be carried out as planned, the project team was able to use the data collected to conduct a semi-experimental evaluation to measure the impact of the individual/bundled activities under the IFTS model, comparing the data of a set of control schools using propensity score matching. The ICR (page 14) suggested that the poor outcomes in upper secondary graduation may have been driven by the fact that once more students graduated from lower secondary due to project support, some of the students that moved on to upper secondary were unable to keep up with the requirements and therefore did not graduate. However, the quality of the experimental impact evaluation is questionable given the lack of comprehensiveness of the study. The Bank team also secured additional funds to conduct studies to supplement M&E data, including (i) an assessment of the reform approach to improving governance and accountability; (ii) a proposal of standards of efficiency in school management, student learning, and governance; (iii) a proposal to create a control panel for school management; (iv) a communication strategy; and (v) a study on the implementation of providing lunch to students in the IFTS Model. There was also a phone survey to directors from the project schools to understand the reasons for the low student take-up of the extended school day. However, the activity to implement a student-level information system was not completed. c. M&E Utilization The ICR reported the following examples of the use of M&E: the project's impact evaluation contributed to dialogue with MINED to raise awareness about the importance of evaluating the project rather than only monitoring results; the Trust Fund for Statistical Capacity Building funds were used to develop educational indicators and raise awareness about the importance of evaluating education policies; the results of the telephone survey led to the revision of the content taught during extra-curricular hours. M&E Quality Rating Substantial 10. Other Issues a. Safeguards Page 14 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SV Education Quality Improvement (P126364) The project was classified as an Environmental Category "B" project due to minor civil works. With the project restructuring in July 2015 that led to more in-depth infrastructure activities, the potential environmental impact increased but there was not an increase in environmental monitoring. The Bank team and MINED subsequently agreed on the use of an environmental monitoring tool to ensure closer supervision and developed a Project Management Plan for 19 of the 36 construction works to address issues. By the end of the project period, the environmental safeguards performance was considered satisfactory, with all 36 infrastructure works having been environmentally certified. The Bank team also provided training to the MINED staff and the construction firms on environmental monitoring. The project design also triggered OPBP 4.10 on Indigenous Peoples and therefore an Indigenous Peoples Plan was developed. Implementation of the Plan had some initial delays due to difficulties in hiring an organization to implement the Plan. However, once implemented, the Project supported MINED to develop a program on the indigenous Náhuat language and identity, an activity not originally considered in the Project’s design. The ICR noted that this was a significant achievement because it marked the first time in El Salvador that MINED offered an institutionalized teacher training on "Náhuat language and cultural identity," thus helping to preserve a dying language. b. Fiduciary Compliance Financial management: The ICR reported that the project was in compliance with financial management requirements. Audit reports were submitted in a timely manner, with no major qualifications, although there were some issues with internal controls including documentation of transfers and some ineligible expenditures. These issues were resolved with support from the Bank team. Procurement: The Bank team provided training to MINED's procurement staff and close supervision to ensure quality control. The ICR reported that procurement arrangements and performance were adequate throughout most of the implementation period and that efficiency in the procurement processes improved significantly over time, reducing the internal turnaround times for each step. No major procurement problems were reported. c. Unintended impacts (Positive or Negative) According to the ICR (p. 18), and as discussed above in Section 10a, the project supported the development of a program on Náhuat language and cultural identity, and institutionalized teacher training, thus contributing to preserving a language that was disappearing. d. Other -- Page 15 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SV Education Quality Improvement (P126364) 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory Satisfactory Moderately Bank Performance Moderately Satisfactory Satisfactory Quality of M&E Substantial Substantial Quality of ICR --- Substantial 12. Lessons The ICR (pg. 23) provided several lessons, including the following:  There is trade-off between speed of preparation and technical soundness of project design. For this Project, a swift preparation was required to take advantage of the political window of opportunity. This short preparation timeline, however, did not allow for the use of the capacity-building trust funds to inform project design nor for a full diagnostic of the infrastructure needs of beneficiary schools, which led to an underestimation of the costs and the depth of the interventions needed. The following lessons were also identified by IEG:  M&E capacity as well as an institutional culture that values rigorous assessment is critical to effective M&E implementation. In the case of this project, the country’s capacity to implement an experimental impact evaluation was overestimated. As reported in the ICR (page 19), although MINED was able to collect data for monitoring purposes, "the institutional culture did not give importance to rigorous measurement of impact" and thus there was limited ownership from MINED. Despite several trainings and intensive support for the impact evaluation’s design and implementation, the data collected was not sufficiently robust to be used as intended.  Continuous support to upper secondary students from initial access/transition from lower secondary all the way through to graduation greatly facilitates the objective to improve graduation rates. In the case of this project, project activities were largely focused on lower secondary education, in order to increase likelihood of lower secondary students continuing on to upper secondary, where sustained support is also needed. 13. Assessment Recommended? No Page 16 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SV Education Quality Improvement (P126364) 14. Comments on Quality of ICR The ICR was concise and results-oriented. The theory of change was sound and aligned to development objectives. However, the Efficacy ratings were not correctly determined under the split evaluation, where the achievements for both original and revised objectives (with revised targets) should be measured over the whole project period. The quality of the evidence and analysis was notable for triangulating findings from both project monitoring systems and supplemental evaluations, although there were some inconsistencies in the findings that could not be explained by the ICR team without further investigation. The ICR economic analysis was also notable in that it improved substantially upon the more general analysis conducted for project appraisal, including pinpointing the numbers of students that benefitted from specific interventions during the specific phases of the project period, and the varying degrees to which each participating school implemented the IFTS model. The Lessons section was somewhat generic, missing an opportunity to draw specific lessons to inform future operations on the provision of secondary education. a. Quality of ICR Rating Substantial Page 17 of 17