Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT Education for All Project - Phase II (P124134) Report Number : ICRR0021555 1. Project Data Project ID Project Name P124134 HT Education for All Project - Phase II Country Practice Area(Lead) Additional Financing Haiti Education P132756,P147608 L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-H7400,TF-17666,TF-17830 30-Jun-2015 103,209,542.42 Bank Approval Date Closing Date (Actual) 01-Dec-2011 30-Jun-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 70,000,000.00 38,850,000.00 Revised Commitment 108,848,058.05 38,850,000.00 Actual 103,209,542.42 38,850,000.00 Prepared by Reviewed by ICR Review Coordinator Group Judith Hahn Gaubatz Judyth L. Twigg Eduardo Fernandez IEGHC (Unit 2) Maldonado 2. Project Objectives and Components a. Objectives According to the Project Appraisal Document (PAD, page 5) and the Financing Agreement (page 4), the project objective was as follows: to support the Strategy for Rebuilding the Education System through implementation of sustainable programs to: • Improve access, particularly of underserved populations, to primary education; • Improve quality of primary education; and Page 1 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT Education for All Project - Phase II (P124134) • Improve institutional capacity in the Recipient’s education sector. In June 2014, according to the Project Paper (June 2014, page 14), the project objectives were revised as follows: • To support enrollment of students in select non-public primary schools in disadvantaged areas; • To support student attendance in select public and non-public primary schools in disadvantaged areas; and • To support strengthened management of the Recipient’s primary education sector. The project was also restructured in October 2014; this entailed Additional Financing to scale up successful interventions such that targets were revised upward, but neither the project design nor objectives were materially changed. A split rating for that restructuring is therefore not applicable, but is applied for the June 2014 restructuring due to the change in project objectives. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 04-Jun-2014 PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? Yes d. Components This operation was the second in a planned three-phase operation. 1. Improving Access to Quality Primary Education (Appraisal: US$ 45.0 million; Actual: US$ 55.6 million): This component aimed to address barriers to access for poor families. Activities included: financing of tuition waivers for families in underserved communities; provision of grants to rural communities to establish schools; financing of community school teacher salaries; and provision of school meals, de- worming, and Vitamin A supplements (note: this activity was later moved to Component 2). 2. Support to Teaching and Learning (Appraisal: US$ 10.0 million; Actual: US$ 33.9 million): This component aimed to improve quality of teachers and curriculum in primary schools. Activities included: intensive pre-service teacher training; development of teacher's guides; and early grade reading programs. Page 2 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT Education for All Project - Phase II (P124134) 3. Institutional Strengthening and Governance (Appraisal: US$ 4.0 million; Actual: US$ 1.2 million): This component aimed to support the Ministry of Education to manage the primary education sector. Activities included: modernizing systems; introducing program-based budgeting; and establishment of the Office for Public Private Partnerships (ONAPE). 4. Project Management, Monitoring and Evaluation (Appraisal: US$ 6.0 million; Actual: US$ 12.3 million): This component aimed to support project implementation. Activities included: financing for project management; support for monitoring; and financing of evaluation studies. Due to multiple project restructurings, the final project components were as follows, with changes noted per component: 1. Support to Primary Education Enrollment: Reduction in scope of community schools activity, addition of reconstruction activities following Hurricane Matthew, and shifting of school health and nutrition activities to Component 2. 2. Support to Improved Student Attendance: Revision to the content of the early grade reading program. 3. Strengthening Sector Management: Dropping of the ONAPE and program-based budgeting, and addition of new school accreditation activities. 4. Project Management, Monitoring and Evaluation: No significant changes. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost • The appraised project cost was US$ 70.0 million. Due to additional funding approved for the project on two occasions, the actual project cost was US$ 103.2 million. • Project funds in the amount of US$ 3.5 million were shifted in May 2017, to support emergency reconstruction needs following Hurricane Matthew, Financing •The original project was financed by a Global Partnership for Education (GPE) grant of US$ 70.0 million. Additional financing of US$ 14.75 million was approved as a grant from the Haiti Reconstruction Fund, followed by another grant of US$ 24.1 million from GPE. Borrower contribution • There was no planned Borrower contribution. Page 3 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT Education for All Project - Phase II (P124134) Dates • June 2014: Additional Financing was approved in the amount of US$ 14.75 million from the Haiti Reconstruction Fund to continue the school health and nutrition activities. The project was also restructured such that the project objectives and key indicators/targets were revised. The safeguard policy on Involuntary Resettlement (OP 4.12) was triggered. The project closing date was extended from June 2015 to September 2016. • October 2014: A second Additional Financing was approved in the amount of US$ 24.1 million from GPE to scale up activities. Targets for key indicators were revised upward accordingly. The project closing date was extended from September 2016 to June 2017. • May 2015: The results framework was modified to account for changes in project design; no key outcome indicators or targets were revised. • August 2016: The closing date of the first Additional Financing grant (Haiti Reconstruction Fund) was extended to align with the parent project closing date. • May 2017: Following Hurricane Matthew in October 2016, the project was restructured to shift project funds to emergency reconstruction needs. The project closing date was extended from June 2017 to June 2018. 3. Relevance of Objectives Rationale Haiti is among the poorest and most fragile countries in the Latin America and Caribbean region, with external shocks such as global economic crises and natural disasters having devastating impacts on a population with already low human capital development (ranked 148 out of 172 countries on the Human Development Index at the time of project appraisal). The education sector is faced with shortages of adequate school infrastructure and trained teachers, as well as a lack of effective governance. The cost of education, given the low availability of public schools, is also a prohibitive barrier for poor families. Substantial investment in the social sectors is therefore critical to address these poverty and fragility issues, but public sector capacity is very limited. The government's education sector strategy at the time of appraisal was reflected in its Operational Plan for Education, which included as priorities reinforcing sector governance, improving programs and curricula, improving education sector staff training, and supporting early childhood and basic education sectors. Universal primary education, in particular, was and remains a key priority for the government. The Bank's interim strategy note (2012) at the time of appraisal identified building human capital as a key strategic objective, including increasing access to schools. The Bank's strategy at closing, the Country Page 4 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT Education for All Project - Phase II (P124134) Partnership Framework for FY2016-19, retained this explicit focus. Therefore, two of the original project objectives to improve access and institutional capacity were strongly relevant, but as the project period progressed, the third original objective to improve quality was less relevant given the unexpected need to respond to natural disasters, as nearly 1,000 primary schools were affected by Hurricane Matthew in 2016. The revised objectives were relevant to aforementioned country conditions and strategies. Rating Substantial 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective Implementation of sustainable programs to improve access, particularly of underserved populations, to primary education Rationale The theory of change for this objective was overall sound, as the planned interventions were intended to overcome barriers for poor families (financial support for tuition and availability of classrooms) and to increase demand (provision of school health and nutrition programs). Selection of schools was based on poverty mapping, which identified disadvantaged areas/underserved communities, alongside some degree of self-targeting among beneficiary schools, as the small tuition grant size would likely be attractive only to schools charging low tuition. Sustainability of programs was also an overarching aspect of the three original project objectives (including establishing a financing plan and process of graduation from full subsidization of the tuition waiver program for participating schools). Outputs • Provision of 482,932 tuition waivers, cumulative over the project period, through the Tuition Waiver Assistance Program (TWP) (target: 482,000). This represented the total number of waivers provided over the project period (including for multiple years of schooling per student) rather than the total number of students supported in the program. The original indicator measured the number of beneficiaries, but it was later determined that existing monitoring systems did not have the capacity to collect such data. • Provision of 465,897 school health and nutrition interventions, cumulative over the project period (target: 464,000), benefiting 93,538 children (original target: 70,000; revised target: 92,000) . An additional 15,467 students benefitted from the interventions in the aftermath of Hurricane Matthew. Page 5 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT Education for All Project - Phase II (P124134) • Provision of grants to establish 57 community-based schools (original target: 200) and construct 21 classrooms (original target: 700). Due to the following factors, financing for this activity was significantly reduced: higher than anticipated unit costs of school construction, low implementation capacity, shift in project financing to emergency reconstruction needs, and shift in financing from construction of community schools to community teacher salaries (due to inability of government to assume the cost of teacher salaries, as planned). • Provision of furniture for 24 schools and 18,000 school kits. • Rehabilitation of 91 schools (including to repair or replace roofs or build semi-permanent shelters), following the destruction caused by Hurricane Matthew. Outcomes • The initial year of enrollment in TWP schools was 80,000 students, with the peak year enrollment of 135,000 students in 2012/13; however, this steadily decreased each year afterwards such that in 2016/17 there were 45,027 students enrolled. As reported in the ICR (page 20), in part due to the inability of the government and other donors to maintain financing of the TWP after the last project cohort of grade 1 students enrolled in 2012/13, the number of students supported annually decreased. In addition, the legal status of the National Fund for Education, which was created to sustain financing for the waivers, was not approved until 2017, and therefore there were no alternative sources of funding to support participation of additional cohorts. • The ICR reported findings from an impact evaluation of the TWP conducted in 2015, which indicated that TWP schools in the 2011/12 school year had higher enrollment compared to a control group (although actual figures were not reported in the ICR, the project team provided data showing the differences in enrollment in "treatment" (project) schools vs. "control" (non-project) schools)), and that the national net enrollment rate at the primary level increased from 50-60% in the early 2000s (pre-TWP) to 70-80% in 2012. However, the study concluded with the 2011/12 school year, and the findings cannot be extrapolated to the remainder of the project period to verify that enrollment gains were sustained; also, it is not clear what proportion of the national net enrollment figures were represented by TWP schools. Focus group findings indicated that 59% of parents of TWP beneficiaries reported that without the program, some of their children would not have been enrolled in grade 1. • The community-based schools reached enrollment of 6,500 students per year (no target was established for this outcome). In addition, • Efforts to establish a financing framework or plan for the government to take up recurrent sector activities did not lead to any lasting agreements or adoption of plans by the government, thus diminishing Page 6 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT Education for All Project - Phase II (P124134) sustainability of achievements. A 2014 shift in priorities favored financing mainly public schools for tuition waivers, rather than nonpublic schools. Provision of tuition waivers achieved targets and likely contributed to increased access; however, enrollment data showed peak enrollment in 2012/13 and declining enrollment thereafter, and there were also significant shortfalls in outputs for school and classroom construction. Therefore, achievement is rated Modest. Rating Modest PHREVDELTBL PHINNERREVISEDTBL Objective 1 Revision 1 Revised Objective To support enrollment of students in select non-public primary schools in disadvantaged areas Revised Rationale Outputs See outputs reported above. Outcomes See outcomes reported above. The objective was vaguely worded (to "support" enrollment) and no clear outcome was identified. The provision of tuition waivers achieved targets and likely supported enrollment in primary education, but there was a lack of clear data on the number of children benefitting from the intervention, and enrollments declined after 2012/2013. Revised Rating Modest PHEFFICACYTBL Objective 2 Objective Implementation of sustainable programs to improve quality of primary education Rationale Page 7 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT Education for All Project - Phase II (P124134) The theory of change for this objective was overall sound, as the planned interventions (intensive teacher training and early grade curriculum revisions) were likely to contribute to the outcome of improved quality of primary education. Outputs • Provision of intensive pre-service training to 3,570 teacher trainees (target: 3,300). The ICR (page 14) noted that despite certifying 2,600 of these trainees, the government was limited in its ability to replace unqualified teachers in public schools or require non-public schools to hire them. Therefore, the actual impact on the quality of education and the sustainability of any quality improvements is unclear. • Introduction of M’ap Li Net Ale early grade reading program, reaching 26,718 students in 299 schools (no target set). • Introduction of pilot program to use smartphones to take photos of teachers in schools to monitor teacher attendance. The ICR reported that implementation delays and technical difficulties hindered smooth functioning of the pilot. In addition, • Distance education policies and technology activities were not implemented as planned. • Teacher guides were not developed as planned, as the United States Agency for International Development provided support in this area. Outcomes • According to an April 2018 phone survey of teacher training graduates, of those who were employed, 92% were working as teachers, 74% had worked as teachers immediately after the completion of training, and 57% reported that the training played a role in helping to obtain the teaching job. In addition, 67% of all survey respondents were "satisfied" with the training program and 29% were "somewhat satisfied." However, 43% of respondents were unemployed at the time of the survey, and there was no evidence of actual improved quality of teaching. • An impact evaluation of the M’ap Li Net Ale early grade reading program indicated an improvement in foundational reading skills as measured by the Early Grade Reading Assessment (EGRA) tool. Grade 1 and 2 students increased their knowledge of sounds associated with letters by 64%, the number of familiar words read correctly in one minute by 22%; and their phonemic awareness by 136%. However, Page 8 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT Education for All Project - Phase II (P124134) • The original key indicator on the percentage of children reading at grade level in grade 3 was dropped due to non-availability of data. • The pilot program to monitor teacher attendance with smart phones did not reduce teacher absenteeism, according to the ICR (no actual figures reported). Although there was some evidence of improved learning outcomes in early grades, the number of students impacted was small. Also, outcomes in teacher quality were modest. Therefore, achievement of this objective is rated Modest. Rating Modest PHREVDELTBL PHINNERREVISEDTBL Objective 2 Revision 1 Revised Objective Original objective 2 was dropped, with no revised objective 2. Revised Rationale n/a Revised Rating Not Rated/Not Applicable PHEFFICACYTBL Objective 3 Objective Implementation of sustainable programs to improve institutional capacity in the Recipient’s education sector Rationale The theory of change for this objective was overall sound, as the planned interventions of technical assistance (including strengthened monitoring and evaluation capacity) were likely to lead to improved institutional capacity. Outputs • Provision of technical assistance on data collection and assessments, including a Quality Assurance System (QAS) to establish school profiles based on agreed quality standards. • Conducting of EGRA and two impact evaluations (TWP, early grade reading program). Page 9 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT Education for All Project - Phase II (P124134) • Provision of training on oversight of the school health and nutrition program (contacting beneficiary schools at least twice annually through a beneficiary feedback mechanism (phone surveys)), school roof construction, an enhanced TWP supervision approach, and preparation of a multi-year sector operational plan. • Conducting of three diagnostic sector studies (survey of local education officials and school directors on the communication and understanding of central policy directives; mapping of donor activities across all levels of education; and assessment of results from the 12 Measures policy document implemented by the Ministry of Education). • Technical assistance towards operationalizing a national school accreditation database. • Provision of transportation equipment to facilitate on-site school visits. However, • The activity to establish the ONAPE, a key intervention for public-private partnerships and sustainability, was dropped. • Support for program-based budgeting for decentralized education units was also dropped. Outcomes • The QAS was piloted in a sample of 234 schools, leading to the development of school score cards. According to the ICR, the QAS laid the foundation for a framework to monitor quality of service delivery at the school level and is being scaled up in a follow-up Bank operation. • 73% of schools received on-site inspections, falling short of the original target of 95% but nearly achieving the revised target of 77%. • 62% of TWP schools publicly posted annual expenditure reports, falling short of the target of 95%. Regarding “sustainable” programs: • Efforts to establish a financing framework or plan for the government to take up recurrent sector activities did not lead to any lasting agreements or adoption of plans by the government. • The original outcome indicator on the percentage of community teachers financed by the government was dropped. The government was unable to finance community teacher salaries as planned, and therefore the project shifted funding from community school construction to teacher salaries. The technical assistance activities likely contributed to improved institutional capacity; however, due to shortcomings in implementing key activities to ensure sustainability, achievement of this objective is rated Modest. Page 10 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT Education for All Project - Phase II (P124134) Rating Modest PHREVDELTBL PHINNERREVISEDTBL Objective 3 Revision 1 Revised Objective To support strengthened management of the Recipient’s primary education sector Revised Rationale Outputs See outputs reported above. Outcomes See outcomes reported above. As "sustainability" was dropped as an overarching aspect of the project objective, achievement is rated Substantial. Revised Rating Substantial PHEFFICACYTBL Objective 4 Objective See Objective 4 below, which is entered here as a Revised Objective 4 as it was not part of the original objective. Rationale n/a Rating Not Rated/Not Applicable PHREVDELTBL PHINNERREVISEDTBL Objective 4 Revision 1 Revised Objective Page 11 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT Education for All Project - Phase II (P124134) To support student attendance in select public and non-public primary schools in disadvantaged areas (not an original objective, but added as a new objective) Revised Rationale Although this objective is vaguely worded, the ICR (page 24) suggested that the intended outcome was to “maintain” student attendance, rather than "improve," as at the time of the project restructuring, the key project intervention – the school and health nutrition program – had already been under implementation for several years and therefore the expected reasonable impact was to maintain already improved attendance levels. Outputs See outputs reported above for original objective 1, especially for the school health and nutrition programs. Outcomes • 83.5% and 88% attendance rates were recorded during the last two years of the project period, in schools with the health and nutrition program (target: 78%). Achievement is rated Substantial due to attendance rates surpassing the target, although comparison to a baseline attendance rate would have further substantiated this outcome. Revised Rating Substantial PHOVRLEFFRATTBL Rationale Project under original objectives Achievement is rated Modest all three original project objectives, therefore the overall rating is Modest. Project under revised objectives Achievement is rated Modest for the objective on enrollment, and Substantial for the other two objectives on attendance and sector management; the overall rating is therefore Substantial. Overall Efficacy Rating Not Rated/Not Applicable PHREVISEDTBL 5. Efficiency Page 12 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT Education for All Project - Phase II (P124134) The PAD (Annex 7) provided a brief summary of an economic analysis of the tuition waiver and community schools activities (57.1% of total project cost). Costs were calculated as project costs for these activities (US$ 31.0 million and US$ 7.0 million, respectively). Benefits were calculated as the increase in lifetime income earnings only, without capturing additional non-monetary benefits such as improved health and increased productivity of the economy. The estimated internal rate of return ranged from 15% to 17%, with a net present value range of US$ 122.5 million to US$ 173.9 million, depending on various unemployment and enrollment scenarios. The ICR's follow-up economic analysis (Annex 4) increased the scope of interventions included in the analysis: the two activities above and additionally, the school health and nutrition program and the reconstruction activities following Hurricane Matthew (approximately US$ 27.0 million and US$ 55.6 million, respectively, equivalent to at least 80.2% of total project cost). The internal rate of return ranged from 16% to 22%, depending on the discount rate applied; there was no updated estimate of net present value. However, there were indications of implementation inefficiencies: the damage caused by Hurricane Matthew to some of the infrastructure already built through project funds; high staff turnover in the project implementation unit; shifting government policies; and lengthy procurement processes (for example, for the teacher guide and community school construction activities). There were also disruptions to the tuition waiver program due to lack of government financing, and thus full benefits from this activity likely did not materialize. Given these significant implementation inefficiencies, efficiency in the use of project resources is rated Modest. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 57.00 Appraisal  16.00 Not Applicable 80.20 ICR Estimate  20.00 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Page 13 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT Education for All Project - Phase II (P124134) Project under the original objectives - Moderately Unsatisfactory Relevance of the project objectives was Substantial. Achievement of the objectives to improve access to primary education, to improve quality of primary education, and to improve institutional capacity are all rated Modest due to significant shortcomings in achieving outcomes. Efficiency is rated Modest due to significant implementation inefficiencies. Project under the revised objectives - Moderately Satisfactory Relevance of the revised project objectives was Substantial. Achievement of the objective to support enrollment for disadvantaged groups is rated Modest, while achievement of the objectives to support attendance and to strengthen sector management are rated Substantial. Efficiency is rated Modest due to significant implementation inefficiencies. Overall rating - Moderately Satisfactory According to harmonized OPCS/IEG guidelines on restructured projects, the overall outcome rating is determined by weighting outcomes by the amount of the grant that disbursed before and after the restructuring. In June 2014, approximately US$ 18.57 million of the grant had disbursed, or 18% of the total grant financing. Therefore, the weighted outcome rating is Moderately Satisfactory (original project = 0.54, revised project = 3.28; total weighted = 3.82), representing moderate shortcomings in the project's preparation and implementation. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome Some important gains were made in institutional capacity, including laying foundations for quality assurance of schools and supervision of education programs. However, education outcomes achieved in the areas of enrollment and attendance are at high risk due to lack of financial sustainability. New sources of financing, including from the government, are not yet confirmed, and the National Fund for Education is yet to be firmly established. In addition, while the follow-up Bank operation is continuing some policy directions set by this project, there is also a general shift in government policy away from supporting nonpublic schools. Therefore, risk to development outcomes is considered High. 8. Assessment of Bank Performance Page 14 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT Education for All Project - Phase II (P124134) a. Quality-at-Entry The project was the second operation in a planned three-phase operation. The previous phase 1 project had provided direction on priority interventions and implementation arrangements, upon which this phase 2 project was built. The project design also provided scope for innovation; for example, the community schools approach was piloted in the preparation phase before being launched on a larger scale during the implementation period. However, the original formulation of the project objectives, while relevant, were not realistic or clearly measurable given limited capacity and weak monitoring systems in the country (including the lack of learning assessment systems to measure improved quality). The design also did not take into account the ability of the sector to absorb the newly trained teachers, or effectively mitigate the risk of the failure of government to provide financing. Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision The project implementation period was marked by a long period of political uncertainty (including a period with a transitional government) and a major natural disaster, both of which had negative impacts on implementation. The Bank team responded with significant project restructurings to better align the project with existing conditions and intensified support in key areas. In particular, the ICR (page 32) noted the use of new mechanisms for specific activities to increase focus on the quality of implementation, such as beneficiary feedback and verification mechanisms. The revised objectives and results framework were more realistic and measurable (for example, focusing on enrollment rather than quality or sustainability), although the intended outcomes were still somewhat vague, and the key indicator on tuition waiver support tracked the number of cumulative outputs rather than actual beneficiaries. Overall, most key outputs were delivered and project funds disbursed nearly in their entirety. Safeguards and fiduciary performance were overall satisfactory, with resolution of issues that arose over the project period. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization Page 15 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT Education for All Project - Phase II (P124134) a. M&E Design The results framework was detailed and relevant, and monitoring arrangements and evaluative studies were clearly defined in the project design. However, the M&E design included a large number of indicators to track and did not properly take into account limitations in institutional and system capacity. Neither baseline data nor accurate tracking systems were available to clearly measure outcomes in quality or access (i.e. the revised key indicator measured number of tuition waivers provided, which is not necessarily the number of students enrolled). b. M&E Implementation There were numerous changes to the results framework during the six project restructurings - both in the selection of indicators and the setting of targets - which posed challenges for tracking continuous results from project start to finish (forcing the ICR to rely heavily on external sources of data to report on achievements). Data collection arrangements were clarified with the assistance of the Bank's Fragile States Hub, through the development of a more feasible data monitoring plan. Enhanced supervision mechanisms for specific interventions (tuition waivers and school health) also enabled more accurate data collection for these activities. Planned evaluative studies were conducted for the tuition program and early grade reading program. c. M&E Utilization The ICR (page 33) reported that the enhanced supervision mechanisms enabled a more effective and immediate feedback loop to "more immediately inform project management and decision making" and that the two impact assessments (TWP and early grade reading program) were used to "refine project intervention and advance dialogue on the follow-up Bank operation." No specific examples were cited. M&E Quality Rating Modest 10. Other Issues a. Safeguards The project was classified as an Environmental Category "B" project. Under the original project, safeguard policies on Environmental Assessment (OP/BP 4.01) and Natural Habitats (OP/BP 4.04) were triggered, the latter due to the use of charcoal for cooking for the school health activities. For the former, an environmental Page 16 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT Education for All Project - Phase II (P124134) management framework was prepared to guide school site selection and to mitigate potential adverse impacts from construction and cooking activities. Under the project restructuring, the safeguard policy on Involuntary Resettlement (OP/BP 4.12) was triggered and a screening tool was updated for new community school site selection. The ICR (page 34) reported that the project was in compliance with the safeguard policy on Natural Habitats by providing improved cooking stoves. Two cases of involuntary resettlement arose under the community schools sub-component, the first following delays in training community members on safeguards management (before the safeguard policy was formally in place). Remedial resettlement action was taken for both incidents to compensate the affected parties. The project team confirmed that compliance with environmental safeguards was satisfactory. b. Fiduciary Compliance Financial management: Audit and financial reporting requirements were met, though there were some delays in submission and several issues of non-compliance; hence, there were several ratings in the unsatisfactory range for a period of time. The issues of non-compliance included: lack of a financial specialist in charge of the finance and accounting department; lack of proper accounts reconciliation; lack of interim financial reporting; and lack of statements of expenditures. There were also delays in payments to service providers and beneficiaries. The 2013-14 external audit report issued a qualified opinion due to non-segregation of Caribbean Development Bank disbursements from World Bank disbursements, unsupported expenses of US$ 3,770,733 and US$ 3,766,134, and errors in presentation related to exchange rates. The ICR (page 36) reported that these issues were addressed in financial management action plans, and therefore overall financial management performance was satisfactory. Procurement: Procurement performance was rated in the satisfactory range throughout the project period, with moderate shortcomings noted including: delays in procurement execution (delayed starts of procurement processes combined with lengthy bidding processes) and low-quality procurement documentation. Measures were taken to address these shortcomings, including periodic increases in procurement thresholds and the establishment of multiyear contracts for certain recurrent activities (such as independent service verification, food service providers, audit) that were usually implemented by the same service providers each year. c. Unintended impacts (Positive or Negative) None reported. Page 17 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT Education for All Project - Phase II (P124134) d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory --- Satisfactory Moderately Bank Performance Moderately Satisfactory --- Satisfactory There were shortcomings in the M&E design that were only Quality of M&E Substantial Modest partially rectified in project restructurings. Quality of ICR Substantial --- 12. Lessons Lessons drawn from the ICR: • In a context of weak country monitoring systems, intervention-specific monitoring mechanisms can provide an immediate feedback loop to both improve implementation and provide data for project assessment. In the case of this project, phone surveys, scorecards, and site visits were introduced to provide data and address quality of implementation. • While multiple project restructurings provide the flexibility necessary in fragile situations, they can also pose difficulties in properly assessing project achievements. In the case of this project, the objectives, indicators, and targets were modified on multiple occasions, and thus it was a challenge to identify trends from project start to finish. 13. Assessment Recommended? No 14. Comments on Quality of ICR Page 18 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT Education for All Project - Phase II (P124134) The ICR did an admirable job of laying out the many implementation challenges encountered during the project period, as well as condensing all the project changes and context within the ICR guidelines. The limited data available through formal project indicators were supplemented by data from other sources. The ICR referred to the TWP assessment and summarized its results but did not provide actual data. Moreover, it drew heavily upon this TWP impact study that compared 2002/03 and 2011/12 school census data; although this time period is consistent with the peak of TWP support, there was no additional data to verify that these achievements were sustained through the remaining project period. The ICR (including the Lessons) rightly framed the project approach to fragile countries as a challenging balancing act between high risk/high reward and simplicity in design/scope, and there are some specific, useful lessons articulated such as flexibility to account for fluctuating government priorities and incorporating immediate feedback mechanisms to inform project activities. a. Quality of ICR Rating Substantial Page 19 of 19