Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LS: Basic Education FTI Grant (P116426) Report Number : ICRR0020077 1. Project Data Project ID Project Name P116426 LS: Basic Education FTI Grant Country Practice Area(Lead) Lesotho Education L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) TF-97043 31-May-2013 26,800,000.00 Bank Approval Date Closing Date (Actual) 30-Jun-2010 30-Apr-2015 IBRD/IDA (USD) Grants (USD) Original Commitment 0.00 20,000,000.00 Revised Commitment 0.00 19,593,431.06 Actual 0.00 19,593,431.06 Sector(s) Primary education(80%):Pre-primary education(20%) Theme(s) Education for all(100%) Prepared by Reviewed by ICR Review Coordinator Group Judith Hahn Gaubatz Judyth L. Twigg Joy Behrens IEGPS2 2. Project Objectives and Components a. Objectives According to the Project Appraisal Document (PAD, page 7) and the Grant Agreement (page 3), the project objectives were as follows: • To provide improved facilities at existing primary schools; • To support and contribute to expanding access to pre-primary education; and • To support improvements in quality of teaching. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LS: Basic Education FTI Grant (P116426) The project objectives were not revised; however, targets for key outcome indicators were significantly revised downward in May 2013, at which point approximately 7.2% of the grant had disbursed. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? No c. Components (Note: The appraised and actual costs below reflect Catalytic Fund grant financing only, and do not include co-financing from Irish Aid.) Component 1: Improve the Quality of Primary School Infrastructure (Appraisal: US$ 9.1 million; Actual: US$ 9.2 million): This component aimed to improve the quality of physical facilities through the provision of new classrooms, latrine blocks, and furniture. An important feature of this component was to decentralize responsibilities for bidding, certification and supervision of civil works, in an effort to lower costs and accelerate the pace of construction. Activities also included training for district officials, school officials, and small-scale contractors. Component 2: Support and Contribution to the Expansion of Pre-Primary Education (Appraisal: US$ 1.5 million; Actual: US$ 1.8 million): This component aimed to support the expansion of pre-primary education, particularly for poor and vulnerable children, in an effort to improve retention and repetition rates in later primary school years. Pre-primary education was to be provided through "reception classes" consisting of an extra room in the existing primary school for children aged five years old. Activities included training for reception class care-givers and provision of learning materials and food supplements. Component 3: Contribution to Improvements in Quality of Teaching (Appraisal: US$ 7.8 million; Actual: US$ 9.2 million): This component aimed to improve quality in primary schools, primarily through teacher development in designated "difficult" schools (i.e. small, mountain schools) and curriculum revision. The first set of activities was to include provision of incentive bonuses to teachers who accept posts in "difficult" schools, and upgrading the skills of unqualified individuals currently teaching in difficult schools through training. The activities in this component were intended as a pilot, in order to identify the most cost-effective approach to attracting and retaining qualified teachers in difficult schools; hence, this component also included a planned impact evaluation. The second set of activities was to include the development of teaching materials with an increased focus on early literacy and numeracy, training for teachers, and provision of textbooks for core subjects. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost • The actual project cost was US$ 25.1 million, compared to the appraised amount of US$ 26.8 million. The latter included approximately US$ 1.1 million in taxes and duties. • Actual costs for Component 3 (Improvements in Quality of Teaching) were higher than appraised due to the increased focus on these interventions, to balance the decreased scope of construction activities. Financing • The actual grant financing from the Bank was US$ 19.6 million (according to the Bank's project portal), compared to the appraised amount of US$ 20.0 million. The ICR (page 23) reports the actual amount as US$ 20.0 million; the ICR does not provide an explanation for the discrepancy. • The government of Ireland committed to provide US$ 6.8 million at the time of project appraisal. Following the Mid-Term Review in November 2012, the government of Ireland withdrew from the country and therefore discontinued its co-financing support, with a final disbursement of US$ 4.9 million. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LS: Basic Education FTI Grant (P116426) Borrower contribution • There was no planned Borrower contribution. Dates • May 2013: The project was restructured due to slow progress in implementation. The scope of the components was significantly changed to account for low capacity of the Ministry of Education and Training (MoET). Targets for key outcome indicators were revised accordingly. The project closing date was also extended to November 2014 to allow completion of restructured activities. • October 2014: The project closing date was extended to April 2015 to allow completion of activities. 3. Relevance of Objectives & Design a. Relevance of Objectives The project objectives were highly relevant at project approval and remained as such until project closing. The primary education sector in Lesotho has seen a marked increase in net enrollment due to the introduction of free primary education in 2000. However, quality of education remains a significant challenge, as reflected by high repetition rates, high pupil:teacher ratios, high proportion of unqualified teachers, and sub-standard school facilities. The need to improve the physical state of school facilities is acute: as noted in the PAD (page 9), "the terrain and climate in Lesotho make provision of adequate shelter an unusually important factor in the attendance and retention of pupils." At the time of project appraisal, more than half of the classrooms in use were open-air or had temporary roofs, thereby falling short of the required standards. In addition, quality of instruction was inadequate due to low qualifications of teachers (particularly in the remote mountains regions) and poor/unavailable curriculum materials. The government's education strategic plan for 2005-2015 prioritizes provision of quality education to ensure completion of primary school and access to early childhood education. The Bank's most recent Country Partnership Strategy for 2010-2014, effective at project effectiveness and closing, similarly identifies improved access to and quality of education as key outcomes in human development. The project objectives remained highly relevant under the revised targets. Rating Revised Rating High High b. Relevance of Design The project design was substantially likely to achieve the stated objectives. Interventions to improve the quality of school infrastructure centered on the construction of additional classrooms and latrines to existing schools. This was a relevant strategic decision given the limited fiscal budget and capacity to construct new school buildings altogether. Project schools were selected according to current enrollment and infrastructure needs, with an explicit focus on remote mountainous areas. Interventions to improve the quality of teaching included measures to attract teachers to the "difficult" schools and upgrade skills of unqualified teachers, as well as support for curriculum revision and textbook distribution. Lastly, project activities supported increased access to pre-primary education, which was considered to be a factor in improving lower primary grade repetition rates and learning outcomes. These activities included training for pre-primary caregivers and provision of materials. However, the likelihood of achieving original targets for infrastructure activities was unclear, given low capacity of the implementing entities. The project design remained substantially relevant under the revised targets, including increased likelihood of achieving the downwardly revised targets. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LS: Basic Education FTI Grant (P116426) Rating Revised Rating Substantial Substantial 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective To provide improved facilities at existing primary schools. Rationale Outputs • Construction and furnishing of 143 classrooms in 30 existing schools (original target: 330; revised target: 135). • Construction of 93 latrine blocks (original target: 100; revised target: 87). • Training to MoET staff and contractors on technical aspects of school construction. Outcomes • See information reported above on number of classrooms and latrine blocks constructed. There is no information provided in the ICR on the quality of the infrastructure provided. The original indicator was as follows: "% of project schools meeting agreed standards." Achievement of the objective under the original targets is rated Modest due to significant shortfalls in achieving targets. Rating Modest Revised Objective n/a Revised Rationale (see outputs and outcomes with revised targets above) Achievement of the objective under revised targets is rated Substantial due to meeting of targets. Also, the ICR (page 15) reports that enrollment in the 30 project schools increased from 5,834 in 2013 to 5,948 in 2014. Though representative of only one year of data, the ICR (page 15) reports that this increase was "contrary to enrollment trends in other primary schools, which did not benefit from school construction under this project" and that school enrollment in non-project schools was declining on average 1.3 percent annually between 2009 and 2014, although the school-age population was estimated to have grown 1.3 percent annually. Revised Rating Substantial Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LS: Basic Education FTI Grant (P116426) PHEFFICACYTBL Objective 2 Objective To support and contribute to expanding access to pre-primary education. Rationale Outputs • Provision of support to 140 reception (pre-primary) classes in existing schools (original target: 120; revised target: 140). This support included provision of appropriate learning materials, training to pre-primary caregivers, and food supplements. Outcomes • The number of children enrolled in project-supported reception classes at the beginning of 2015 was 3,682. This achieved the original target of 3,600 and the revised target of 3,220. Approximately 60 percent of enrollment in pre-primary education nationwide took place in project-supported reception classes, although this still covers only about 5 percent of the age-5 population. Achievement of the objective under the original targets is rated Substantial due to enrollment of over 3,600 students, who otherwise would not have access to pre-primary education. Rating Substantial Revised Objective Not applicable; this objective was not changed. Outcome targets were revised. Revised Rationale (see outputs and outcomes under revised targets above) Achievement of the objective under the revised targets is rated Substantial due to enrollment of over 3,600 students, who otherwise would not have access to pre-primary education. Revised Rating Substantial PHEFFICACYTBL Objective 3 Objective To support improvements in quality of teaching. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LS: Basic Education FTI Grant (P116426) Rationale Outputs • Provision of incentive grants to 1,096 teachers to work in "difficult" schools (original target: 800; revised target: 1,000). • Supplemental distance training for 380 teachers working in "difficult" schools (original target: 800: revised target: 600). The shortfall was due to the suspension of the program in 2013 and 2014 over concerns about the distance training program's effectiveness and sustainability. The ICR does not report on whether these concerns were resolved. • Development of simplified curriculum focused on early reading and math. This curriculum, including assessment materials, was piloted in 70 schools. • Training of 11,158 teachers, principals, district education authorities, and inspectors in teaching methods adapted to the new curriculum (original target: 4,500; revised target: 6,515). • Purchase of 1.1 million textbooks, of which 928,276 were distributed (original target: 13,300; revised target: 913,300). Field visits confirmed that teachers were using the new textbooks. • Preparatory study on the development of a teacher qualification framework. • Baseline study to measure early grade literacy (grades 1-3). However, • Baseline data on teaching and learning were collected but subsequently "misplaced" before being entered into the information system. Outcomes • The pupil:qualified teacher ratio improved from 59:1 in 2010 to 45:1 in 2015. This achieved the target of 45:1. • The ICR also provides data on survival rates for primary school; however, given the time frame of the project, attribution of project interventions to these outcomes is not strong. Achievement of this objective under original targets is rated Modest due to lack of clear evidence that the teaching materials, curricular development, and training led to improved teaching practices and/or learning outcomes. Rating Modest Revised Objective Not applicable; this objective was not changed. Outcome targets were revised. Revised Rationale (see outputs and outcomes under revised targets above) Achievement of this objective under revised targets remains Modest. Revised Rating Modest Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LS: Basic Education FTI Grant (P116426) 5. Efficiency Efficiency of the project over the entire project period is rated Modest due to limited evidence on actual benefits realized, although the project achieved a number of targets in the delivery of outputs (classroom construction, textbooks, teacher bonuses). The project design featured a decentralized approach to classroom construction that was expected to reduce the cost of classroom construction. These design elements included procurement by district tender panels, use of standardized bid documentation and design, use of local and smaller contractors (with lower overhead and transport costs), and reduction in size of civil works lots. However, it is unlikely that these expected gains in efficiency were realized, given significant implementation delays and the subsequent decision to revert to a more centralized approach due to weak local capacity. The aforementioned delays included logistical challenges in construction work in remote rural areas and slow procurement processes due to weak capacity. A unit cost analysis of classroom construction (ICR, Annex 3) indicates comparable costs to other school construction programs within the country. The unit cost per classroom for this project was estimated at US$ 36,800, while unit costs for a government-financed classroom were US$ 48,000, another donor-financed classroom was US$ 24,117, and an international NGO-financed classroom was US$ 44,445. However, the number of classrooms actually constructed was significantly lower than appraised, due to the weak capacity, while the overall cost for this component remained the same. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Project under original targets - Moderately Unsatisfactory Relevance of the project objectives is rated High, and relevance of the project design is rated Substantial. Achievement of the objective to provide improved facilities at existing primary schools is rated Modest due to significant shortfalls in meeting (original) targets. Achievement of the objective to expand access to pre-primary education is rated Substantial due to evidence of increased enrollment. Achievement of the objective to support improvements in the quality of teaching is rated Modest due to lack of clear evidence. Efficiency is rated Modest. Project under revised targets - Moderately Satisfactory Achievement of the objective to provide improved facilities at existing primary schools is rated Substantial due to meeting of revised targets. Other ratings are the same as above. Overall outcome rating - Moderately Satisfactory According to harmonized OPCS/IEG guidelines, the combined outcome rating is determined by weighting the proportion of the grant that was disbursed before and after the restructuring (revision of targets). The grant had disbursed only US$ 1.43 million out of US$ 20.0 million, or 7.2%, at the time of restructuring. Therefore, the combined outcome rating is Moderately Satisfactory, indicative of moderate shortcomings in the project's preparation and implementation. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LS: Basic Education FTI Grant (P116426) a. Outcome Rating Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating Facilities are still operational, although there has been limited attention given to operations and maintenance matters. Financial responsibility for reception classes has been absorbed into the MoET's own budget, which will contribute to sustainability of the pre-primary activities. However, the teacher incentive scheme was ended with the project closing date, and it is unclear whether qualified teachers will remain in remote areas. A follow-up Bank operation has been proposed in order to continue support to the primary education sector; however, this project is still in the preparatory stages. a. Risk to Development Outcome Rating Substantial 8. Assessment of Bank Performance a. Quality-at-Entry The original project implementation arrangements aimed to create efficiency and sustainability, as well as to accelerate construction, by contracting school construction activities to decentralized entities and small contracting firms that needed capacity building experience. However, these arrangements proved untenable due to continued weak capacity and the underestimation of the logistical challenges of completing civil works in rural mountainous areas. Although the former was recognized as a significant implementation risk, mitigation measures were inadequate. As noted in the ICR (page 8), there was inadequate analysis of decentralized capacity, and the expectation of implementing all activities within the three-year planned project period turned out to be unrealistic. Shortcomings in the results framework included a lack of indicators measuring outcomes, rather than simply outputs. Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision Project implementation was affected early on by an unrelated incident in the country portfolio. A misappropriation of funds was flagged in a prior Bank education sector project, resulting in an internal Bank investigation. This led to a slow-down in implementation as the Bank team awaited findings of the investigation and possible implications for this project. This incident also created some tension between the Bank and the government. However, the Bank team used the opportunity afforded by the Mid-Term Review to recommend changes to the scope of activities, implementation arrangements, and M&E framework. The centralizing of procurement back to the MoET, clarification of the responsibilities of the main implementing unit in the MoET, and hiring of additional technical staff helped to accelerate the pace of activities, combined with an increased level of support from the Bank task team and management. Proactive efforts by the task team also helped to bring the project into safeguards compliance (see Section 11a), as the MoET had no previous experience in addressing the Bank's safeguard policies and had made limited progress in implementing safeguard measures during the initial project period. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LS: Basic Education FTI Grant (P116426) 9. Assessment of Borrower Performance a. Government Performance The government has demonstrated strong commitment to implementing its primary education sector plan. Having introduced free primary education to raise the net enrollment rate, the government is now pursuing measures to improve quality of education, including among the most vulnerable populations in remote mountainous regions. It has provided a supportive and stable policy environment, although capacity remains a significant constraint. Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance The primary implementing agency was the Ministry of Education and Training (MoET). Although previous donor projects were implemented by a project management unit, efforts to build local capacity led to the decision to mainstream project implementation in the MoET. However, this led to significant implementation shortcomings, include extremely slow procurement (the use of decentralized tender panels for classroom construction was eventually discontinued) and the lack of an effective supervising entity for school infrastructure. A hybrid arrangement was eventually created (full-time project management unit integrated within the MoET) and implementation improved considerably, as reflected by the significantly improved pace of disbursement. The pace of classroom construction increased after the first restructuring when implementation arrangements were revised. This included the hiring of five civil works supervisors to closely monitor school construction sites. According to the ICR (page 11), this measure played a critical part in ensuring the meeting of construction targets by the end of the project period. There is inadequate information reported in the ICR on financial management performance to verify satisfactory performance. Implementing Agency Performance Rating Moderately Satisfactory Overall Borrower Performance Rating Moderately Satisfactory 10. M&E Design, Implementation, & Utilization a. M&E Design The short-term, fast-track nature of the operation led to a focus on outputs as the key indicators, while actual outcomes were to be assessed through impact evaluations. However, some of the key indicators were not easily measured (decline in shortfall of classrooms, decline in shortfall of qualified teachers) and also lacked baseline figures. The evaluation design included an impact evaluation of the pilot teacher incentive program, an assessment of the reception classes, and a baseline survey on early literacy. Data were to be collected through the regular school census and joint sector reviews. b. M&E Implementation The M&E framework was significantly revised during the project period, including the downward revision of targets for key indicators due to significant implementation problems prior to the first project restructuring. There was also a revision in the wording of indicators during the second restructuring. As reported in the ICR (page 10), the initial design of the teacher incentive evaluation needed to be revamped due to data collection challenges. Therefore, a new evaluation was designed and conducted, with the active participation of the MoET, which subsequently increased its M&E capacity; analysis of the data is still being undertaken. The baseline early reading assessment was also carried out; however, the results were misplaced before they could be entered into a database and the ICR does not report on any attempts to rebuild the baseline data. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LS: Basic Education FTI Grant (P116426) c. M&E Utilization Close monitoring of construction progress helped ensure completion of activities. Specifically, five additional supervisors were hired to monitor each school construction site bi-monthly, "which played a critical part in meeting the revised school classroom construction targets by the end of the project (ICR, page 11). Initial findings from the teacher incentive impact evaluation are being used to inform the design of the follow up Bank project (specific details are not provided in the ICR). M&E Quality Rating Modest 11. Other Issues a. Safeguards The project was classified as an Environmental Category "B" project, which triggered the safeguard policy on Environmental Assessment (OP/BP 4.01). An addendum to the existing Environmental and Social Monitoring Framework (from the prior education sector project) was completed. The MoET initially had very limited understanding of and experience with the Bank's safeguards policies. Therefore, documentation of safeguard measures was inadequate during the initial project period, including on community stakeholder engagement on civil works and soil erosion assessments. In addition, there were no safeguard ratings recorded in the implementation status reports. However, following the Mid-Term Review, the Bank team took proactive measures to bring the project into safeguards compliance, including the hiring of an engineering firm to undertake an environmental and social audit of civil works sites and the identification of remedial safeguards actions. According to the ICR (page 11), the MoET completed remedial actions as identified and finalized documentation of social safeguards to IDA's satisfaction. b. Fiduciary Compliance Procurement: Due to the decision to mainstream donor project implementation into the MoET, rather than in a separate project management unit, fiduciary capacity was initially inadequate and led to significant implementation problems. However, a procurement consultant was eventually hired to support the procurement unit in the MoET, which led to marked improvement in the pace of disbursements and implementation. Another initial design measure to decentralize procurement for classroom construction led to implementation challenges, as most district-level tender panels had limited experience. The use of small procurement lots did lead to a wider market for contractor but, according to the ICR (page 12), did not necessarily reduce the cost and delivery time of construction or improve the quality of construction. Therefore, the decentralized arrangement was discontinued at the first restructuring. Financial management: There is limited information provided in the ICR on financial management performance. A supervision firm (to support management of infrastructure) was hired late into the project period, and then its contract was terminated for poor performance. A hybrid arrangement was finally created (full time project coordination unit integrated in the MoET), which provided improved implementation and fiduciary support to the project. c. Unintended impacts (Positive or Negative) None reported. d. Other --- Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LS: Basic Education FTI Grant (P116426) 12. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Moderately Satisfactory Moderately Satisfactory --- Risk to Development Outcome Substantial Substantial --- Bank Performance Moderately Satisfactory Moderately Satisfactory --- Borrower Performance Moderately Satisfactory Moderately Satisfactory --- Quality of ICR Substantial --- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons Lessons drawn from the ICR (page 22), adapted by IEG: • Innovative institutional arrangements (to foster efficiency and sustainability) can be appropriate but require intensive analysis and preparation in the design stage. A phased approach may be particularly useful in order to quickly incorporate lessons learned. • A centralized procurement strategy for peripheral civil works, although initially counter-intuitive, can be more effective and efficient, since it consolidates small and scattered bids. In the case of this project, procurement for classrooms and latrines within the same schools was adequately managed in one lot. In addition, • A teacher incentive program can be an effective approach to improving the quality of teaching in remote areas; however, the lack of sustainability of such a program and the lack of timely evaluation of the results detract from the apparent short-term success of the program in this case. 14. Assessment Recommended? No 15. Comments on Quality of ICR The ICR does a commendable job in articulating the results chain (linkages between specific project activities and intended outcomes) and focusing the analysis of achievements on plausible outcomes. It also clearly explains the rationale behind the various decisions made for the project, such as the initial implementation arrangements for decentralized procurement/contracting, the switch back to centralized procurement, re-allocations of project financing, realignment of implementing responsibilities, and revisions to the results framework. The analysis of achievements for the first objective ("to provide improved facilities at existing primary schools") is notable for providing evidence of a comparison group (enrollment trends in primary schools which did not benefit from school construction under this project). Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LS: Basic Education FTI Grant (P116426) a. Quality of ICR Rating Substantial