ICRR 13476 Report Number : ICRR13476 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 06/24/2011 PROJ ID : P074320 Appraisal Actual Project Name : Rehabilitation Of US$M ): Project Costs (US$M): 20.45 22.79 Basic Education Country : Sierra Leone Loan/ US$M ): Loan /Credit (US$M): 20.00 22.79 Sector Board : ED Cofinancing (US$M ): US$M): 22.55 25.72 Sector (s): Secondary education (40%) Primary education (40%) Central government administration (20%) Theme (s): Conflict prevention and post-conflict reconstruction (50% - P) Education for all (50% - P) L/C Number : CH020 Board Approval Date : 05/09/2002 Partners involved : AfDB Closing Date : 06/30/2008 12/31/2009 Evaluator : Panel Reviewer : Group Manager : Group : Reehana Rifat Raza Robert Mark Lacey IEG ICR Review 1 IEGPS1 2. Project Objectives and Components: a. Objectives: The Project Development Objectives (PDO) as stated in the PAD were “to assist the Government of Sierra Leone (GoSL) to re-establish education services, and prepare the grounds for building up the education sector . Specifically, the Project would: (i) assist participating schools to achieve a basic operational level (BOL)*; and (ii) build up the capacity of the Ministry of Education Science and Technology (MEST) to plan and manage the delivery of education services� (pg.3). The Grant Agreement (GA) states the objectives as “(i) assisting eligible schools to achieve basic operational level .; and (ii) improving the capacity of MEST to plan and manage the delivery of education services .� This ICR Review will use the statement of objectives in the GA as this is more monitorable . [* Basic Operational Level (BOL) is achieved in schools which have a physical structure safe for children and teachers, which are supplied with basic furniture, a set of core textbooks for each pupil, teaching and learning materials, where at least 50% of teachers are trained, and where there is an adequate teacher :student ratio (normally no more than one teacher per 40 students), a school management committee (SMC ) for primary schools and a Board of Governors (BOG) for junior secondary schools (JSS).] b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): Component 1: Support to Service Providers to Rehabilitate the Provision of education services (Appraisal : US$18 US$ 17. 17 .96 million ) US$ 18 million; Actual : US$17 (1a) Support to education service providers (ESPs – NGOs, community-based groups) to bring primary and junior secondary schools (JSS) to BOL, and provision of funds in form of grants to finance approved proposals from ESPs to bring institutions to BOL standard . An additional goal of this component was to expand coverage to under -served areas. (1b) Introduce peace education into the curriculum and the school environment . Under this program, teachers, parents and students would be trained to engage in peace -related processes to build a broad culture of peace within and beyond schools. The program would develop tool-kits/ manuals for teachers, parents and students to address curriculum materials and specific teaching and classroom activities as well other activities that would build towards creating a peaceful environment in schools and communities . (1c) Support to non-formal and emergency education programs, and to Internally Displaced Persons (IDPs) or refugees who missed out on years of schooling during the war by offering the Complementary Rapid Education for Primary Schools (CREPS) Program. The program targeted approximately 500,000 children of 10-14 age group who missed out on years of schooling during the conflict . Component 2: Enhanced Institutional Capacity of the Ministry of Education Youth and Sport MEYS ) (Appraisal (MEYS) ( Appraisal : US$ 2.0 million; Actual : US$ 4.38 million ) US$2 (2a) Establishment of a Project Coordinating Unit (PCU) with overall responsibility of planning, coordinating and managing activities to be funded by the project as well as providing overall technical support as required for the MEYS. (2b) Strengthening MEYS’s capacity in planning and management . Under this sub-component the following were to be established: (i) a procurement unit, (ii) a civil works unit, (iii) a financial management system, (iii) an Education Management Information System (EMIS), and (iv) Human Resource Management capacity . (2c) Supporting and monitoring of education service delivery . This sub-component aimed to support the training of 80 teacher trainers who would serves as resource people for the provision of in -service training. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Financing : Parallel financing was provided by the African Development Bank (AfDB), whose funds were managed through an IDA-administered trust fund, and a US$ 2.0 million grant from DfID was also anticipated . The latter, however, failed to materialize for reasons unexplained in the ICR, and the consequent financing gap was filled by increased contributions from the other partners . Actual financing by project closure totaled US$ 48.2 million -- IDA US$22.5 million (appraisal US$20 million), AfDB US$24.4 million (appraisal US$22.5 million), and GoSL US$1.3 million (appraisal US$0.5 million). The increase in the IDA contribution was made possible through the depreciation of the US dollar against the SDR. The reallocations resulting from these changes, together with the fact that AfDB financing became available a year later than that of IDA, necessitated an amendment to the Grant Agreement in July, 2006. A second reallocation was made in May 2008 to strengthen and support the district education officers . Dates : The project closing date was extended twice to ensure (i) that all civil works were completed and schools were handed over to communities and (ii) to ensure that the MEYS had time to implement the capacity building component aimed at training district inspectors to monitor and supervise basic services . The first extension was from June 30, 2008 to June 30, 2009 and the second from to December 31, 2009 . 3. Relevance of Objectives & Design: high The objectives were appropriate for the post conflict environment facing the Relevance of Objectives is rated high. country at appraisal and continue to be relevant today . At the end of the conflict, a Transitional Support Strategy (TSS) for 2002-2003 was prepared by the Bank to support the implementation of the GoSL ’s Interim Poverty Reduction Strategy (I-PRSP). The aim of the TSS was to support Sierra Leone ’s transition from post-conflict reconstruction to sustainable poverty reduction . As provision of education services to rural and underserved areas was prioritized in the TSS, both to impede renewed conflict and to lay the foundation for sustained poverty reduction, this project was highly relevant and was one of the lending activities proposed in the TSS for FY 03. The current Country Assistance Strategy (FY10-FY13) continues to emphasize the importance of human development , particularly education. As the CAS emphasizes, “The quality of education remains a problem, however, with limited teacher skills and frequent absenteeism calling for efforts in training, capacity building ,and incentives for teachers, particularly in remote areas� (p 18). Relevance of Design is rated as modest .. Design built on existing organizations already operating in the area (local and international NGOs and local contractors ), which undertook a range of tasks, including managing school construction and rehabilitation, distribution of new learning materials, and teacher and school management committee (SMC) training. However, insufficient account was taken of these organizations ’ lack of capacity and the weak, conflict-affected, institutional environment in which they operated . Capacity building resources within the project were insufficient to mitigate these . The results framework was over-ambitious for a post-conflict environment, especially given the limited technical and financial resources assigned to the project . Many of the outcome indicators were difficult, if not impossible, to measure, because of poor data availability . The BOL concept, while useful, embodies several targets beyond the scope of the operation . Goals were set for whole districts in which activities were located, although not all schools within a district were chosen to benefit from the project . 4. Achievement of Objectives (Efficacy): (i) Assisting participating schools to achieve a basic operational level (BOL). Substantial . Outputs 60 percent of primary schools met BOL, as compared to the established target of 50 percent. 60 percent of junior secondary schools have met BOL as compared to the established target of 50 percent Completion rate of primary enrolment in targeted districts increased to 65 percent, short of the target of 70 percent. The pass rate for the Basic Education Certificate Examination (BECE) increased by 8 percent from 64 percent to 72 percent, short of the target of 84 percent Increasing primary girls’ and JSS enrollment in target districts by 20% was not monitored, but the ICR infers from the high gross enrollment rate (GER) that good progress was made. The following outputs were achieved with IDA funds : 192 primary schools constructed or rehabilitated; 48 JJS constructed or rehabilitated; 563,789 set of primary textbooks distributed; 51,006 JSS textbooks distributed; 192 SMCs set up; and 3,337 untrained teachers trained. Outcome Although enrollment and quality education targets were, respectively, unmonitored and not fully met, the project succeeded in its objective of assisting schools to achieve the BOL standard . The target of reaching the BOL level was exceeded for both primary and junior secondary schools . Significant improvements were also made in completion and basic education examination pass rates, even though these were less than anticipated at appraisal . (ii) Strengthening MEST’s capacity to plan and manage the delivery of education services . Modest . Outputs Established a functioning PCU which was able to conduct procurement and financial management activities, undertake technical review of grant proposals for the approval of the project steering committee (PSC) and monitor activities. Building capacity of seven district level inspectors and SMCs to monitor education service delivery through training in all seven districts. Strengthened capacity of all SPs and contractors involved in the project through their active involvement in project implementation and through training . Developed teacher trainers (80) as a resource to undertake teacher in -service training. In total 3,337 teachers received in-service training. This was less than the initial number planned of 10,000 but exceeded the proposed revised targets. Training of MEYS staff in areas of financial, procurement and project management . Although the EMIS was institutionalized through the project, it was not functioning at project closure (see below). A school census was conducted in 2006/2007 and a second one was also conducted although the data from the latter was not entered in the system. Even the first proved insufficient to help MEYS with overall planning and tracking of key indicators for the project. Outcomes Although the project helped to support MEYS in the provision of schools infrastructure, training staff, procurement and management of education delivery, capacity at both central and local levels remains weak and undermined project implementation. MEYS did empower Local Councils to delivery education services, and capacity at the district level was improved through provision of key inputs . However, the empowered Local Councils were unable to supervise construction and supervision activities effectively . Similarly, while service providers and contractors did benefit from “learning by doing� through their involvement in project implementation, a majority found it difficult to comply with procurement and fiduciary rules and procedures . The project was unable to establish a functioning EMIS (ICR, page 26). Lack of human resources and the inability to develop in -house capacity once the consultants departed undermined the system ’s sustainability. The ICR reports that one reason for this was the inadequate salaries offered to Sierra Leone civil servants . The absence of a properly functioning EMIS negatively affected MEYS’s ability to undertake overall planning for the education sector as well as monitoring and evaluation . 5. Efficiency (not applicable to DPLs): The PAD reports that the conditions in Sierra Leone at the time of appraisal made calculation of an EER or NPV infeasible. Efficiency in the ICR was measured in terms of achieved outputs and, to some degree, technical efficiency through calculating unit costs, although these were not compared to others in ways that would have enabled cost effectiveness to be determined . Given the lack of information on which to base a judgment, efficiency is rated as modest . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re-estimated value at evaluation : re- Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The project’s objectives were highly relevant to Sierra Leone ’s post-conflict situation and development strategy, though design was over-ambitious for the prevailing conditions . The project succeeded in bringing about a considerable improvement in the quality of educational services delivered in directly benefiting schools in challenging circumstances, and the number of both primary and junior secondary schools reaching the BOL standard exceeded the targets. Capacity strengthening was, however, less successful, and efficiency is rated modest given the lack of sufficient information to reach a judgment . Outcome is assessed as moderately satisfactory . a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: A number of key factors constitute a significant risk to the project ’s development outcomes. First, there is a risk that supply of facilities, material and training provided under the project cannot be sustained . Maintenance of school infrastructure may be under-funded. Already there is some evidence that Local Councils are unable to sustain the project’s interventions. Second, there is a risk that the Damage Index used to select schools for rehabilitation under the project will not continue to be used, thereby undermining transparency and credibility in the eyes of stakeholders . Third, the EMIS, which is essential to ensuring adequate monitoring and evaluation and better decision -making, is not functioning. a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: at -Entry : Moderately Satisfactory Ensuring Quality -at- The Bank team prepared the project in a challenging and rapidly evolving socio -economic environment. The PHRD grant was effectively used to fill information gaps, and lessons were drawn from Bank experience in other conflict environments. Sufficient resources were allocated to the project . However, inadequate consideration was given to the risks of working with NGOs in a weak institutional environment and the risk assessments proved to be optimistic given the context in which the project was being implemented . The ICR, p.12, states "In retrospect, the fully satisfactory assessment by QAG seems a bit generous given the experience during implementation . For example, while in principle the decision to use NGOs or Service Providers for implementation was a good one, there was an under assessment of the risks associated with working with NGOs that could not have been screened and in relying entirely on a weak commercial banking system following a war to help compensate for failure. The challenges associated with school construction in difficult areas were also not sufficiently taken into account. Both of these factors significantly delayed implementation and could have been foreseen ." Furthermore, the ICR indicates that the results framework was overly ambitio us; and the challenges associated with school construction in difficult areas were not sufficiently taken into account during project preparation . These factors significantly delayed implementation . Quality of Supervision : Satisfactory Supervision took place in a very challenging environment The Bank team ensured that the Government and the PCU were kept on top of potential problems, and proposed appropriate actions to resolve challenges . The implementation status reports (ISR) were realistic in rating achievements of project objectives and project implementation. The team also monitored safeguards and fiduciary compliance, effectively and over time, hired the appropriate procurement and financial management specialists that were located in Sierra Leone . at -Entry :Moderately Satisfactory a. Ensuring Quality -at- b. Quality of Supervision :Satisfactory c. Overall Bank Performance :Moderately Satisfactory 9. Assessment of Borrower Performance: Government Performance : Satisfactory The Government was actively engaged and committed to the project throughout its implementation . Bank and GoSL counterparts had good interactions and government officials maintained appropriate levels of review, approvals and follow-up. An area of concern was the Government ’s unwillingness to request revisions in indicators/targets agreed to during the MTR . There were also initially some delays in counterpart funding . This did not signify a lack of commitment to this project, but is rather a systemic problem in Sierra Leone, reflecting the financial challenges a poor country faces in a post conflict environment . Implementing Agency Performance : Satisfactory The PCU fiduciary staff was hired before the project became effective and received continuous training from Bank staff throughout implementation . The ICR reports (page 34) that the PCU was effective in carrying out all aspects of project management such as financial management, procurement arrangements, reporting activities and disbursements. The PCU provided important guidance to service providers in procurement procedures, financial management and the reporting of expenditures . The capacity of many service providers improved as a result of their involvement in the project . Still, substantial delays were incurred because a number of service providers were unfamiliar with the Bank procurement procedures . There were also weaknesses in financial and contractual management. There were difficulties in finding and retaining suitable counterpart staff for the procurement unit in MEYS. Frequently, counterpart staff would leave the Ministry soon after having received training. However, according to Regional comments, despite difficult country conditions, "the delays in the first two years were followed by stronger performance as the PCU gained experience, training, and had access to resources. The PCU was disbanded after the project closed but some key staff have since been recruited by the MOE ensuring institutional memory...". a. Government Performance :Satisfactory b. Implementing Agency Performance :Satisfactory c. Overall Borrower Performance :Satisfactory 10. M&E Design, Implementation, & Utilization: Design : The proposed M&E design was over -ambitious for the context in which the project was being implemented. The indicators were too focused on outcomes and higher level targets which proved difficult to monitor because of the lack of good data and because achievement of the targets depended on factors outside the scope of the project. For example, the BOL concept, the underlying idea for the first two indicators, is an index which includes elements such as teacher:student ratios which were beyond the project ’s influence. The indicator of 20% improved pass rates for the BECE exam was also outside the project ’s scope since BECE pass rates are restricted by the number of JSS places available. Implementation : It has not proved possible to establish a functioning and sustainable EMIS system or to institutionalize the annual survey . This has undermined the achievement of the second PDO -- building MEST capacity to plan and deliver education services . The principal difficulty was the inability to hire and retain a director for the EMIS partly because of inadequate salaries offered in the Sierra Leone civil service .. Utilization : Without the EMIS, MEST has been unable to undertake overall planning or to monitor the delivery of education services. The lack of data also weakened the monitoring of four of the six PDO indicators and the use of the Damage Index. a. M&E Quality Rating : Modest 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): Safeguards : The project involved school construction activity and was classified as Category B, thereby triggering OP 4.01. Although the ICR reports that there were no significant deviations from the Bank ’s safeguards policies and procedures, it provides no details to substantiate this assertion . Fiduciary : The ICR reports that there were no significant deviations from the Bank's fiduciary policies and procedures. Again, however, there is no discussion or supporting evidence . Unintended (positive and negative ) consequences . There were a number of positive impacts of the project not initially considered. These include: The project set up comprehensive standards for school construction and water wells, including architectural plans, which are now being regularly used by MEYS and other service providers . Schools were selected for rehabilitation in an equitable and transparent manner which, according to the ICR, fostered community solidarity. The publicity the project received led to large scale public awareness and encouraged transparency and accountability. Local Councils started to play a larger role in the education sector in various communities . 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Satisfactory Satisfactory Risk to Development Moderate Significant The risks of a serious reduction in the Outcome : post-project supply of facilities and basic material, of lack of transparency in choice of schools for improvement, and of continued inoperability of the EMIS are significant. See Section 7. Bank Performance : Satisfactory Moderately There were weaknesses in project Satisfactory preparation. Design and the resultsframework were over ambitious and inadequate consideration was given to the risks of working with NGOs in a weak institutional environment . See Section 8. Borrower Performance : Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: (i) The decision to opt for SPs in a post conflict environment could well be appropriate if those organizations are already functioning effectively in war -affected areas. However, they need to be continually assessed and trained during project implementation and be provided with sufficient resources to perform their tasks . The necessary monitoring and evaluation structures need to be in place to ensure that this is taking place . (ii) Implementation of projects financed by more than one external partner is enhanced if procedures and guidelines on fiduciary matters are harmonized . (iii) There are major challenges facing post conflict countries in monitoring indicators in a capacity constrained environment. Focusing on outcomes which are challenging to measure and difficult to achieve in a short -time period should be avoided and more simple monitoring and evaluation frameworks adopted . 14. Assessment Recommended? Yes No Why? An assessment of this project could deepen understanding of the challenges of education sector project implementation in a post-conflict environment. There are a number of lessons regarding how operations can be designed to build capacity within the constraints such an environment presents . An assessment could also measure some of the project’s long term benefits. 15. Comments on Quality of ICR: Although the ICR is reasonably well argued, it would have benefited from a more detailed discussion on specific lessons learnt from implementing a project in the post -conflict environment which otherwise may be lost . There are a number of unintended positive impacts that could have been discussed and documented further . The ICR should have discussed the reasons for the positive assessment of safeguards and financial management . There are a number of inconsistencies between the ratings discussed in the text and the overall rating given . Finally, the analysis of efficiency could have been strengthened by drawing on secondary research as well as by calculating the project's comparative cost effectiveness . a.Quality of ICR Rating : Satisfactory