ICRR 13352 Report Number : ICRR13352 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 04/13/2010 PROJ ID : P075978 Appraisal Actual Project Name : St. Kitts & Nevis US$M ): Project Costs (US$M): 6.92 7.77 Education Development Project Country : St. Kitts & Nevis Loan/ US$M): Loan /Credit (US$M): 5.00 4.84 Sector Board : ED Cofinancing (US$M): US$M ): Sector (s): Secondary education (64%) Tertiary education (20%) Central government administration (16%) Theme (s): Education for the knowledge economy (33% - P) Participation and civic engagement (33% - P) Social analysis and monitoring (17% - S) Education for all (17% - S) L/C Number : L7125 Board Approval Date : 06/13/2002 Partners involved : Closing Date : 09/30/2008 09/30/2009 Evaluator : Panel Reviewer : Group Manager : Group : Silke Heuser Susan A. Stout IEGSE ICR Reviews IEGSE 2. Project Objectives and Components: a. Objectives: According to the PAD, the overall objective for the OECS (Organization of Eastern Caribbean States ) program was to build human capital to in turn help diversify the Caribbean economy and lead to more sustainable growth . This objective was to be achieved by : (i) increasing access to secondary education; (ii) improving the quality and efficiency of secondary education; and (iii) strengthening the institutional capacity of the education sector . The three objectives stated in the PAD were consistent with those stated in the Loan Agreement dated September 26, 2002. b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): Project components for the OECS Education Development Program (OEDP) as listed in the PAD were the following: (1) Increasing equitable access (Appraisal: US$2.72 million; Actual: US$3.69 million): This component included the construction of a new secondary school in the rural area of Saddlers and the provision of equipment, furniture and materials; it also included the introduction of cafeterias in secondary schools in St . Kitts. (2) Improving the quality of teaching and learning (Appraisal: US$2.56 million; Actual: US$3.48 million): This component included four sub -components: (a) The development of a curriculum for forms I -III (grades 7-9) and the provision of instructional guides and textbooks, as well as training of educators, including fellowships, and the development of a coordinated teacher appraisal system . (b) The establishment of learning resource centers, including science and IT labs, in seven schools in St . Kitts and Nevis. (c) The elaboration of a program to support participation of disadvantaged students and related teacher development in special education, and (d) the organization of school-based improvement projects and extra -curricular activities. (3) Improving governance and management in the education system (Appraisal: US$0.26 million; Actual: US$0.002 million): This component included the expansion of the Education Management Information System (EMIS), including networking of schools and EMIS training . (4) Project management (Appraisal: US$0.60 million; Actual: US$0.60 million): Establishment of an efficient Project Management Unit (PMU). Component costs exclude contingencies and front -end fees amounting to US$0.78 million. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Financing : The project was part of an "horizontal" Adaptable Program Loan (APL) applied sequentially to countries in the OECS sub-region. The Education Development Program supported the education sector of countries in the OECS (Saint Lucia, St. Kitts and Nevis, Dominica, Grenada and Saint Vincent and the Grenadines ). Each country was to join the program when it met the explicit criteria for participation . The first projects, as discussed in the PAD, covered St. Kitts and Nevis and St. Lucia. Projects for both countries had separate loan /credit agreements. ICRs were prepared separately for St . Kitts and Nevis and St. Lucia, because the St. Lucia segment had already closed one year earlier on 09/30/2008 date. The ICR for the St. Lucia project (P077712), was reviewed by IEG in September 2009. The project outcome was rated Moderately Satisfactory . Costs The total budget of the project was reduced from US$ 9 million to US$6.92 million at appraisal. Although Costs: corresponding adjustments were made to the scope of activities, the project budget became extremely tight during implementation due to an increase in construction costs which caused significant cost overruns, further reducing the scope of some activities. Under component 3 only US$2,000 was spent instead of US$260,000, because the EMIS system was not procured. Dates The original implementation schedule was ambitious . Most of the activities experienced delays including Dates: construction of the new school : the school was planned to be fully operational by 2004, but it opened in September 2009. The closing date of the project was extended by nine months from September 30, 2008 to June 30, 2009 in order to complete the project activities, including the construction of the new school and the provision of equipment and furniture to school facilities . 3. Relevance of Objectives & Design: high The project’s objectives, improving access and quality of secondary Relevance of project objectives is rated high: education as well as expanding the Education Management Information System (EMIS) system for the Ministry of Education (MOE) were highly relevant at the time of project appraisal and they remain highly relevant today . Project objectives are in line with several statements of sector strategy, including the St . Kitts and Nevis Long-term Education Plan covering 1998-2011, and the 2000 Caribbean Education Strategy Report No . 20452-LAC. These documents were prepared through a participatory approach which engaged civil society in discussions of how to achieve national, regional and international targets in education . Furthermore, the country’s 2006 National Information and Communication Technologies (ICT) Strategic Plan emphasized the need for effective EMIS as well as further promotion of ICT in teaching and learning . Project objectives were also relevant to the latest 2005 OECS Country Assistance Strategy (CAS). The 2005 CAS affirms the Bank’s continued support of education sector reform in the OECS countries, with particular reference to skills enhancement and improving the quality of secondary education . ii) Relevance of project design is rated substantial : The PAD included a detailed results framework with clearly (ii) stated objectives that were linked to time -bound targets at both the output and, in part, at the outcome levels . The causal chain between Bank funding and intended outcomes was clear and convincing for components one and two, but the outcome indicators for component three were not clearly conceptualized . 4. Achievement of Objectives (Efficacy): Objective (1): Increasing access to secondary education is rated substantial . Output: Output A new secondary school was constructed and equipped in Saddlers, a poor rural sugar cane area in St . Kitts, which created 250 new secondary school places (compared with the original target of 350 new school places). Basic equipment and furniture were provided for cafeterias in three secondary schools (compared with the target of six schools). Free meals for 180 eligible students were not made available as originally planned because private cafeteria operators were not assured of able to make sufficient profits by project closure . Outcome : The new secondary school opened in September 2009 rather than in 2004, with 70 students enrolled in the first year of operation to be followed by more classes in the coming years . This will further contribute to decongestion of neighboring secondary schools . The country’s total number of grade 11 students increased by 22 percent, from 550 in 2001/2002 to 670 in 2007/2008. However, this increase in enrolment cannot be attributed to the construction of the new secondary school, as the first grade 7 students were not enrolled in the school until September 2009. Equity indicators for the new school were not available at the time the ICR was written, but the PAD’s underlying assumption that schools built in poor areas would be used by poor families in these areas was plausible. While the new school decongested two neighboring secondary schools by 39 percent and 13 percent respectively, it is too early to evaluate outcomes with respect to access and efficiency . The three cafeterias are in operation and offering more nutritious and cheaper meal options . According to focus group discussions conducted by the ICR team, students, teachers and principals mentioned that cafeterias positively influence students' attendance in afternoon sessions. Objective (2): Improving quality and efficiency of secondary education is rated modest . Output The curriculum for the lower secondary school was revised as planned for most subjects by June 2009 Output: (against the target of 2004); common end-of-year exams were developed for three subjects; and equipment and materials to support the curriculum were purchased and are in use at the school level . Five teachers completed post-graduate training in five subject areas . A total of 156 teachers (about half the number of teachers on St . Kitts and Nevis) were trained in curriculum writing and delivery in various subject areas . The original target was, however, to train 200 teachers in Information Technology (IT) and other emerging topics. Eight persons were trained as librarians and eight as IT technicians, 50 percent of the target of 32 teachers trained in the use of Learning Resource Centers (LRCs). A Teacher Appraisal System (TAS) was developed and approved during implementation and has been in use since academic year 2005/06, later than against the original target of 2003. Over 90 percent of teachers have been evaluated. In addition, six LRCs were refurbished under the project and IT, science labs, and home economics rooms were upgraded by 2009 (against a target of 2008). However, maintenance of computers and electricity fluctuations remain a challenge; plus, computers are not fully utilized internet connection remain weak or non-existence in "some" of St. Kitts’ nine secondary schools . In addition, not all schools have an IT technician, and maintenance capacity for IT equipment is inadequate . Support services for disadvantaged students were developed with reading recovery programs in place in 17 primary schools (or 53 percent of all primary schools in St . Kitts and Nevis), and 52 teachers were trained in reading recovery and dyslexia by 2007 (against a target of 40 by 2006). Last, the project supported a total of 57 School Improvement Projects in 10 schools and 21 Extra-Curricular Activities in seven schools, which helped integrate students who had been considered at -risk of dropping out. Outcome : Overall, the project contributed to improved reading skills for disadvantaged students, student -centered learning environments, and relevance of the curriculum . The ICR measured the objective of improving quality of teaching and learning by (i) gross completion rate for Form 5 (grade 11) and (ii) the proportion of students passing at least five subjects in the secondary school leaving examination CXC (the Caribbean Examination Council). In 2009, the gross completion rate is estimated at approximately 85 percent, meeting the target (against the 2001 baseline of 70 percent). However, this achievement can hardly be attributed to the project because of the slow start of the activities in 2006 and 2007. Furthermore, the proportion of students passing at least five CXC subjects, including Mathematics and English, decreased from 26 percent in 2000 to 18 percent in 2009, half of the target of 36 percent. However, it is too early to measure the outcome of improved quality of teaching and learning with the above indicator, since it takes at least five years for the students who study only under the new lower secondary curriculum to sit for the CXC exams and the curriculum was introduced only in June 2009. Objective (3): Strengthening the institutional capacity of the education sector is rated substantial . Output: The acquisition of EMIS software was dropped due to a failed regional attempt to procure common EMIS Output software and high maintenance costs . Nevertheless, e-mail communication and broadband connections were established for all public schools . In addition, simple instruments for data collection were developed with the help of UNESCO. Finally, training was carried out for school principals (instead of MOE staff and education officers as planned) on the use of electronic tools and data collection instruments . Outcome : Data collection capacity was enhanced . Basic school data such as student enrolment and teacher employment is being collected from schools on a timely basis, and time lags for reporting from schools dropped from over two years in 2003 to less than a year in 2008. A Statistical Digest was published for the school year 2007/2008, which contained key education data at all levels such as enrolment, CXC results, and number of trained teachers . However, according to the ICR, p . 17, there systematic utilization of data to guide policies and programs remains weak. 5. Efficiency (not applicable to DPLs): Efficiency is rated modest The construction program was affected by the rising costs of construction material and labor, as well as reduced scope and delays, which undermined the project ’s cost-effectiveness. A cost-benefit analysis was undertaken at appraisal, but no recalculation was carried out for the ICR . According to the appraisal analysis, Net Present Value (NPV) was estimated to be approximately US$ 14.7 million and the rate of return 35 percent. Individuals with complete secondary education would earn an additional US$ 16,171 during their lifetimes compared to those with only primary education . Thus, the main benefits of the project were thought to lie in the increased number of secondary graduates (produced either through increased access or fewer dropouts ) and in substantial quality gains, which would both be reflected in higher future earning streams . The ICR reports that dropouts from secondary school were minimal during project implementation, but the absence of baseline data makes it difficult to assess this observation . However, the pass rate declined and was much lower than in the PAD assumptions, so that the benefits were likely smaller than estimated at appraisal . Furthermore, the NPV is lower than estimated because of the cost escalation, the fact that the Government and the Bank reduced the size of the new school in Saddlers, and the fact that the number of cafeterias and the scope of rehabilitation work were reduced in two additional schools. Overall, the physical output of the new secondary school and the cafeterias accounted for 47 percent of total project costs, which is in line with the fiscal year 2000-2004 average of 53 percent spent on civil works for all World Bank-financed education projects in that period . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal Yes 35% 0% ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Outcome is rated moderately satisfactory . The project was relevant in that it responded to a regional education reform effort to improve the skill level in OECS countries . In addition, a new secondary school was constructed in a rural area of St. Kitts, allowing for improved access, better education quality, and closer proximity to school . The lower secondary school curriculum was revised, teachers trained, and more student -centered learning opportunities were created, despite substantial delays relative to the original project schedule . In addition, the education sector ’s institutional capacity was strengthened . Project efficiency was reduced because of rising costs of construction materials, reduced scope of construction, rehabilitation, equipment installation, and delays . Efficiency was also reduced in light of lower than anticipated outcomes . a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: Risk to development outcome is rated significant . Financial crisis : The global financial crisis since 2008 has had a negative impact on the country and the Government ’ s budget, as evidenced by the increase in the Bank ’s financing share in the last few months of the project to 100 percent of total project cost . The education budget for FY 2010 was cut by about five percent . Therefore, fiscal constraints could affect the operations of schools, and activities such as school improvement projects may be discontinued for lack of funds . Maintenance : Maintenance costs for school facilities and equipment pose another challenge . A maintenance policy has been prepared and a maintenance manual will be developed and implemented in all schools and colleges by 2012. While this has been highlighted in the budget policy statement as well, provision of sufficient funds to maintain all the infrastructure and equipment is uncertain . Natural disasters : St. Kitts and Nevis is vulnerable to natural disasters, including frequent hurricanes . While newly constructed or rehabilitated buildings were built to withstand hurricanes, damage can still be caused to school facilities. Teacher retention : A risk that has not been mitigated during the project is teacher retention . Many teachers have been trained in subject matters, reading recovery, and IT skills . The risk remains that trained teachers will move on to other jobs or leave the country, endangering the sustainability of reform and quality improvements in secondary schools. a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: at -Entry: a. Ensuring Quality -at- Entry This review rates quality-at-entry as moderately satisfactory because of its high relevance to the needs of the country and the sub -region, though with minor shortcomings with respect to the results framework. A Quality-at-Entry Assessment (QEA) by QAG which was conducted for this project in 2002 (after appraisal) rated overall quality-at-entry and key elements of the design as satisfactory . The QEA noted the high level of Government ownership, but also pointed out that preparatory studies did not have any significant impact on the design of the project . An horizontal APL was an appropriate choice of lending instrument, since it supported regional integration, and worked well with respect to achieving curricular reform . The horizontal APL proved less appropriate for the procurement and operation of the proposed EMIS since St . Kitts and Nevis and St. Lucia were unable to coordinate procurement activities to support the implementation of the EMIS as originally intended . In terms of scope, project design included multiple activities which were not implemented due to the cost pressures on civil works which limited funds available for these activities . As a result, the scope of construction of the new school in Saddlers, the number of cafeterias, and the rehabilitation work at Sandy Point and Cayon High Schools were reduced. Cost estimates at appraisal for the new secondary school and the cafeterias were too low . During implementation, the project experienced cost overruns of over 100 percent of the original price estimate . Although the major factor for the cost overrun was the increase of the cost of building materials and labor that affected the whole sub-region, the original scope of activities was quite ambitious given the limited funds available, particularly for the civil works component . b. Quality of Supervision : Quality of supervision is rated moderately satisfactory . The Bank provided intensive technical support and training to the PMU to enhance and sustain its capacity, and the staffing of the PMU was closely monitored. In addition, the team facilitated cross -country learning by conducting sub -regional workshops on architectural design of schools and EMIS . However, at times the PMU received mixed messages from different Bank officers that confused the project team and added to the delays . When this became clear, the Bank took appropriate action to address these issues . at -Entry :Moderately Satisfactory a. Ensuring Quality -at- b. Quality of Supervision :Moderately Satisfactory c. Overall Bank Performance :Moderately Satisfactory 9. Assessment of Borrower Performance: a. Government Performance : Government performance is rated satisfactory . The Government showed a high level of ownership and consistent commitment to education reform and the project throughout the implementation period. Counterpart funds were made available in a timely manner . The Government also financed all additional construction costs arising from significant price escalation (over 100 percent of the original price estimate ), making the new secondary school possible . The global financial crisis forced the Government to cut the budget and request an increase in the Bank ’s financing share during the last months of implementation . b. Implementing Agency Performance : PMU performance is rated moderately satisfactory . The PMU and a Project Advisory Committee (PAC) were established to coordinate and direct the implementation under the supervision of the Permanent Secretary of the Ministry of Education . The PMU’s implementation capacity was a critical issue during the first half of the project, which delayed the implementation of most of the components and disbursements. Insufficient supervision of works and communication between the PMU and the contractors caused delays in civil works and affected the quality of some of the works . Procurement also lagged behind, which caused frustration among stakeholders . In response, the roles of the Permanent Secretary, the PAC, and the heads of sub-components was expanded in order to facilitate implementation . This change of the project management arrangement as well as staffing restructuring of the PMU improved implementation and resulted in successful completion of the activities . a. Government Performance :Satisfactory b. Implementing Agency Performance :Moderately Satisfactory c. Overall Borrower Performance :Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: M&E Design : The project’s results framework was sound . In addition, the QAG review pointed out that drawing lessons on the effectiveness of the project's software dimensions would require a considerable amount of qualitative assessment and evaluation which proved to be the case . M&E Implementation : Performance indicators were tracked during Bank supervision missions . However, the ICR team had difficulty in acquiring accurate data for several results indicators, mainly because the population projections were not updated regularly. In addition, output indicators on the number of schools connected to the internet could have been more specific. Nevertheless, the project did contribute to improvements in the information collection processes in the sector . M&E Utilization : The ICR provides very little information about utilization of data to inform decision -making and resource allocation. However, it indicates that the Government is working on a sector plan that links key education sector indicators to the budget . a. M&E Quality Rating : Modest 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): Safeguards : The project was assigned to Category B (potential adverse environmental impacts on human populations or environmentally important areas ) because it included the new construction of a school . An Environmental Impact Assessment (EIA) was carried out for the new school site in Saddlers prior to negotiations . The EIA recommended, among other specifications, using hurricane -resistant design standards, which were incorporated in the building. Construction of the new school also triggered the resettlement safeguard, since the school site had been occupied by two large farming plots . The Ministries of Education and Agriculture relocated the affected farmers to appropriate sites that are equally accessible and comparable to their former plots . The ICR does not provide further information. Procurement : The project encountered difficulties with procurement, leading to significant delays . The situation improved as the PMU staffing was restructured in 2005, and procurement was conducted in accordance with Bank guidelines according to the ICR . 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Satisfactory Satisfactory Risk to Development Significant Significant Outcome : Bank Performance : Moderately Moderately Satisfactory Satisfactory Borrower Performance : Satisfactory Moderately Lower rating due to delays in project Satisfactory implementation and weak implementation capacity at project start. Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: When establishing IT labs in secondary schools, ensure that teachers are familiar with integrating the use of computers into their lessons. In this project, the actual number of teachers trained in the use of computers was much smaller than the target, which reduces the utility of the equipment . Realistic cost estimate and scope of work are important . The project experienced significant cost overrun . This led to the reduction of scope of the activities, which caused additional delays . Although the major factor for the cost overrun was the increase of the cost of building materials and labor that affected the whole sub-region, the original scope of activities was also quite ambitious given the limited funds available, particularly for the civil works component . A balance between sub-regional and country-specific approach is warranted . The project adopted a sub-regional approach under the OEDP APL to tackle common problems and to benefit from economies of scale. This approach facilitated knowledge sharing among OECS countries on educational issues and was particularly effective during the preparation stage, although much less so with respect to implementation . While these countries share many of the problems and objectives, each country's priorities toward the objectives varied significantly due to different levels of advancement . Applying a single set of objectives to five countries has brought some challenges in supporting their educational needs, and weakened the link between project activities and PDO indicators . Flexibility in implementation arrangements would enhance the project effectiveness by accommodating individual differences while benefiting from knowledge sharing around common themes. At the same time, this needs to be reconciled with the benefit from economies of scale by closely looking at the timing of activities that can be implemented simultaneously . A maintenance strategy needs to be discussed at the project preparation stage with the involvement of the Ministry of Finance and other key stakeholders as it has long -term financial implications. The project built a new school, cafeterias, Learning Resource Centers, and IT labs . It also provided computers to expand the use of IT in teaching and learning. 14. Assessment Recommended? Yes No Why? To verify the ratings. Also, a comparison of the regional approach taken for St . Kitts and Nevis and St. Lucia, which both completed secondary education reform projects with IT objectives in the education sector, could inform the ongoing ICT evaluation. An assessment would be informative, especially given St . Kitts and Nevis’ 2006 National ICT Strategic Plan, which emphasized the need for effective EMIS as well as further promotion of ICT in teaching and learning. In addition, the an assessment of the effectiveness of the horizontal APL in achieving economies of scale while adapting to country -specific circumstances could inform future projects as well as an evaluation IEG is planning on small states in the coming fiscal year . 15. Comments on Quality of ICR: The ICR quality is satisfactory; the document provides the information to understand what happened during project implementation, although it could have been more concise . There have been discrepancies in total project costs at appraisal between the PAD and the ICR : Component amounts as listed in the PAD on p . 13 and pp. 44-53 differ from the ones listed in the ICR on p. 27. In addition, appraisal estimates in the ICR on p . 27 do not add up to project costs, unless the front-end fee of US$0.05 million is added under the total baseline cost . In addition, the ICR (p. 33) could have been more specific in providing the exact number of schools with the internet not yet connected . A meeting took place with the task manager in which data issues, teacher retention, and the regional program approach were discussed . a.Quality of ICR Rating : Satisfactory