ICRR 13187 Report Number : ICRR13187 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 02/04/2010 PROJ ID : P081269 Appraisal Actual Project Name : Second Education US$M ): Project Costs (US$M): US$27.2 M US$34.34 M Sector Development Project (phase 2) Country : Lesotho Loan/ US$M): Loan /Credit (US$M): US$21.0 M US$22.7 M Sector Board : ED Cofinancing (US$M): US$M ): Sector (s): Primary education (52%) Secondary education (20%) Central government administration (20%) Vocational training (5%) Tertiary education (3%) Theme (s): Education for all (40% - P) Administrative and civil service reform (20% - S) HIV/AIDS (20% - S) Education for the knowledge economy (20% - S) L/C Number : C3813 Board Approval Date : 07/17/2003 Partners involved : n/a Closing Date : 12/31/2007 12/31/2008 Evaluator : Panel Reviewer : Group Manager : Group : Victoria Monchuk Ridley Nelson IEGSE ICR Reviews IEGSE 2. Project Objectives and Components: a. Objectives: The Project Appraisal Document (PAD) defined the PDO to be “to further improve the access, equity, and quality of primary and secondary education, promote lifelong learning through building capacity in early childhood, technical and vocational, higher and non -formal education, and strengthening the institutional capacity of the Ministry of Education and Training� . The wording of the objective is slightly different in the credit agreement but the objectives are the same. The project was the second phase of an Adaptable Program Loan (APL). The overall objective of the APL was to produce more and better educated Basothos and enabling them to participate in local and regional labor markets. b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): Component Description (from PAD) Costs at Actual appraisal, costs,US$M US$M (IDA (IDA finance) finance) Primary education Primary classroom construction; targeted equity based program 14.18 (11.47) 15.01 (9.76) (bursaries for orphans and disadvantaged children); procurement and distribution of teaching and learning materials; training of primary teachers, inspectors and advisers and school management staff including refresher courses and distance teacher education; and basic education curriculum development and implementation of standardized examinations. Secondary education Secondary classroom construction; targeted equity based 5.95 (4.18) 10.28 (8.09) program (bursaries for orphans and disadvantaged children); develop textbook rental scheme and revolving fund; and continuous training of secondary teachers, inspectors and advisers and school management staff. Lifelong learning in Establish Early Childhood Care and Development (ECCD) 2.16 (1.83) 1.53 (1.05) early childhood care home-based centers and support ECCD unit’s role for and development, providers; continue phase I Technical and Vocations Education technical and and Training (TVET) activities including reorienting the sector to vocational training, be demand-driven and implement the TVET policy; transform higher and non-formal higher education to be more cost-efficient and responsive to education development goals while maintaining quality; and continue to support the phase I Learner Post non-formal education (NFE) pilots. Ministry of Education Strengthen district management and MoET’s capacity in policy 3.60 (2.46) 3.39 (1.75) and Training (MoET) development, planning, monitoring and evaluation; provide institutional MoET staff development and training; and strengthen the MoET strengthening to deal with the urgent challenges of HIV/AIDS. Project Project support and coordination unit, salaries and benefits, 1.31 (1.06) 4.14 (1.29) implementation technical assistance, training and procurement. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: The project’s total financing increased by over US$ 7 million from the appraisal estimate due to favorable exchange rates. At the request of the government and approved by the Bank there was a reallocation of US$ 4 million in IDA finance from primary (school construction) to secondary education (textbooks) in 2005 and a further reallocation of over US$1 million to training of staff in 2007. Moreover, the unplanned expansion of the temporary Project Support and Coordination Unit (PSCU) increased the operating costs almost fourfold, financed through government counterpart funds. The project closing was extended for one year to December 31, 2008 to allow for the completion of some activities. Otherwise progress on project activities and disbursement were generally according to schedule . Total IDA financing was estimated at 77 percent of project costs at appraisal and stood at 66 percent at closing. The Government of Lesotho financed the remaining parts . The project (phase II of the three-phased 1999-2011 Second Education Sector Development Project APL ) comprised 31 percent of overall estimated program IDA funding . 3. Relevance of Objectives & Design: Relevance of objectives : Government commitment to education is strong in Lesotho . Advances in the education sector, especially technical skills development, are one of the government ’s means of achieving increased employment and growth (CAS, 2006, most current). The main sectoral issues in basic education are stagnant primary and low secondary enrollment, low completion rates, large socio -economic disparities, and poor school facilities. Large groups of unschooled children (e.g. orphans, herd boys) have accumulated. Moreover, secondary level school fees and higher education costs are high, the TVET program needs to be more demand driven to cater to the labor market, and Non-Formal Education (NFE) needs to be refocused away from catering to youth drop outs towards building skills of illiterate adults . The government’s strategy is identical to the project development objectives. The focus on equitable access and quality of primary and secondary education is consistent with the country’s needs and the sectoral priorities . Against this background the relevance of objectives of the project were Substantial . Relevance of design : The two first components were designed to meet the project objectives in primary and secondary education. The design was adequate but with some weaknesses . At the primary and secondary levels the focus on inputs such as school construction, textbook provision, teacher and supervisor training, and targeting disadvantaged children was consistent with efforts to promote both equitable access and quality improvements . The selection of poor localities for school construction and providing bursaries to disadvantaged children agreed with the focus on reducing inequalities . However, the quality indicator at the primary level (test scores) was unreliable and there were no indicators for measuring quality at the secondary level so achievement was measured in terms of outputs produced. No real measures of how to evaluate equity improvements in access to primary and secondary schooling were set. The third component was designed to address the project's objectives in life -long learning in ECCD, TVET, higher education and NFE. Building the skills of the work force and linking training to labor market needs is a priority in Lesotho but the resources provided in this project were not sufficient for advancing capacity in life-long learning. Only 8 percent of total project costs were allocated to activities to support ECCD, TVET, higher education and NFE. There were also important weaknesses in the results framework for measuring capacity building in the four areas. The fourth component aimed at building MoET's capacity and had similar weaknesses in the results framework. No indicators for measuring achievement of objectives related to the third component were established . In sum, the design and results framework supporting the primary and secondary education objectives --the primary focus of the project--was adequate albeit not strong . The designs of components 3 and 4 were much weaker. Therefore, relevance of design was rated Modest. 4. Achievement of Objectives (Efficacy): Objective 1: Further improve the access, equity, and quality of primary and secondary education . Rating: Substantial Access (substantial) Outcomes: During the project timeframe, primary enrollment dropped (from 85 to 82%) but completion rates increased significantly (from 77% to 90%). Secondary enrollment increased beyond targets (43% to 53%). At the same time, the staggered Free Primary Education (FPE) policy moved up in Standards (grades) and may have contributed to the increase in secondary enrollment, especially in poor areas, but also to the overall drop in primary enrollment. The parallel effects of the FPE make it difficult to separate out the contributions of the project to changes in enrollment rates. Outputs:The anticipated new schools were constructed to good standard and were targeted in areas otherwise under-served by education services and where demand was high . Equity (substantial) Outcomes: There are no data on the number and gender of enrolled children from disadvantaged backgrounds but Bank missions verified that enrollment and attendance of the scholarship children were strictly enforced by project district officers. It is likely that access by the poor and disadvantaged in both primary and secondary improved as a result of the project. Outputs: Bursaries reached over 20,000 primary and 20,000 secondary children beyond targets (15,000 in each level). Quality (modest) Outcomes: Primary achievement rates were mixed but it should be noted that test scores were highly unreliable and problematic. Given that outcomes take a few years to materialize it is difficult to judge to what extent the project increased quality. Outputs: Text books were printed and provided, one per child, in primary education . In secondary education it is still unclear if the revolving text book fund that was established is functioning well as it takes a few years for the rental scheme to circulate through. The intention was that costs would be reduced by renting out books . Over time, parents would pay one fifth of the books per student . Refresher and distance training to teachers and staff was largely achieved although there was only limited training to inspectors . Distance training was given to teachers in remote areas while teaching. Curriculum development and strengthening of assessment was largely completed . Objective 2: Promote lifelong learning through building capacity in ECCD, TVET, higher education and NFE . Rating: Negligible The two indicators to monitor progress outside of primary and secondary education were linked only to two of the four sub-sectors (ECCD and TVET) under this objective. Furthermore, the indicators did not reflect capacity building .The ECCD coverage improved but did not reach targets (from 31% to 36%, target 40%). A draft TVET strategy bill was prepared but not presented as anticipated during the project . The anticipated TVET financial management system, monitoring system, and innovation fund were not completed but the hostel for TVET students was constructed as planned. TVET reform continues to be difficult in Lesotho with low capacity and no experience of reform in this sector. Although no project performance indicators were established for higher education the expected outputs were not achieved. The NFE work was not evaluated in the project and performance was unclear . In sum, despite progress the indicator targets were not met and the majority of the planned outputs we re not achieved. Objective 3: Strengthen the institutional capacity of the Ministry of Education and Training . Rating: Modest The project reports on the share of sub -sectoral budget allocation for primary and secondary education which is not a useful indicator for measuring achievement of the objective of building institutional capacity of the MoET . The target of allocating 65 percent of the budget to primary and secondary education was most likely not achieved . In 2007 the allocation had only increased to 61 percent. To strengthen the MoET’s capacity the project planned to train and organize study tours for district level staff; provide new equipment for 10 District Resource Centers (DRCs); review the Education Sector Strategic Plan; improve donor coordination; strengthen the Planning Unit; and train MoET staff . At completion the project had provided equipment to DRCs and trained 3 staff in the Planning Unit and other MoET staff. An HIV/AIDS coordinator was put in place in the MoET but only to leave after project closing . Only some of the planned activities mapped to the objective of improving the MoET's capacity were achieved . The ICR states that MoET has been strengthened in human resources and in coordination between the PSCU and the MoE T. Progress was made on the elaboration of the Lesotho Education Sector Strategy and establishment of Joint Technical Review meetings for discussing reform. However, MoET capacity continues to be weak and the objective ha s only partly been achieved. 5. Efficiency (not applicable to DPLs): The ICR does not estimate cost -effectiveness or unit costs per student . This is a weakness of the project as many outputs, especially in components 1 and 2 could have been costed and economic benefits could have been calculated. The project could have compared costs and expected benefits to other project activities in comparable countries. According to the project task team costs in the education sector in Lesotho are generally high . Text books and school buildings were expensive and project management staff were very highly paid . Therefore, efficiency was rated Modest. ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The shortcomings discussed in sections 3, 4, and 5 above amount to significant shortcomings . a. Outcome Rating : Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: Progress on primary and secondary education outcomes is being made in Lesotho . The key primary and secondary components of the project were in operation by the MoET after project closing and the risk to development outcomes in primary and secondary education are therefore low . Nevertheless, efforts need to be made to understand the drop in primary enrollment and any changes in education quality and test scores . Moreover, the work on examinations that started in phase II needs to be significantly strengthened to monitor outcomes . On the contrary, progress is slow on capacity building and in reforming and implementing operations in ECCD, TVET, higher education and NFE. Capacity in the districts and the MoET remains weak . The recent attention to strengthening TVET is promising but unless the reform objectives can move ahead quickly based on the outputs produced in phase II there is a high risk that Lesotho will not be able to meet its objectives in equipping its population with skills for competing on the labor market. Politically the project objectives have strong support but capacity to plan, monitor and financially manage developments in the education sector in Lesotho remains weak . Overall, risk to development outcomes is Moderate. a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: Quality at Entry . Quality at entry was Moderately Satisfactory . This phase II of the APL was designed very soon after the end of the first phase even though only 3 of the 6 triggers for moving from phase I to phase II were achieved. The Bank’s knowledge of the conditions in the sector was substantial given the prior phase and lessons were incorporated. There were two key weaknesses of the design of the second phase . These were poor attention to the sub-sectors outside of primary and secondary education and a mismatch between the project objectives and the indicators . With limited capacity in the MoET and lower than expected progress in phase I, the Bank could have aimed for a less ambitious plan . Supervision . Quality of Supervision was Satisfactory . Throughout the project the Bank worked closely and proactively with the MoET, provided advice and drew on technical experts for guidance . However, the team did not alter the performance indicators during implementation even though they were not well matched with the development goals and did not attempt to focus on cost efficiency although costs were seen to be high . On balance and given the Moderately Unsatisfactory project outcome, Bank performance is rated Moderately Satisfactory . at-Entry :Moderately Satisfactory a. Ensuring Quality -at- b. Quality of Supervision :Satisfactory c. Overall Bank Performance :Moderately Satisfactory 9. Assessment of Borrower Performance: Government Performance . Government performance is rated Moderately Satisfactory . Government and the MoET showed strong commitment to education for improving the technical skills of the country ’s labor force. But attention to TVET, higher education and NFE was lacking despite pressure from the Bank . Poor M&E design and performance indicators also factor into the government performance rating . Policy development and institutional building did not take place to the extent planned and the low prioritization of meeting the second and third development sub-objectives contributed to poor project outcomes . Implementing Agency . Implementing agency (MoET) performance is rated, on balance, Satisfactory . Implementation of the primary and secondary education components was largely achieved although not without weaknesses. Overall Borrower Performance was Moderately Satisfactory . a. Government Performance :Moderately Satisfactory b. Implementing Agency Performance :Satisfactory c. Overall Borrower Performance :Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: M&E design was poor. The 7 indicators established were not aligned with the objectives but linked to performance in the sector as a whole. Important indicators of equity and secondary quality were missing . Outside of primary and secondary, indicators were established for measuring progress only in ECCD and TVET . For higher education and NFE, the project expected to increase capacity but without indicating how this would be measured and evaluated . No indicators for measuring achievement of building capacity in the MoET were established . Without comprehensive, clear and time-bound output indicators (and establishing links to outcomes ) it is unclear how the objectives would be achieved. Moreover, many output targets were poorly defined and difficult to measure . “Capacity building� was difficult to measure based on the outputs . M&E design was heavy on monitoring but lacked a rigorous evaluation system for outcomes. Instead it relied on national assessment data of student learning which were new and unreliable. M&E implementation progressed well in the early stage but the attrition of skilled staff from the PSCU in the end limited data collection and processing . Some data from 2007/08 are still not available. Utilization of the data helped the team to monitor progress and suggest when special studies were needed . Mainly because of the poor design of the performance indicators that were detrimental to project ratings, M&E quality is rated Modest. a. M&E Quality Rating : Modest 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): There were no particular issues with safeguards or fiduciary performance . The project achieved two unintended impacts. As a result of the weak progress on strengthening TVET new Bank AAA resources have been dedicated to issues of building technical and vocational skills . Also, the Lesotho Education Sector Strategy 2005-2015 has been elaborated. 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately The shortcomings discussed in Satisfactory Unsatisfactory sections 3, 4, 5 above amount to significant shortcomings. Risk to Development Moderate Moderate Outcome : Bank Performance : Satisfactory Moderately Quality at Entry is rated Moderately Satisfactory Satisfactory as there were moderate shortcomings in design (please see section 8). Borrower Performance : Moderately Moderately Satisfactory Satisfactory Quality of ICR : Unsatisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: A broad project scope can put the outcomes of secondary objectives and small components at risk since they may not be provided sufficient funding and government attention. In this case, the strong focus on procuring goods and structures for primary and secondary education left the reforms envisioned in the other components without resources or needed attention . Institutional capacity building should be viewed as an investment that can be sustained for the future . Training courses will only increase institutional capacity if staff remain in the workplace and if the institutional incentives are conducive. In this case, high staff turn over and the temporary PSCU solution for managing the project did not help build the capacity in the ministry . Objectives-based performance assessment needs to build on indicators that are strongly linked to project objectives and with measurable outputs and outcomes . In designing indicators, where possible, it is important to be able to distinguish results produced by the project and results generated by related government reforms . 14. Assessment Recommended? Yes No Why? To better assess outcomes and verify ratings, given the absence of adequate evidence in the ICR . 15. Comments on Quality of ICR: As acknowledged in the ICR, the ICR was handicapped due to poor data availability and a weak results framework of the project. However, despite data limitations, there were substantial weaknesses in the ICR . The ICR did not evaluate project performance against the same objectives as listed in the PAD and the DCA . Instead, the ICR reduced the objectives in primary and secondary education to enrollment, completion and primary learning achievement. Equity and secondary education quality, which are part of the development objective, were not well assessed. For instance, there are no data on gender and socioeconomic status of bursary recipients to know whether disadvantaged children attended school to a greater extent as a result of the project . The ICR also did not attempt to clearly measure achievement on strengthening higher education and NFE which was part of the development objective. The ICR focused on outputs instead of outcomes and did not discuss the results chain through which inputs and outputs have /have not lead to outcomes. The ICR also did not adequately assess efficiency. a.Quality of ICR Rating : Unsatisfactory