ICRR 12600 Report Number : ICRR12600 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 05/08/2007 PROJ ID : P003249 Appraisal Actual Project Name : Basic Education US$M ): Project Costs (US$M): 340 224 Subsector Investment Program (BESSIP) Country : Zambia Loan/ US$M): Loan /Credit (US$M): 40 37 Sector Board : ED Cofinancing (US$M ): US$M): 300 187 Sector (s): Primary education (59%) Central government administration (20%) Tertiary education (10%) Sub-national government administration (10%) Health (1%) Theme (s): Education for all (25% - P) Rural services and infrastructure (25% - P) Access to urban services and housing (24% - P) Child health (13% - S) Gender (13% - S) L/C Number : C3190 Board Approval Date : 04/08/1999 Partners involved : DANIDA, DfID, Irish Closing Date : 06/30/2002 07/11/2006 Aid, Netherlands, NORAD, EU, Finland, UNICEF, USAID, ADB, JICA, OPEC, SIDA Evaluator : Panel Reviewer : Group Manager : Group : H. Dean Nielsen Kris Hallberg Alain A. Barbu IEGSG 2. Project Objectives and Components: a. Objectives: Original from the Development Credit Agreement: "Support the Program which aims to increase the gross enrollment ratio in Grades 1-7 to 100% by the year 2005 and to narrow the gap between gross and net enrollment while improving learning achievement in those grades ." The first two of the original three objectives were inappropriately presented in term of indicators and not objectives (which, for the first was presumably to increase basic school access and the second to improve internal efficiency ), but the PAD version of the first objective is very vague ("provide relevant education for all children in the age range ") and for the second nonexistent . For this reason, the DCA version will be used here . The third objective, improving learning outcomes, is appropriately stated, appearing in both the DCA and PAD versions . Given the deep poverty in Zambia, the PAD version appropriately called for improved access "particularly for disadvantage groups, " but this is implicit in the DCA-stated goal of enrolling 100 percent of grade 1-7 children. Revised: In April of 2001 the Development Credit Agreement was revised to include a new objective, "the construction of two secondary schools in the Borrower's Eastern Province ." This is clearly not an objective statement, but the description of a new activity . Its focus on secondary schools is out of step with the original project objectives and the main rationale for the project, which was to improve basic education access and outcomes for the poor. b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives /key associated outcome targets? Yes Date of Board Approval: 03/30/2001 c. Components (or Key Conditions in the case of DPLs, as appropriate): 1. Teaching/Learning/Readiness (Estimated: US$156.9 million; Actual: US$151.5 million) covering instructional materials; school health and nutrition; bursaries for girls and supports for children "at risk;" and teacher employment and deployment (including compensation). 2. School Infrastructure (Estimated: US$109.4 million); Actual: US$43.4 million) consisting of the demand-driven construction of schools (through community construction modality ). 3. Pre-service and In-service Teacher Education (Estimated: US$31.1 million); Actual: US$8.2 million) covering of a comprehensive pre- and in-service teacher education reform program; accelerated in -service teacher training; and rehabilitation of teacher training colleges . 4. Curriculum (Estimated: US$2.2 million); Actual: US$0.2 million) supporting the revision of basic school curriculum; and the design/piloting of a new instrument for curriculum and learning assessment . 5. Administration of the Education System (Estimated: US$40.8 million); Actual: US$12.1 million) covering training and advisory services towards MoE restructuring, decentralization, and improved planning, monitoring and evaluation, and management information functions . 6. Kafumbew and Lumezi Secondary Schools (Actual: US$7.9 million) consisting of the completion of these two schools in the country's Eastern Province started under a previous project but left 90% incomplete in 1988 when the Bank suspended disbursements (portfolio-wide). d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: The total estimated cost of this sub -sector investment project, BESSIP 1, jointly prepared by the Government of the Republic of Zambia (GRZ) and many development agency partners including the Bank, and structured as the first phase of an adaptable program loan (APL), was US$340.4 million, but the actual total expenditures were only US$224 million, a 34 percent shortfall. IDA commitments at approval were estimated US$ 40 million with final disbursements being about US$ 37 million (92.5 percent). The bulk of project funds were to have been contributed by the GRZ (estimated US$167; actual US$113 (68%)), and a group of 14 other agency partners (estimated US$133 million and actual (estimated) US$74 (56%). In all components but the first (teaching/learning/readiness ) spending was dramatically below estimations, particularly curriculum (9 percent of original estimates) and teacher education (26 percent). There was no direct explanation for the funding shortfalls of the GRZ and partners, but it appears to be mainly a consequence of the Government and its other agency partners redirecting their support as of 2003 to the government's new overall sector program based on its "Education Sector Strategy 2003-2007." For this reason there was no phase 2 of the APL. The phase 1 APL was extended three times, the first from June 30, 2002 to June 30, 2003 justified because of: a) an assertion that the investment credit was shorter than the norm, b ) restructuring and personnel change in the MoE, and c ) delays in completing the two secondary schools . The additional two extensions to June 30, 2005 and June 30, 2006 were both mainly to allow completion of the two secondary schools, which experienced further delays . From 2003 to 2006 only IDA was financing the Basic Education Subsector Investment Program APL. The actual cost of the two secondary school buildings was US 7.9 million, or about 21 percent of the disbursed IDA funds, compared to the US$4.3 million estimated at amendment approval, or about 11 percent of the estimated total. This is in contrast to the construction of new schools for basic education in the original design using the community-driven approach. This approach led to schools being constructed at costs below estimates, making it possible to construct 13 percent more classrooms than estimated . 3. Relevance of Objectives & Design: The objectives to increase enrollments, improve internal efficiency and improve learning outcomes were all highly relevant to the weaknesses in the country's education system, which had seen recent declines in enrollment, especially among the poorest, high dropout and retention rates, and extremely low student academic achievement . The CAS for Zambia in 1996 provided a framework for greater access to primary education services and improved quality, targeting the poor; the 2004 CAS, effective at project completion, similarly committed to 100 percent admission to grade 1, 80 percent completion of grade 7 and raised learning outcomes, including in underserved regions and among the poorest . These objectives are also strongly supported in Banks 2005 Education Sector Strategy, which encouraged continued progress on "education for all" goals, especially improved student learning outcomes. The relevance of the added secondary school construction "objective" was problematic. It did resonate with the CAS (2004) aim to increase enrollment rates at all levels of schooling, including secondary, but being focused exclusively on buildings and furnishings, it did not even have any enrollment targets; also, it "deviated from the overall strategic coherence of the IDA credit " (ICR phrase) and its focus on basic education for the poorest . In the design of the project, there was a reasonably good fit between the objectives and the components specified, except that the fifth component, reforming the administration of the education system, was very diffuse and more ambitious than needed for fulfilment of the objectives . However, there were some relevance problems with certain design features. First, the objective of increasing the national gross enrollment ratio (GER) to 100 percent had to be set aside at the outset, since the country did not have a recent enough census to even set an accurate benchmark level (the last census was in 1990). The GER could only be tracked beginning in 2003 (after the end of BESSIP1) when the 2000 census figures became available . This was knowable during project preparation and represents a design flaw. In the interim, the project was to monitor improvement in access by tracking growth in enrollments from year to year, with a targeted annual improvement of 3.9 percent (ICR says 4.0). However, population growth was already 3.3 percent (PAD, Annex 11, p. 12). The target was thus only 0.6 above population growth. From the PAD's GER baseline (1998) of 84 percent, reaching 100 percent at that rate would take about 27 years! A similar problem is found with objective 2 (internal efficiency), which was to compare the net enrollment ratio to the gross -- neither of which could be calculated until 2003. When the ratios were calculated for 2005 it became clear that this indicator was not as appropriate as others for determining internal efficiency . Second, there was problem in the design of learning outcomes indicators . In this case no targets were set (how much learning improvement would be satisfactory? ) and the final outcomes were only to be measured in 2005, three years after the close of the BESSIP 1. This brings up a general issue : the target year for reaching full enrollment and improved learning was 2005 but BESSIP1 was to close in 2002. The target date was at the end of the BESSIP program (APL phase 1 and a presumed phase 2). However, phase 2 was never implemented, removing many of the inputs needed for the ultimate attainment of the goals (even 34 percent of BESSIP1 inputs were not delivered). This might have been avoided if there had been targets for BESSIP 1 that were attainable during its implementation period. This also calls into question whether the APL was the appropriate instrument for this project and whether it was designed/used appropriately. It proved not to be "adaptable" when in 2003 all of the Bank's partners moved from it to another government program but the Bank stayed by itself with BESSIP 1 for another 3 years. Why didn't the APL adapt to the change in government strategy? Finally, the program for the BESSIP1 were overly ambitious given the absorptive capacity of the Ministry . At project entry it was acknowledged that the "institutional performance" of the MoE was weak, nevertheless during BESSIP 1 the program expanded the share of external funding in the total ministry budget from 20 to almost 40 percent. The mitigation strategies (e.g., "policy dialogue on civil service effectiveness ") were not adequate and as a result work was delayed (contributing to the need for the first extension ) and many programs were not carried out and funds not spent. 4. Achievement of Objectives (Efficacy): This covers the three original project objectives plus the forth, added by amendment . 1. Increase in enrollment. The target of this objective was to increase the gross enrollment ratio to 100 percent in grades 1-7 by 2005 (now within BESSIP1, given project extension). The reported GER in 2005 was 119 percent, exceeding the target. Exceeding 100 percent is not necessarily positive . In this case it was likely a result of overage children entering the education system in response to the official elimination of school fees in 2002, reflecting pent-up demand. Enrollment growth was unquestionably influenced by the project's school construction and rehabilitation programs, the provision of bursaries, and the growth of community schools, all of which met their goals . However, growth was greater than what was provided for by such programs . The elimination of school fees was clearly also a strong factor: in fact, the 2004 CAS strategy document showed enrollment growth only being 2 percent during the three years of BESSIP1 (1999-2001), below the population growth rate, but above 7 percent after the free education initiative was announced. Achievement of objective: substantial 2. Narrowing the gap between gross and net enrollment ratios . The estimated GER and NER in 1999 were 85 and 71 percent respectively, showing a 14 percent gap. In 2005 they were recorded as 119 and 96 percent, a 23 percent gap. Thus, the objective was, strictly speaking, not fulfilled . There was no explanation for this gap increase, but it was probably at least in part a reflection of recent increases in overage enrollments mentioned above . The ICR reported on three other measures (not in the design) that suggest some improvement in internal efficiency : namely, completion rate: increased from 66 percent in 1998 to 82 percent in 2005; dropout rate: fell from 5 to 2.8 percent on average; and repetition rate; reduced from 7 to 6.6 percent on average (the latter two indicators were found in the notes to the ICR, Annex 5 and are not confirmed elsewhere ). Achievement of objective: modest 3. Improving learning achievement. Learning achievement was measured in grade 5 (the first year of middle basic education) in 1999 and 2003. There was no target level mentioned in the text, but the LogFrame (PAD Annex 11, page 12) set a goal of 50 percent by 2005 (presumably, the average test score ). The outcomes show an increase in English scores from 33 to 34 percent, and in mathematics from 34 to 38 percent. These are modest to negligible changes and nowhere near the 50 percent target. Measuring achievement in 2005 as planned would have provided more time for achievement improvement but for reasons not explained this was not done . Holding achievement rates constant during a period of rapid expansion could be considered an accomplishment, but the big expansion would not have reached grade 5 by 2003. More relevant is the fact that availability of books was below planned levels (1 book for 3-4 children compared to the planned 1:2), school health programs were not widely implemented, curriculum revisions were not completed, and hours of instruction, already low by African standards (CAS document, 2004) were falling even further due to more double and triple shifting . In-service training was provided as planned, including that covering the use of local languages for initial literacy . The latter had the potential to greatly improve literacy levels (even in English), but it was not strongly emphasized (not mentioned in ICR text) and only occurred during the year before the 2003 assessment. Reduced disparities in achievement was to be tracked in project supervision (ISRs), but there was no report on this in the ICR . Achievement of objective: negligible 4. Construction of two secondary schools in the Eastern Province . The work on these schools was seriously delayed and costs were far above budget (and above norms for their kind of school ). When the schools were finally completed, water and electricity were not yet available . Achievement of objective: negligible 5. Efficiency (not applicable to DPLs): ICR figures show student unit costs in grades 1-7 to have risen during 1999 to 2005 from about $32 to $35, but given a improved graduation rate the cost per graduate is estimated to have fallen slightly (from $302 to $296). Cost-effectiveness during that period could not be estimated given the lack of outcomes data in 2005, but there was not much improvement as of 2003, given the marginal changes in student learning outcomes . Funding flows and utilization were facilitated by new decentralized system (using school grants), but there were problems with the funding formulas, poor school planning, and weak procurement capacity at the district and school levels . Improved donor coordination and harmonized /formalized procedures reduced the transaction costs to government . Concerning the two secondary schools, inefficiencies are found in the cost overruns (absorbing 21 compared to the estimated 11 percent of BESSIP1 budget), the long delays in construction (requiring a three year extension ), and the relatively low number of students each school accommodated (the first intake was about 100 students, indicating the numbers over the three years of secondary school could be as low as 300). ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re-estimated value at evaluation : re- Rate Available? Point Value Coverage/Scope* Appraisal % % ICR estimate % % * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The relevance of the project is rated as modest , lower than the substanti al that it would have been based on its three main three objectives because of poor relevance of the added objective and relevance problems in some design features. The efficacy is considered modest , based one substantial rating on a project objective, one modest , and two negligible . The efficiency rating is modest . Given that the modest rating for relevance was borderline, a moderately unsatisfactory rating is given for the overall outcome rating . a. Outcome Rating : Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: Strategic planning in the education sector in Zambia has improved over the past three years, with the BESSIP program transitioning into a sector -wide strategy for 2003-07 and that feeding into the education chapter of the National Development Plan (2006-2010), which ensues continued expansion of education access at all levels, and gives priority to improved education quality along the lines begun in BESSIP . Much of the institutional strengthening started in BESSIP will also be carried forward, including the decentralization program, moving school planning and management to district education boards, and funding school through school grants . Efforts under BESSIP to create coordination among development agencies under MoE's leadership has also been regularized, epitomized by an annual review process in which all major donors participate . Shortfalls under BESSIP also raise questions about the future of certain programs. Problems with book distribution were never fully solved nor was there a clear plan for decentralized book procurement . Also since few changes in school health plans and curricular development came to fruition under BESSIP it is hard to know what can be can be carried forward . Finally, although the hiring of teachers has begun again (unfrozen), it will take time and perseverance (sometimes lacking) before there are enough trained teachers to provide quality instruction in the poorest communities . a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: The Bank team is recognized for its leading role in coordinating the contributions of 15 development agencies to BESSIP1 and its marshalling solid technical support to the government during project preparation . However, there were some notable weakness in project design, including the inappropriate use of the APL format, problems with poorly specified indicators, and overestimation of GRZ absorptive capacity (see above). Also, the Bank's team decision to amend the project to include the construction of two secondary schools was unfortunate, leaving it with an APL1 sub-sector project to complete, after three extensions -- mostly for completing the secondary schools, when other development agencies and even the government had moved on to a new sector -wide program. The Bank carried out regular supervision missions, kept up a high level of dialogue with the GRZ and development agency partners, and used the ISR system to identify critical issues on time . However, the team showed weaknesses in working with the GRZ on the secondary school construction effort, failing to alert the GRZ to design and budget inadequacies and to recommend remedial measures when the program faltered . It is also hard to understand the failure of IDA join its agency partners in supporting the government's new sector -wide plan after 2003, a possibility encouraged by the APL format . This review weights quality at entry higher than supervision and thus rates overall performance moderately unsatisfactory. at -Entry :Moderately Unsatisfactory a. Ensuring Quality -at- b. Quality of Supervision :Moderately Satisfactory c. Overall Bank Performance :Moderately Unsatisfactory 9. Assessment of Borrower Performance: The Government of the Republic of Zambia showed strong ownership and commitment to sub -sector planning during the late 90s and to the creation of BESSIP. It also instituted the key policy reforms that made the achievement of project objectives possible, including decentralization of education service delivery, the use of local languages in literacy training (not emphasized in the ICR), and demand-driven school construction through its Micro-Projects Unit. Its performance with respect to teachers showed some weaknesses, since it granted increases in salaries and benefits that it could not pay for and thus had to freeze the hiring of teacher for many years, driving up the pupil-teacher ratios and undermining school quality . Its shifting away from BESSIP after three years to a sector wide plan meant that not as much counterpart funding was available to BESSIP as expected. Ministry of Education managers took leadership in BESSIP and coordinating and cooperating fully with its many agency partners. However, there was considerable turnover in MoE staffing, project areas which fell behind or were not implemented, and failure to produce progress reports in a timely manner . Also, the MoE showed poor judgment and planning concerning the construction of the two secondary schools that were added to the project . This review weights government perform higher than implementing agency performance and thus rates the overall borrower performance as moderately satisfactory . a. Government Performance :Moderately Satisfactory b. Implementing Agency Performance :Moderately Unsatisfactory c. Overall Borrower Performance :Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: M&E design weaknesses were already described in the relevance section. Concerning implementation, there was the problem of learning assessment not being undertaken in 2005 as planned. Also, problematic was the project's failure to take into consideration population growth when reporting on enrollment changes . One of the subgoals of the project was to establish a stronger quantitative data base for decision making, and progress was made on this; however, the ICR notes that external technical assistance is still needed for data analysis and that school level planning needs improvement. a. M&E Quality Rating : Modest 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately The ICR did not assess the Satisfactory Unsatisfactory achievement of the second objective (which this IEG review assesses as modest). Also, the ICR portrayed the modest to negligible improvements in test scores as positive, but did not present convincing evidence to justify this. Risk to Development Moderate Significant Several programmatic and institutional Outcome : changes in BESSIP did not come to fruition, creating a heightened risk of uncertain development futures . Bank Performance : Moderately Moderately Some design flaws in the project could Satisfactory Unsatisfactory have been avoided during preparation and the inclusion of secondary school construction objective, plus the three year extension to cover it, showed poor judgment. Borrower Performance : Moderately Moderately Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: 1. Government campaigns to increase assess to basic education can and should be accompanied by a commitment to improve learning outcomes . This project did have both as objectives, and pursued them at the same time, for example, building schools while providing in -service training to teachers . The increase in expenditure per student even as enrollments were growing was also evidence of that commitment . The fact that progress in improving access was greater than improving learning outcomes demonstrates the relative difficulty in delivering the latter and the need for perseverance in creating the conditions for better learning . Governments eliminating school fees to stimulate universal access (the case of Zambia) need make sure this is accompanied by enough building, teacher, and learning materials resources to maintain the minimum conditions for learning . 2. Amending and extending a project for a narrowly focused objective (in this case, the construction of two secondary schools) risks reducing the project's strategic relevance . Following its preparation of a new sector -wide strategic plan for education (2003-07) the GRZ did not continue its cooperation with the Bank on the subsector investment program. Even so, the Bank continued with BESSIP on its own, extending it three times to 2006, mainly to support the completion of the two secondary schools . This not only put the Bank's operations out of sync with the government and other agencies, but also prevented IDA's potentially more significant involvement as a partner in the new strategic plan . (Adapted from the ICR.) 3. The APL is not an appropriate IDA instrument under certain circumstances . A combination of conditions made the use of APL inappropriate for this project, for example, there was a lack of long -term government commitment to the program upon which the APL was built, a situation where expenditure items were not completely identified, a set of objectives which needed a relatively long period to come into fruition, and partnership arrangements with other development agencies, many of which chose to provide their aid through budget support and the pooling of funds instead of conventional investment support . Development Policy Lending may have been more appropriate under these circumstance (assuming adequate assurances of good fiduciary management ). 4. Community-driven school construction and rehabilitation can be advantageous when there is adequate management and technical support . The project shows a dramatic contrast between the rapid and effective construction of hundreds of primary schools using the community driven and managed approach and the two secondary schools constructed with large cost overruns and significant delays by contractors . Of course, primary and secondary schools have different engineering demands, but this experience demonstrates that primary schools should continue to be built using the community -driven approach and even secondary school construction may do well to add some community-driven elements. (Adapted from ICR.) 5. Decentralization of educational service delivery and funding flows (e.g., though grants to districts and schools ) can have a positive effect on school system management and funding utilization . This is revealed by the management restructuring undertaken during BESSIP, which included the establishment of District Education Boards which were then funded through grants . However, a long-term process of capacity building at the district and school levels is needed in order to assure effective local planning and financial management capacity (including procurement). (Adapted from the ICR.) 6. Significant learning outcomes improvement requires more than the delivery of some improvements and a marginal increase in expenditure per student ; it requires a full range of integrated inputs and processes as minimum conditions . BESSIP delivered many good services and inputs, such as teacher in -service training and student books, but did not fulfill its promises of providing one book per two students and curriculum revision, and underwent changes (double and triple shifting) that reduced its already substandard number of hours of instruction . Literacy instruction in the local language was an innovation in BESSIP of much promise, but received very little attention (at least in the project description and ICR ). 14. Assessment Recommended? Yes No Why? To verify ratings 15. Comments on Quality of ICR: The ICR is well founded on evidence and lays out critical issues . This said, it is not as critical as it could have been of the design flaws of the project, and the Bank team's decision to keep the APL going for six years (mostly to cover the construction of two schools ) when the government and the other agency partners had moved onto another, more comprehensive program. Also, given the evidence provided it is difficult to see the justification for a positive rating on learning outcomes. a.Quality of ICR Rating : Satisfactory