ICRR 14263 Report Number : ICRR14263 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 12/16/2013 Country : Yemen Project ID : P076185 Appraisal Actual Project Name : Yemen: Basic US$M ): Project Costs (US$M): 121.1 121.2 Education Development Program L/C Number : C3988 Loan /Credit (US$M): Loan/ US$M ): 65.0 60.1 Sector Board : Education US$M): Cofinancing (US$M ): 48.0 48.1 Cofinanciers : DFID, Netherlands, Board Approval Date : 09/23/2004 Kfw Closing Date : 06/30/2010 12/31/2012 Sector (s): Primary education (70%); Central government administration (20%); Sub-national government administration (10%) Theme (s): Education for all (40% - P); Rural services and infrastructure (20% - S); Gender (20% - S); Participation and civic engagement (20% - S) Prepared by : Reviewed by : ICR Review Group : Coordinator : Judith Hahn Gaubatz Judyth L. Twigg Christopher D. Gerrard IEGPS2 2. Project Objectives and Components: a. Objectives: Original objectives: According to the Project Appraisal Document (PAD, page 3) and the Development Credit Agreement (DCA, page 16), the objective of the project was : To assist the Borrower in expanding the provision of quality basic education for all with particular attention to gender equity . Revised objectives: According to the Project Restructuring Paper (page 7) and the Amended Development Credit Agreement (page 19), the objectives of the project were : To increase enrollment of children in basic education, with particular attention to gender equity, in 10 governorates; To improve quality of teaching in basic education nationally; and To establish foundations for improved Ministry of Education (MOE)MOE ) capacity to carry out its core functions . b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives /key associated outcome targets? Yes Date of Board Approval: 12/16/2009 c. Components: 1. Expansion of Access (Appraisal: US$55.0 million; Actual: US$62.7 million): This component aimed to increase the enrollment and the retention of students, particularly girls . It was to focus on districts with under-served populations. Activities were to include: review and updating of basic school designs; construction and/or rehabilitation of schools and classrooms; recruitment of female teachers in rural areas; training on gender equity; and piloting of a conditional cash transfer program to promote school attendance for girls . The activities were to be implemented initially in ten governorates, and then scaled up nationwide to all twenty governorates . 2. Quality Improvement (Appraisal: US$35.7 million; Actual: US$43.8 million): This component aimed to improve the quality of education in teaching and learning . Activities were to include: a curriculum review to assess relevance and improve linkages with learning achievement monitoring; provision of in -service teacher training; training of school managers and inspectors; and strengthening of the school supervision and inspection function . 3. Institutional Change and Capacity Building (Appraisal: US$14.7 million; Actual: US$14.7 million): This component aimed to strengthen the capacity of the Ministry of Education . Activities were to include: establishment of task forces and action plans on topics such as curriculum, teacher development, quality assurance, and community participation; establishment of a new General Directorate for Community Participation; and a pilot program to provide small grants to schools for community -led school improvement plans. Under the project restructuring, the following modifications were made to the components : 1. Expansion of Access: The project activities were limited to the initial ten governorates, with the planned national scale-up dropped. 2. Quality Improvement: The project activities were limited to teacher and management training, and inspection, with the planned curriculum review dropped . 3. Institutional Change and Capacity Building : Evaluative activities for three pilot programs - conditional cash transfers, a rural female teacher program, and school improvement plans - were added. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project cost The appraised cost of the three components was US$ 105 million. An additional US$15.7 million was added for physical and price contingencies, for a total appraised cost of $ 121 million. The final project cost was US$121.2 million, out of an appraised US$121.1 million. During the initial project period, the cost of construction almost doubled due to significant increases in international prices for steel and cement, coupled with the elimination of fuel subsidies in the country . This led to an increase in costs for Component 1 (Expansion of Access) even though the geographic scope of the project was reduced to ten governorates . The ICR does not provide an explanation for the increase in costs for Component 2 (Quality Improvement). Financing The project was financed by a IDA Credit and a Multi -Donor Trust Fund (MDTF) Grant. The funds were pooled together to support project implementation, although the MDTF funds were disbursed first, according to agreements with the donors. The final amount of the IDA Credit was US$ 60.1 million, out of an appraised US$65.0 million. US$7.7 million was cancelled from the Credit; the ICR does not explain the reason for this cancellation . The final difference was due to exchange rate fluctuations . The final amount of the MDTF Grant was US$ 48.1 million, out of an appraised US$48.0 million. At project appraisal, the UK Department for International Development (DfID) agreed to provide US$25.0 million, and the Government of the Netherlands agreed to provide US$ 23.0 million. However, near the end of the project period, both donors reduced their contributions . DfID did not transfer the remaining GBP 1.14 million out of its initial GBP 15 million; Netherlands did not transfer US$ 2.8 million of its original US$24.0 million. The donors cited the slow rate of disbursement for the MDTF funds, shifts in donor priorities, and the likelihood that many of the core project goals could be achieved with the remaining project funds . Additional parallel financing from the KfW had been approved in 2008, and favorable exchange rate fluctuations resulted in no actual changes to the amount of that financing available to the project . At the time of project restructuring in December 2012, the IDA Credit had disbursed US$48.2 million. Borrower contribution The Borrower provided US$10.2 million in counterpart funds, out of an appraised amount of US$ 9.0 million. Dates June 2007 : The Credit agreement was amended to enable the project to participate in a Bank -wide e-disbursement pilot program. September 2007: The Credit agreement was amended to enable the conditional cash transfers, contracting of female teachers in rural schools, and establishment of 90-day advance accounts . November 2008 : The Credit agreement was amended to allow financing of Whole School Improvement grants. December 2009: The project was restructured to reflect the reduced geographic scope of Component 1, the significantly revised results framework, and an extension of the project closing date from June 2010 to June 2012. The extension was approved to allow completion of activities, which had experienced significant delays at the start of the project period . June 2012: The project was restructured to reflect a partial cancellation of the Credit (US$7.7 million) and an extension of the closing date from June 2012 to December 2012. The project extension was approved to allow completion of activities, which had experienced delays due to the period of civil unrest in 2011, during which disbursements were suspended for most of the Bank's portfolio in the country . 3. Relevance of Objectives & Design: a. Relevance of Objectives: Original : High Although the country has made notable progress in improving access to education in recent decades, enrollment in primary education remains low at approximately 65%. Inequity is also an issue, with only 55% of primary school-aged girls enrolled nationwide and fewer than 30% enrolled in rural areas. The project objectives, both original and revised, were highly relevant to the National Basic Education Development Strategy for 2003-2015 and to the Bank's Interim Strategy Note for FY 2013-14, which identifies improved access to basic social services, with an increased focus on gender inclusion, as a key pillar . The objectives were also highly relevant to the Bank's corporate priorities as reflected in the Millennium Development Goals . Revised : High The project objectives were refined to focus only on quality of teaching and management and to include capacity building for the Ministry of Education, while also limiting the geographic scope of the project to ten governorates . The objectives remained highly relevant . b. Relevance of Design: Original : Substantial The project's design addressed multiple aspects of expanding quality education that were likely to lead to the intended outcomes. Supply-side elements included providing new facilities or rehabilitating existing ones to expand availability of classroom spaces . Gender aspects included designing classrooms that meet the needs of girls and recruiting more female teachers . Quality of learning was to be improved through both improved teacher quality through training and improved learning materials . The project geographically targeted governorates with higher incidences of poverty, which was a significant factor affecting enrollment of girls . Revised : Substantial Given capacity constraints in implementing the numerous project activities, the revised project design was more limited in scope. The geographic scope of the project was limited to ten governorates, and the quality improvements were limited to teacher and management elements . 4. Achievement of Objectives (Efficacy): Original objective To assist the Borrower in expanding the provision of quality basic education for all, with particular attention to gender equity . Achievement of this objective is rated Modest . Although there was increased access to basic education particularly for girls, there was insufficient evidence on improved quality. Outputs Construction of 201 schools in project areas, falling slightly short of the target of 213. Construction of 1,939 classrooms in project areas, surpassing the target of 1,892. Rehabilitation of 1,768 classrooms in project areas, surpassing the target of 1,446. Provision of conditional cash transfers to 38,039 girls in grades 4-9 in two targeted governorates, surpassing the target of 33,784. Provision of training to 47,674 teachers nationwide, falling short of the target of 90,000. Provision of training to 16,186 headmasters and deputy headmasters nationwide, surpassing the target of 15,000. Provision of training to 6,666 school inspectors nationwide, surpassing the target of 4,500. Recruitment of 513 female teachers benefiting 234 schools in project areas. Provision of training to 2,867 Ministry of Education (MOE) staff, surpassing the target of 1,200. Provision of $1500 annual grants to 60 schools for the Whole School Improvement community participation program. Outcomes Access The number of girls enrolled in grades 1-9 in the ten project governorates increased from 783,454 in 2009 to 932,761 in 2011/12, achieving the target of 924,788. The number of boys enrolled in grades 1-9 in the ten project governorates was maintained at 1.1-1.2 million during the project period. Gender Equity The gender parity index for enrollment improved from 0.70 at project start to 0.78 in 2012, falling slightly short of the target of 0.80. Quality The percentage of students attending schools where the majority of teachers received training in science or math increased from 5% in 2009 to 21% in 2012, surpassing the target of 15%. The transition rate in grades 1-6 decreased from 0.59 in 2009 to 0.51 in 2012 for girls, and from 0.69 to 0.61 for boys. The ICR (page 18) refers to findings from a qualitative Evaluation Report conducted for the conditional cash transfer program in Hodeidah governorate, including reduction in dropouts and increased enrollment . However, no specific data are provided . The ICR (page 18) refers to findings from the Borrower's ICR, classroom observations, beneficiary interviews, and "various project evaluation reports, " citing improved quality of classroom instruction as a result of the project trainings . However, no specific data are provided . Revised objectives : To increase enrollment of children in basic education, with particular attention to gender equity, in 10 governorates . Achievement of this objective is rated Substantial due to increased enrollment, particularly of girls . Outputs See outputs reported above. Outcomes The number of girls enrolled in grades 1-9 in the ten project governorates increased from 783,454 in 2009 to 932,761 in 2011/12, achieving the target of 924,788. The number of boys enrolled in grades 1-9 in the ten project governorates was maintained at 1.1-1.2 million during the project period. The gender parity index for enrollment improved from 0.70 at project start to 0.78 in 2012, falling slightly short of the target of 0.80. To improve quality of teaching in basic education . Achievement of this objective is rated Modest due to insufficient evidence of improved quality of teaching . Outputs See outputs reported above Outcomes The transition rate in grades 1-6 decreased from 0.59 in 2009 go 0.51 in 2012 for girls, and from 0.69 to 0.61 for boys. The ICR (page 18) refers to findings from the Borrower's ICR, classroom observations, beneficiary interviews, and "various project evaluation reports " citing improved quality of classroom instruction as a result of the trainings. However, no specific data are provided . MOE ) capacity to carry out its core functions . To establish foundations for improved Ministry of Education (MOE) Achievement of this objective is rated Substantial due to effective implementation of activities related to M&E, school supervision, and sector management . Outputs Provision of training to 2,867 Ministry of Education (MOE) staff, surpassing the target of 1,200. Establishment of a Girls Education Section and the Center of Management and Evaluation in the Ministry of Education. Conducting of evaluative studies for capacity building, in -service teacher training, the Whole School Improvement program, and female teacher recruitment . The assessment of the conditional cash transfer program was only partially completed due to the civil unrest . Development of an education management information system, which was completed but is not yet functional. Outcomes The Ministry of Education was able to carry out the following activities to monitor the education sector : Collection of data for international verification for the Trends in International Mathematics and Science Study (TIMSS) 2011. Conducting of at least three supervision visits annually at 8,779 schools. Conducting of a Comprehensive School Survey for 2011/12. However, the planned quality control activities for the survey were not carried out . The ICR (page 14) also reports findings from the Capacity Building assessment, including increased motivation of education staff due to project -funded training, improved frequency and quality of inspections, and increased community awareness. However, no specific data are cited . 5. Efficiency: Modest . There were significant delays in civil works activities (which directly impacted ability to enroll students and comprised about one-third of project costs), as prices for cement and steel increased significantly, almost doubling the price of school construction . The scope of the project was also significantly reduced, although the overall project cost remained the same . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re -estimated value at evaluation : re- Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Original Objectives : Moderately Unsatisfactory Relevance of the original project objective is rated High and relevance of the project design is rated Substantial . Achievement of the original objective to assist the Borrower in expanding the provision of quality basic education for all, with particular attention to gender equity, is rated Modest due to lack of evidence of the provision of quality education. Efficiency is rated Modest. Revised Objectives : Moderately Satisfactory Relevance of the revised project objectives is rated High and relevance of the project design is rated Substantial. Achievement of the revised objective to increase enrollment of children in basic education, with particular attention to gender equity, in 10 governorates, is rated Substantial due to evidence of increased enrollment of girls. Achievement of the objective to improve quality of teaching in basic education is rated Modest due to insufficient evidence . Achievement of the objective to establish foundations for improved Ministry of Education capacity is rated Substantial due to effective implementation of activities related to M&E, school supervision, and sector management . Efficiency is rated Modest. According to OPCS/IEG harmonized guidelines, the final outcome rating of a restructured project is determined by weighting the amount of the Credit that disbursed before ("original objectives") and after ("revised objectives") the project restructuring. The Credit had disbursed US$48.2 million out of US$68.66 million, or 70.2%, at the time of restructuring. Therefore, the combined outcome rating is Moderately Unsatisfactory . a. Outcome Rating : Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: A follow-up Bank operation (Basic Education Development Project II, US$ 66.0 million; approval February 2013) was approved to continue and expand key interventions from this project, including rural teacher recruitment, conditional cash transfers, and whole school improvement . Civil works outputs are likely to be used as intended for classroom instruction; however, the lack of a clear operations maintenance plan may affect longer -term use. a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: a. Quality at entry: The project built on a preceding Bank operation (Basic Education Expansion Project, US$ 41.4 million, FY2001-07) that had also aimed to increase enrollment in underserved areas, although it was implemented in only four governorates. Institutional arrangements, including the project management unit, were similar to the preceding project. The project was prepared in a supportive policy environment, as there was strong government and donor collaboration to ensure consistency with the government's broader sector reform program. Risks were accurately identified, including the risks of delays in broader macroeconomic and civil service reforms, and lack of adequate technical capacity or ability to scale up nationwide . However, mitigation measures were not effectively planned . In addition, there were significant shortcomings in the M&E framework. The indicators in the initial results framework represented outcomes from the national program, not from the project directly. Baseline data and targets were identified on the basis of invalid data . at -Entry Rating : Quality -at- Moderately Unsatisfactory b. Quality of supervision: Major shortcomings in the project's M&E framework were addressed in a formal project restructuring in December 2009, following recommendations from a Quality Assessment of Lending Portfolio review . Civil unrest in 2011 led to the evacuation of Bank staff and suspension of the entire Bank portfolio until January 2012, although project administration costs were exempted from the suspension . Fiduciary performance was overall satisfactory, with proactive actions taken to address initial procurement problems . There is no information in the ICR on safeguards compliance, although the project team subsequently reported that safeguard compliance matters were addressed through government Bank actions . Quality of Supervision Rating : Moderately Satisfactory Overall Bank Performance Rating : Moderately Unsatisfactory 9. Assessment of Borrower Performance: a. Government Performance: The government was overall effective in providing a supportive policy and institutional environment for the project. However, the period of civil unrest in 2011 had a major impact on the overall political and security situation. This led to a suspension of project disbursements, as well as the halting of numerous sector activities. However, the ICR (page 12) notes that as the bulk of project activities had already been implemented prior to the period of unrest, the overall negative impact was mitigated . Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance: Despite drawing upon project management arrangements from the prior Bank operation, the project still experienced implementation delays . There was frequent turnover in staff (in part due to remuneration and benefits issues) and remaining capacity weaknesses . There were also significant delays in civil works activities due to significantly increased costs of materials (which led to re-advertising and re-tendering of contracts in some cases) and weak procurement capacity at the governorate level . Financial management performance was satisfactory, although some procurement issues were identified during the project period . There is no information in the ICR on safeguards compliance, although the project team subsequently reported that safeguard compliance matters were addressed through government Bank actions . Implementing Agency Performance Rating : Moderately Satisfactory Overall Borrower Performance Rating : Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: There were major shortcomings in the results framework . Key indicators reflected outcomes for the entire sector program (i.e. gross enrollment rates, gross intake rates ), rather than being directly linked to the project activities . Baseline data (and related targets) were based primarily on Annual Education Surveys, which were subsequently deemed unreliable due to the voluntary nature of survey responses . During project restructuring, there were major revisions to the key indicators in order to more accurately capture the project's impact (i.e. enrollment increases, gender parity index ). Due to the less-than-reliable nature of existing surveys, the M&E arrangements included drawing from other sources such as international test data, project progress reports, and independent assessments . The revised M&E design also introduced evaluative activities for the pilot programs : conditional cash transfers, female teacher recruitment, and the Whole School Improvement program . b. M&E Implementation: Regular M&E reports were provided by the project management unit . The ICR (page 13) reports that the "quality of the progress reports was considered 'high' by the team despite the challenges ... in obtaining accurate and timely information." The Education Management Information System was installed but is not yet fully functional . Evaluative activities - Capacity Building Assessment, Whole School Improvement Assessment, Rural Female Teacher Study, and In-service Teacher Training Study - were completed. The impact evaluation for the conditional cash transfer program was only partially completed due to the civil unrest . c. M&E Utilization: M&E data were used to help redefine the scope of the project during project restructuring . M&E Quality Rating : Modest 11. Other Issues a. Safeguards: The project was classified as Environmental Category "B" due to school construction and rehabilitation . The safeguard policy on environment assessment (OP/BP 4.01) was triggered, and therefore an environmental management plan was prepared. However, there is no information provided in the ICR on compliance with safeguard policies. b. Fiduciary Compliance: Financial management: Financial reporting was timely and regular . All audit reports were unqualified, albeit delayed at times. Procurement: Procurement performance was moderately satisfactory, although high levels of staff turnover "affected the quality of procurement and contributed to delays " (ICR, page 16). The ICR (page 16) also notes some observations during the ICR preparation process that "highlighted issues with the quality of some procurement lots, particularly school furniture . In many cases, these lots were approved ex -post by the [Bank] and the responsibility for the technical specifications lay with the [Borrower]." Actions were taken to reinforce the procurement unit with additional staff, assign a staff person full -time to control the quality of procurement documents, and recruit an international procurement specialist to provide on -the-job training. c. Unintended Impacts (positive or negative): None reported. d. Other: 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately The project under the original Satisfactory Unsatisfactory objectives is rated Moderately Unsatisfactory due to shortcomings in achievement of the objective to expand quality basic education for all with particular attention to gender equity, as well as modest efficiency. The project under the revised objectives is rated Moderately Satisfactory. As the Credit had disbursed US$48.2 million out of US$68.66 million, or 70.2%, the combined weighted outcome rating is Moderately Unsatisfactory. Risk to Development Moderate Moderate Outcome : Bank Performance : Moderately Moderately There were shortcomings in identifying Satisfactory Unsatisfactory risk mitigation measures, and in the initial M&E design. Borrower Performance : Satisfactory Moderately There were significant implementation Satisfactory delays, particularly for civil works . Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: Lessons drawn from the ICR, adapted by IEG : Ensuring the reliability of data at the project start is critical . In the case of this project, a major project restructuring had to be undertaken in part to correct significant shortcomings in the M&E design . In a context of weak capacity, innovative approaches can still be introduced . In the case of this project, additional technical assistance and time were needed to support the innovations, but they were effectively implemented. 14. Assessment Recommended? Yes No Why? To verify outcomes on increased access to basic education and to learn lessons from the related interventions such as the conditional cash transfers and rural female teacher recruitment . 15. Comments on Quality of ICR: The ICR quality is satisfactory, but marginally so . Evidence contained throughout the ICR is adequate to assess the project outcomes, although the ICR would have benefited from a clearer analysis and presentation of the project's results chain. There is no information reported in the ICR on safeguards compliance . a.Quality of ICR Rating : Satisfactory