ICRR 14116 Report Number : ICRR14116 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 08/26/2013 Country : Chad Project ID : P000527 Appraisal Actual Project Name : Education Sector US$M ): Project Costs (US$M): 119.664 NA Reform Project L/C Number : C3739; CH023 Loan/ US$M ): Loan /Credit (US$M): 42.34 42.34 Sector Board : Education US$M): Cofinancing (US$M ): 2.9 2.9 Cofinanciers : GTZ Board Approval Date : 04/06/2003 Closing Date : 07/02/2007 06/30/2012 Sector (s): Primary education (50%); Central government administration (25%); Other social services (15%); Secondary education (5%); Vocational training (5%) Theme (s): Rural services and infrastructure (29% - P); Education for all (29% - P); HIV/AIDS (14% - S); Gender (14% - S); Decentralization (14% - S) Prepared by : Reviewed by : ICR Review Group : Coordinator : Eluned Schweitzer Alain A. Barbu Christopher D. Gerrard IEGPS2 2. Project Objectives and Components: a. Objectives: According to the Development Credit Agreement (DCA Schedule 2, pp. 18 -20 ), the project objective was "to assist the Borrower in developing its policy framework, strategies and means of achievement of quality universal basic education and implementing its strategy to improve access and equity of education ." There is a discrepancy between this objective and the objective as stated in the Project Appraisal Document (PAD) which reads ( page 2) " Its objective is to ensure that the framework for quality universal basic education is in place and ready for implementation. The project places an emphasis on developing an effective partnership between the State and the communities, creating a national technical capacity for curricula and textbook development and improving basic education quality and efficiency . At the end of the project these objectives will translate into a stronger institutional and managerial capacity, increased enrollment rates, more equitable access and better education quality and efficiency ." The discrepancy in wording does not substantially alter the aims of the objective . The Project Appraisal Document objectives text is longer and also contains descriptions of activities, This review uses the Project Objective from the PAD and the outcomes included within it ( "stronger institutional and managerial capacity, increased enrollment rates, more equitable access and better education quality and efficiency "). The project underwent a level 1 restructuring in December 2010 to address inconsistencies across documents and associated measurement issues, poor project implementation performance due to a mismatch between the capacity of the country and the complexity of the project, and the evolving context of the education system . The re-structuring also addressed changes that had arisen in the school construction program due to the use of new government oil revenue for school construction . The restructured Project Development Objectives were : "to contribute to (i) improving access and equity for primary education, and (ii) increase capacity in strategic management and curriculum development (p. 10, Amended Financing Agreement, December 29, 2010 ). b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives /key associated outcome targets? Yes Date of Board Approval: 03/05/2011 c. Components: The project originally contained four components (PAD, p. 37) 50 .42 million, IDA financing at appraisal US$ 16. 1. Improving Access and Equity (total cost at appraisal US$ 50. 16 .59 million at appraisal, actual cost : N.A.) The Objective of this component was to increase the access and equity of education . Sub-components included: (a) support to community initiatives ( US$18.11 million, IDA Grant US$3.40 million); (b) Promotion of Girls Schooling (US$1.76 million, IDA Grant US$1.62 million); (c) The construction of classrooms, wells and latrines and the acquisition of furniture and equipment (US$30.55 million, IDA Grant US$11.57 million). 47 .17 million, IDA financing at appraisal US$ 14. 2. Improving Quality and Efficiency (total cost at appraisal US$ 47. 14 .36 million, actual cost : N.A.) The Objective of this component was to improve the quality and internal efficiency of primary and general secondary education by improving curricula for training and teaching in basic education and the availability and quality of pedagogical and didactic materials . It also aimed to improve learning assessments, Early Childhood Development (ECD) programming, language and reading programs, distance education and school health and nutrition activities. 3. Literacy and Non -Formal Education (total cost at appraisal US$ 7.99 million, IDA financing at appraisal US$3 US$ 3.73 million, actual cost N .A.). This component aimed to develop a cost -effective, equity enhancing model for delivering literacy programs to out of school youth and adults in rural areas, with an emphasis on improving the quality and access to literacy for women and girls. 4. Strengthening Planning, Management and Program Monitoring 13 .4 million, IDA financing at (appraisal US$ 13. appraisal US$ 7.13 million, actual cost N .A.). The objective of this component was to provide institutional capacity development through strengthening specific units within the Ministry of Education, streamlining departments, improving national understanding of monitoring and evaluation and the development of a permanent curriculum agency . The project was re-structured in 2010 to have only two components . Components previously not funded were dropped as were those activities where the targets had been reached . The Amended Project Components were : (pp. 10-11 Amended Legal Agreement) Part A . Improving Access and Equity to education SDR 5.07 million (at restructuring Credit SDR 2 million, Grant 3.07 million, actual costs N /A) This part of the restructured project continued an emphasis on improving school facilities . It including equipping schools with sanitation facilities and water, and the provision of furniture to 132 classrooms in public primary education and to 3 new eligible Senior Public Schools. It also included support for teacher training for community teachers in primary schools and support for tutoring and incentives for girls enrollment . In addition it supported secondary education through the creation of computer centers, science laboratories and libraries in three new schools. It also continued support for school heath and nutrition services through health, hygiene and nutrition education, introducing education for HIV /AIDS and providing de-worming and micronutrient programs. Part B . Enhancing education capacity SDR 2.60 million (at restructuring Credit SDR 2.40 million, Grant SDR 0.20 million, actual costs N /A) This component continued support to local school communities, including training for parent teacher associations and the provision of grants to public primary schools to finance school action plans . It also continued technical support to the National Curriculum Center, the development of the Education Management Information System and project management. Technical support was also provided to the Ministry of Education for the preparation of the Interim Education Sector Plan (IESP). d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: The project was active from 2003 to 2012 with two periods of suspension over 2008-2009 due to civil conflict and procurement issues. It was originally part grant and part credit . Much of the anticipated co-financing built into the original project did not materialize, as appropriate donor coordination mechanisms were not in place (p. 7, ICR Review). An accurate figure for overall project costs is not available for this reason (no figures available for the African Development Bank, European Commission, French Agency for Development,the Islamic Development Bank and the UN Children's Fund). The Borrower contribution rose from US$ 20.88 million dollars at appraisal to US$75.6 million by the close of the project . Local community contributions also rose to US$ 4.15 million at closure. Counterpart funding was sometimes not received in a timely fashion, although it is not clear whether this had a substantial impact on project activities. The project was extended four times : - from June 30, 2007 to October 31 2007 because of administrative delays . - from October 2007 to June 30, 2009 because of low disbursements and design flaws . - an extension to December 31 2010 to allow completion of three main activities (i) the construction of the Curriculum Center; (ii) setting up an Education Management System and (iii) completion of the construction of wells . At this extension funds were reallocated between disbursement categories to support the construction of wells and reduce the categories of Goods and Community Grants . - an extension granted at the December 2010 restructuring to a final closing date of June 30, 2012. This was to allow completion of the restructured project activities . Re-allocation at this restructuring was from both the grant and credit sections of IDA financing. Major shifts in the use of financing were : (i) to allocate more funding to construction, goods, literacy grants, operating costs, and (ii) to reduce funding to the girls education and school grants and training categories. In addition this re-allocation increased the percentage of expenditures to be financed from 100% of foreign and 82% of local expenditures to 100% of all expenditures. 3. Relevance of Objectives & Design: a. Relevance of Objectives: Relevance of the original objectives is rated Substantial . The project was designed as the first part of a 15 year program of investment. Objectives were chosen to complement funding from other donors, including the African Development Bank (ADB), for other sections of the education system. As the beginning of a system change effort it intended to impact policy areas, in particular through the creation of the National Curriculum Center, the professionalization of undertrained community teachers and the incorporation of community schools into the state school system and fostering community engagement in school construction both to reduce costs and to foster community engagement in the education system . In addition it included a number of activities which were considered pilots for a future SWAP (p. 7, ICR Review). The objectives were related directly to the Chad Government Education strategy presented to the donors in January 2000. The Poverty Reduction Strategy paper of 2005 identified major issues with the education system, including inequities in access, irrelevant curricula, poor infrastructure and lack of qualified teachers, all of which were addressed by the project . These issues still remain relevant . In addition, monitoring and evaluation capacity, student assessment mechanisms and teacher training remain areas of concern for the World Bank and the Government . Focusing on primary education in order to provide a good foundation for literacy and the upper levels of education remains a priority. Relevance of the re -structured objectives is rated Substantial . The project was re-structured very near to its closing date and restructured project objectives were a sub -set of the original project objectives, but did not substantively change the direction of the project . b. Relevance of Design: Relevance of design to the original objectives is rated Modest , The original project design was related to the overall project objectives, but tackled too many disparate aspects of the overall education program to make a solid impact . , Government institutions in Chad were not able to address all the project components, given the political upheavals and the loss of anticipated co -financing. Although the project is a Sector Investment Loan, it was effectively seen as the first part of an education sector reform program and the original Project Development Objective (PDO) reflects this broad goal. Without the overarching support of a more policy based instrument, there was insufficient linkage between the macro goals of the project and the individual activities and indicators for each component . Relevance of design to the revised objectives is rated Modes t. Post re-structuring the objectives were simplified and indicators tailored more concretely to program design (see section on M&E). However the introduction of some small new components on secondary education seem unrelated to the overall focus on primary access and quality which remains in the revised PDOs. 4. Achievement of Objectives (Efficacy): The project had a broad overarching Project Development Objective (PDO) to assist the government in improving the education system through improving quality, making it more efficient and more equitable and improving overall access in remote areas (particularly access for women and girls ). It also intended to improve the management of the education system through administrative changes and reform . Although the PDO and indicators were revised in 2010 to be more project specific, a major indicator " Interim Education Sector Project adopted " remained, indicating that the system reform aspects of the project were still of importance . Project Implementation was affected by multiple external disruptions : (i) a teachers strike in 2007 for five months, (ii) a rebel attack in 2004-5 which delayed community based activities, (iii) two further rebel attacks in 2006 and 2008. The fact that planned activities were never funded meant that it would have been hard to achieve satisfactory results against project outcome indicators from the beginning of the project . It is difficult to assess how project activities contributed to system changes, given the complexity of the situation in country and the many variables involved . It is to be noted that the project was working directly with local beneficiaries mostly in highly disadvantaged and remote areas . Possibly because of the difficult implementation environment, most of the project efforts went into accomplishing outputs rather than being able to focus on the outcomes that these should have facilitated. 1. Assist the government in improving quality is rated Modest (Original and Revised objectives ) NOTE: This review includes construction activities under this objective as the quality of the learning environment is affected by the facilities available . However construction activities such as latrine building also affected equity issues relating to girls. Outputs : -classroom, wells and latrine construction, acquisition of furniture and equipment : 400 classrooms out of a planned 1000 and 400 latrine blocks out of a planned , 1065 and 225 wells were constructed. The remainder of the classroom construction was dropped due to the availability of other financing from oil revenues . Equipment for three secondary schools and 400 classrooms was purchased . - improving curricula for teaching in basic education - A National Languages Program had been developed activity with initial mother tongue instruction . Activities under this component were successfully implemented by GTZ in 130 schools and pedagogical materials were distributed to students . A teacher training curriculum had been developed and 10,000 community teachers had been trained to Level 1 certification. In addition a curriculum for grades 1-6 had been developed (more than had been anticipated ) and piloted in ten schools. - support for development and management of textbooks - by restructuring a total of 3.7 million textbooks and materials had been purchased, adapted and distribution to schools in 2004/5. The original targets had been 2.93 million textbooks, slightly more than 300,000 textbooks for junior high school, reference books and teachers guides . - School Health and Nutrition Program - at mid-term (i) a school health policy had been developed and disseminated country-wide; (ii) 180 school latrine blocs and 180 wells were built and communities trained to maintain infrastructure; (iii) teacher manuals and tools were developed and distributed; (iv) deworming and micronutrients were distributed to schools twice a year for 150,000 students. The Bank and the government continued distribution of de -worming and mictronutrients to an additional 90,000 students between re-structuring and closure. At project completion the materials had been received and teachers trained . - pilot support to secondary education : This was only partially achieved by project closure . Studies were completed but laboratory equipment and books were not acquired . This activity was included to fit with new priorities of the Government. Outcomes : The survival rate to 6th grade (PDO Indicator 6) actually dropped from 44% to 36% over the life of the project and at project closure only two of 8 targeted departments reached target gross enrollment rates (PDO indicator 7). There is no measure in the project of the effect that teacher training or improvements in availability of learning materials on student learning outcomes. No information is available in the ICR as to the effects of the school health and nutrition program and whether training activities meant that deworming efforts had more than a short term outcome . The development of curricula is a substantive achievement, although not fully implemented by project closure . The provision of learning materials was an appropriate input but it is not clear how these fitted with the revised curricula . 2. Assist the government in improving equity is rated Modest (Original and Revised Objectives ) Outputs : A gender sensitization campaign was carried out by GTZ . However planned studies on how to increase the number of female teachers in rural areas were not carried out . A pilot tutoring program for girls was developed and implemented in 264 schools. Training to parent/teacher associations on management and literacy and sensitization activities on the need for girls enrollment were carried out by GTZ . A scheme to give prizes to encourage girls to attend school was operational after 2005 and 3,226 girls received prizes by the end of the project . Gender sensitization modules were developed and incorporated into all teacher training programs . A series of activities were planned to improve literacy for adult learners and out of school youth . Of these only one was funded and implemented. Planned activities for unschooled and out of school youth were dropped as were post literacy activities in targeted villages .Twenty thousand adults (60% women) had received literacy training by restructuring, and a sound system for doing this was in place . . Outcomes : The national Gross Enrollment Rate (GER) for girls increased from 91% in 2004 to 108.0% in 2011, but it is difficult to demonstrate whether there was a direct link to project activities . The survival rate for girls at grade 6 (Intermediate Outcome Indicator) declined slightly by the end of the project . A number of the activities funded by the project were either small scale or there is no direct evidence of their impact during the lifetime of the project . The Ratio of girls to boys in primary school in targeted areas was only met for one targeted area after this indicator was introduced at restructuring . 3. Improving the management of the education system through administrative changes and reform and increasing efficiency . Substantial (Original and Revised Objectives ) Outputs A national agency for subsidizing community initiatives, the Community Initiative Support Agency for Education Development (APICED) was set up. A successful sensitization program for Parent /Teacher Associations was managed by Deutsche Gesellschaft fur Technische Zusammenarbeit (GTZ) and trained 3,420 Associations des Parents/Eleves. Technical support was provided to the National Curriculum Center . Ten CNC staff were trained to MA level. An Education Management Information System (EMIS) was set up at the Ministry of Education and a number of research studies on teacher practices in the classroom were carried out . A textbook management unit was set up and benefited from training and technical support and a textbook management guide was developed . It is not clear if this was distributed to schools or if training was given on the management approaches recommended .These activities are linked to quality improvements . A study on school grants implementation was carried out but the grant was not implemented, the NGOs not recruited and the APEs not trained . . Outcomes : A functional network of parent /teacher associations had been created and was operational, contributing to the decentralization goals of the project . Community Schools were successfully integrated into the mainstream school system, supporting the Decree signed in 2001 supporting this approach. APICED became fully operational and is now channeling subsidies to 13,630 Community Teachers. At project closure the Education Management Information System was beginning to produce national level data although not yet in a timely fashion . An Interim Education Sector Plan was nearing completion at closure . Improved research capacity and data availability meant that from 2009 onwards research findings were published in the Chad National Review of Education . The Ministry also carried out a number of evaluation studies of innovative sub components but information from these was not available to the reviewer. 5. Efficiency: The efficiency of the project is rated as MODEST partly due to the very long implementation delay of five years . No economic rate of return was defined for this project . A cost-effectiveness analysis was carried out based on the unit costs of textbooks, infrastructure,teacher salaries and the budget implications of project investments . Two options were identified at appraisal - the current situation at the time of appraisal and an alternative option simulating the effect of the inputs of the project . The Continuity Option would have led to a stagnation in the Gross Enrollment Rate at about 80%. It was envisaged that the Alternative Option would significantly improve the Gross Enrollment Rate and gender equity. Four scenarios were examined in terms of cost -efficiency. The project design team and the government followed Scenario 3 which focused on government support for more undertrained Community Teachers (CTs). The cost-efficiency arguments provided are based on sector analysis and not specifically for this project . They include options for improving secondary schooling and the transitioning of students from primary to secondary which were not addressed in the project , although it might be expected that if retention to grade 6 were higher, more students would move up the system. As project activities were only ever seen as part of a reform program it is not surprising that this initial phase did not accomplish all that was intended . In the data cited (p. 41, ICR), the internal efficiency of the education system remained low at project closure . The drop out rate at primary level was 10.8% in 2000, 17.4% in 2005 and 11% in 2010. The repetition rate was 27% in 2000, 19.6% in 2005 and 24% in 2011. The Primary completion rate was 35.3 in 2005 and 37.2 in 2010. Thus improvements were marginal. Government expenditures for education stagnated or declined between 2000 and 2005 and annual recurrent expenditures rose between those years . Capital expenditures set aside for education fell by 12.8%. In spite of oil revenues, poverty did not decline . Education budget execution improved however to 97% in 2009. The expected drop in the number of Community Teachers did not occur, with instead a substantial increase taking place (74% of all primary teachers). Only 63% of these have received training and few are certified . This impacts on the quality of education and the internal efficiency of the system (high dropout, high repetition rate and low primary completion). This is not a surprising outcome, but it should have been anticipated in the project design . Having more poorly trained teachers is not necessarily a way to get efficiency gains in education . It has become clear that simply having access to education services is not enough to get students to stay, move up the system and graduate . Parents weigh up the costs and benefits of sending children to school and if the benefits are low as opposed to the costs they will simply keep the children at home . This makes the system extremely inefficient . Unit costs for construction were underestimated at appraisal, with delayed contracting playing a part in increased costs. In addition the construction undertaken outside the project had far higher unit costs contributing to the inflation of construction costs in the latter years of the project . The ICR notes the negative impact of this on the efficiency of the project (p. 16). ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re -estimated value at evaluation : re- Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The relevance of objectives is rated as Substantial, both pre- and post-restructuring, since primary education was and remains a priority for the Government and the Bank, The original project design is rated Modest, both pre- and post-restructuring due to its inappropriate complexity and disconnect between indicators and project results . The Efficacy of the project was rated as Modest overall, both pre- and post-restructuring. Project Efficiency is considered Modest due to implementation delays, increased costs and little impact on the efficiency of the education system . Thus both pre- and post-restructuring outcomes are rated Unsatisfactory . Under standard IEG/OPCS procedures for assessing a project with revised objectives, the outcome of the project under the original objectives and revised objectives are rated separately and the overall Outcome is the weighted average of the two, with the weights based on the share of final disbursements at restructuring . On this basis, as 74% of the funds were disbursed prior to restructuring and 20% after restructuring, the overall Outcome rating is Unsatisfactory . a. Outcome Rating : Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: Stakeholder engagement at the local level is high and the desire for education improvement strong and progress has been made towards gender equity in staffing schools and in girls enrollment . Nonetheless with a fast growing population, reforms such as those envisaged under this project are still greatly needed . The need to balance rapid expansion with quality improvements is key to sustaining those improvements which have been made under the project. Project risk ratings were initially too optimistic as noted by the ICR on page 18 and risk ratings were increased at restructuring. The country is still at risk of political and economic upheaval . The influx of oil revenues has not been targeted to education services as anticipated . Primary education is not seen as a priority by the government and timely counterpart funding was and still remains an issue . Hence government ownership of the agenda could also be considered at risk. a. Risk to Development Outcome Rating : High 8. Assessment of Bank Performance: a. Quality at entry: The project preparation team made a considerable effort made to analyze country and education system needs . It tried to foster links between Chad and other African countries, in particular Mauritania and Mali in order to foster good practice in education, included numerous innovative components and sub -components and had an ambitious sectoral approach to education change in Chad . It also incorporated Lessons Learned from previous IDA supported projects, in particular the need to build the capacity of stakeholders and communities in school management and construction. However it was very overambitious given the recent emergence of Chad from a period of conflict, which in fact continued through much of the implementation The project was designed as the first part of a broad sector reform program and the most suitable World Bank lending product, an Adjustable Program Loan, was rejected as being inappropriate for the country capacity . The activities were therefore put into the form of a Sector Investment Loan . Tension between a programmatic approach addressing broad long term goals for the education sector and the Sector Investment Loan instrument with multiple components and interventions, meant that the project as a whole never hung together and the M & E aspects were confused. The Project Appraisal Document identifies weak management capacity in the Ministry of Education as a key issue (p. 21). Procurement issues which hampered implementation were known at project design, as was the potential for corruption and the fact that Chad was already a post -conflict country in 2003. The appraisal team attempted to mitigate the weak management and implementation capacity through utilizing multiple implementing agencies but this in itself caused difficulties . The fact that upfront risks were not adequately identified and two of the risks in the table on page . 28 of the Project Appraisal Document were rated only Moderate rather than Significant, meant that expectations for project success were not realistic at appraisal . This was also noted at the Mid-Term Review. The Overall Risk rating given at appraisal should have been S instead of M . The project was designed with multiple innovative interventions in Early Childhood Development and adult literacy, Radio Instruction and teacher payment and training . Unfortunately many of these were never funded and it is not clear why they were still included in the project when it went to the Board . at -Entry Rating : Quality -at- Unsatisfactory b. Quality of supervision: Pre- Pre - restructuring : The project was extensively supervised by teams of professionals which included a wide range of expertise. Supervision teams faced considerable obstacles over the life of the project including a temporary re-location to Cameroun for eight months in 2008. A two year extension was signed after the mid -term review, but although it was evident that further restructuring was needed and the mid-term review noted the need for revising indicators this was not accomplished until 2010. Information is not available as to why this took so long . Despite all the issues noted, supervision ratings were at S level for much of the project . These were downgraded to Moderately Unsatisfactory between May 2005 and June 2009. Activities were progressing slowly and there were lags in disbursement noted in some ISRs as well as procurement issues .Any other reasons for the MU ISR ratings are not included in the ICR . Supervision was hampered by a rebel attack in 2008 during which many of the project assets were destroyed including offices. During the seven months the Bank office was moved to Cameroun it was difficult to monitor project activities. Post -Restructuring : At re-structuring there were only two years of implementation remaining . The main activity which related to the broad programmatic objective, the adoption of an interim education sector plan, did not take place within the life of the project and a number of smaller activities (school grants, purchase of equipment for secondary schools) were also not completed. Overall, although individual activities happened ( a considerable achievement by the Task Team in the circumstances) and the money was disbursed, project impact on the issues addressed by the project, as measured by the project indicators, was either limited or still emerging by project closure, five years after the original closure date. More timely interventions on the part of the Bank team could have streamlined and simplified the project as appropriate for the situation which emerged after appraisal . Quality of Supervision Rating : Moderately Satisfactory Overall Bank Performance Rating : Moderately Unsatisfactory 9. Assessment of Borrower Performance: a. Government Performance: The project was beset by local upheavals, both political and in terms of project management . Six months after the launch workshop, rebels attached the eastern part of the country . Project effectiveness was delayed for nine months after Board approval and eighteen months after launch the project Coordinator was removed for corruption. The Project was suspended twice, once from September - November 2005 and again when all Bank projects were suspended between January and May 2006. Government political changes meant that there were twelve different Ministers of Education during the course of the project affecting continuity with counterparts . Although this situation stabilized during the latter part of project implementation, these upheavals meant that Government performance was not effective over the life of the project . Government Performance Rating Unsatisfactory b. Implementing Agency Performance: The project was managed by the Project Implementation Unit at the Ministry of Education but implemented by a considerable number of government agencies, communities with the help of Non -Governmental Organizations (NGOs) and by external agencies (GTZ). The coordinating unit gained capacity over the life of the project and by the end was able to implement most of the project activities . The many individual implementation units participated in supervision visits and provided status reports . However there were challenges with implementation capacity, M&E and component supervision, due to interruptions in communication, changes of staff and civil unrest. The Project Implementation Unit staff changed frequently and one coordinator was dismissed because of corruption. There appears to have been no issue with the implementation undertaken by GTZ (literacy). It is unclear how smoothly the implementation of the construction component by Non -Governmental Organizations and communities went as much of this was taken out of the project at a later date . Implementing Agency Performance Rating : Moderately Unsatisfactory Overall Borrower Performance Rating : Unsatisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: The original project design contained 10 PDO indicators and 22 intermediate indicators which was excessive and not all of these were relevant to project specific activities, "Only one out of the seven PDO indicators was specific to project indicators" as noted by the ICR on page 10. The Planning Directorate of the Ministry of Education (DAPRO) was responsible for the collection, analysis and dissemination of information . . A re-structuring paper dated December 7, 2010, notes that there were changes made at the April 2007 mid-term review that were not formally processed until the 2010 revision. However, for supervision purposes data were collected from 2007 to 2010 on a non-formally agreed project framework. It is not clear how this impacted project implementation and monitoring and evaluation activities . b. M&E Implementation: By project end the DAPRO was operational with key trained staff . However there were consistent issues throughout the life of the project with data collection, partly due to the size and limited communications in country . At project closure, capacity for data collection, analysis and utilization still remained weak at decentralized levels . Secondary data sources and collection units such as independent technical audits and other development partners contributed to data collection efforts over the life of the project . At re-structuring in December 2010, the revised key indicators were : 1. Primary level gross enrollment rate (GER) in targeted zones. 2. Ratio of girls to boys at primary level in targeted zones 3. The adoption of an Interim Education Sector Plan 4. The Validation of a curriculum for grades I to 4 5. Direct project beneficiaries which are female (%) of community teachers and students in primary public schools in targeted areas. The changes in these indicators reflect the shift away from a broad programmatic approach in the original PAD to one focused directly on those indicators which related closely to the remaining project inputs . Some of the project activities had never been started by re -structuring and some activities had been completed, hence these indicators were no longer relevant. The new indicator 3 retained the focus on overall education program development inherent to the concept of a broad policy based program . Thus the changed monitoring framework did not really affect the way the project could be evaluated . A number of baseline studies and evaluations were included in the project but not all were carried out . Analysis of innovative programs was not used in the ICR and was not available to the Reviewer . c. M&E Utilization: The use of M and E data for policy making was limited and possibly based on out of date population statistics . There were delays in the publication of national education data . However, data are now being published and the process of using these for policy analysis is improving . M&E Quality Rating : Modest 11. Other Issues a. Safeguards: Pre- Pre -restructuring : The project was classified as a Category B . An Environmental Mitigation plan was prepared which included actions needed to comply with recommendations . The Project Appraisal Document provides a good review of what is required to adhere to the plan . It is not clear from the ICR whether all the recommendations were carried out. The Environmental Mitigation Plan: "(a) outlines the institutional responsibilities for the implementation of the mitigation measures as well as for the monitoring of the implementation measures, a time horizon for these activities, and cost-estimates; and (b) includes monitoring indicators for latrines and water points as well as for Curriculum Center construction related activities " (p. 24, PAD). Post- Post -Restructuring : A second Environmental Management Plan was agreed during the 2010 restructuring relating to the construction of wells and latrines . This contained clauses relevant for the subprojects of the restructured projects . Apart from the above issue, the project team and the government seem to have taken the safeguard issues seriously and there were no reported safeguard issues during implementation, although with communities largely responsible for construction there may have been an issue in getting accurate data on construction works which were small scale and scattered.There is no further information on this aspect of project implementation . b. Fiduciary Compliance: The Project Implementation Unit was responsible for coordinating fiduciary issues with respect to all of the implementing agencies. It seems that communication issues with remote areas were partly responsible for delays in reporting. Overall financial management was assessed as satisfactory at the supervision mission in May -June 2012. However the firing of the Project Director for corruption early in the project and delays in procurement efforts and a qualified audit from the Bank indicate that there were issues over the project implementation period . c. Unintended Impacts (positive or negative): There may be evidence of beneficial spin off effects and potential for sustainability in the studies carried out and listed in the ICR document, but this information is not included . It would be interesting to know, for example if any of the nutrition and de-worming activities had a medium rather than short term effect on hygiene practices, or if the issues identified in the original beneficiary assessment with regard to the relations between parents and the Ministry of Education had improved. d. Other: The Risk assessment was also revised at restructuring to take into account the new School Grants mechanism and changed financial arrangements . 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Unsatisfactory Although the project managed, in Unsatisfactory difficult circumstances and over an extended period of time, to achieve some major outputs, it neither implemented all its components nor fully disbursed funds, nor did its activities impact substantially on major project objectives, such as girls equity and enrollments in targeted areas . Compounded with modest efficiency this leads to an unsatisfactory rating . Risk to Development High High Outcome : Bank Performance : Moderately Moderately Quality at entry was an issue. The Satisfactory Unsatisfactory inappropriate scope of the project led to significant problems in implementation. This was compounded by a delayed restructuring. Borrower Performance : Moderately Unsatisfactory Although implementing agency Unsatisfactory performance was often challenging, supervision was not so much an issue as the multiple changes in governments and subsequent Ministerial and other staffing changes in the earlier days of the project which is considered in its totality. Quality of ICR : Satisfactory NOTES NOTES: - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: Overall the ICR captures the lessons learned from this project . None of these are new to Bank operations . The choice of lending instrument is directly linked to successful project design . In this case the SI, chosen because of weak government capacity, was used as a substitute for an APL . The Project Development Objective was ambitious and too broad for country circumstances . The project M and E design must be realistic and implementable . In this case there were inconsistencies between objectives and indicators leading to confusion in the supervision team and the government and complicating the evaluation process . Realism in supervision ratings is essential . The supervision team were not realistic in rating the project S in its earlier years. Realistic project ratings might have alerted the Bank team to starting a re -structuring process earlier in the project history. Projects need to be rated on their success in achieving the PDO rather than on disbursement rates alone, although these are of course also marker of progress . Problematic supervision team changes : The Bank should be able to manage staff handovers successfully . It is quite possible that having local supervision capacity would have made a difference to the success of the project. A specific lesson learned not mentioned in the ICR relates to the difficulty of evaluating a project which takes place over such a long time frame, with so many external and internal variables affecting implementation and government performance. 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: Given the complexity of the original project components, the long history of the project and country context, the authors have made a credible attempt to capture the issues and their outcomes . It includes a good review of the country context and the efficiency issues . The document is also commendably realistic about the difficulties of project implementation pre-restructuring and the less than satisfactory outcomes . The ICR is commended for following the guidelines on rating restructured projects and applying the weighting calculations correctly for its ratings . a.Quality of ICR Rating : Satisfactory