ICRR 14140 Report Number : ICRR14140 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 08/09/2013 Country : Haiti Project ID : P106621 Appraisal Actual Project Name : Ht - Meeting Teacher US$M ): Project Costs (US$M): 6.0 5.8 Needs For Efa L/C Number : CH375 Loan/ US$M ): Loan /Credit (US$M): 6.0 5.8 Sector Board : Education US$M): Cofinancing (US$M ): Cofinanciers : Board Approval Date : 05/29/2008 Closing Date : 01/15/2013 06/30/2012 Sector (s): Tertiary education (48%); Central government administration (35%); Sub-national government administration (17%) Theme (s): Education for all (100% - P) Prepared by : Reviewed by : ICR Review Group : Coordinator : Susan Ann Caceres Robert Mark Lacey Christopher D. Gerrard IEGPS2 2. Project Objectives and Components: a. Objectives: According to the Project Appraisal Document (p. 7), the objective was " to improve access and learning in basic education and to reach Education For All by 2015". This document (p. 7) went on to state that "to achieve this, a greater number of higher quality teachers is needed, among other inputs . This project would support the Government in the development and implementation of an accelerated teacher preparation program ". According to the Financing Agreement, the objective (p. 4) was to "assist the Recipient in developing and implementing the Recipient's' Accelerated Teacher Training Program to improve access to and learning in basic education, with a view to enable the Recipient to reach its goal of education for all by 2015". This Review uses the statement of objectives in the Financing Agreement as it is more monitorable . b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components: There were three components: Recruitment and Selection of Teacher Candidates (appraisal, US$ 0.332 million, actual, US$0.80 million) supported recruitment campaigns at the regional level under the management of the Departmental Directorates of Education (DDE) to find recent high school graduates, particularly those from disadvantaged rural areas, who might be potential teacher material . The campaign included radio, print media, and civil society networking. Efforts were made to build on the work of the Haitian Education and Leadership Program, which recruited highly talented, poor high school graduates into post -secondary education. Student-teacher selection was based on an entrance exam and other criteria developed at the national level and implemented at the decentralized level . Other criteria included: age, gender balance, willingness to teach in underserved areas, and examination of documents (e.g. comparing identity card photos with candidates, and diplomas with lists of graduates ). The current teacher preparation exam was revised to focus on candidates' understanding of key topics in the basic education curriculum and commitment to teaching profession. The recruitment process was to be evaluated . Accelerated Teacher Preparation (appraisal, US$ 4.095 million, actual, US$ 5.290 million) financed the revision and implementation of the Accelerated Teacher Training program, as well as the recruitment, professional development and payment of teacher educators, the teacher preparation program, and the certification and placement of student -teachers. No payment of student-teachers was planned (their stipends would be paid by the Government ). The APG I project financed the initial development of the curriculum. The curriculum was to be piloted in years one and two and revised in year three, as needed . The curriculum consisted of two months devoted to basic education content (Creole, French, mathematics, and science) and eight months devoted to subject -specific teaching methods, child development, classroom management and the evaluation of student learning . It also emphasized multi-grade teaching methods. Teacher educators were to be recruited by those currently working in Teacher Preparation Institutions . Teacher educators received professional development on the new curriculum and approaches and continuing professional development during the life of the project . It also funded the implementation of the teacher preparation program in selected Teacher Training Institutions . Institutions were selected based on a survey of capacities. After completing institution-based training, student-teachers were provided practice teaching to transition them into teaching . Certification was given after completion of practice teaching and national exam. The institution-based training and practice teaching was to be evaluated annually . Improved Management of Teacher Preparation (appraisal, US$ 1.573 million, actual, US$0.460 million) built capacity of the Department of Training and Professional Development (DFP) and other central and local structures of the National Ministry of Education and Training (MENFP) to manage teacher preparation . Decentralized structures provided field support to Teacher Training Institutions and schools for student-graduates. The Regional Education Departments (DDE) assisted in the recruitment of student-graduates, and the identification of primary schools for the practicum . The component financed the operating costs associated with management of the project, including an assistant coordinator to support the Accelerated Teacher Training . There were two separate level-two restructurings. On October 27, 2010 the targets were reduced because of capacity constraints as a result of the January 2010 earthquake. On April 27, 2012 the teacher training was scaled down to cover two cohorts rather than three, since the other was picked up by the Education for All Adaptable Program Grant. The targets were also lowered. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project Cost : After the 2010 restructuring, the scope of component 1, recruitment of teacher candidates, was reduced, because of the earthquake and the number of student -teachers failing the mid-term exam and/or dropping out of the program; however, costs for components one and two were higher than estimated, despite the reduction in trainees. Component two was also revised during the 2012 restructuring to include the payment of students-teacher stipends. Costs for component three were lower than appraised . Financing : The project was entirely financed by an IDA grant of US$ 6 million. US$5.8 million had been disbursed by project closure and the balance was canceled . Borrower Contribution : There was no Borrower contribution, nor were any planned . Dates: Dates The closing date was brought forward by six and half months to June 30 2012 from January 15 2013 since the project did not train the third cohort of teachers (see Section 2c above). 3. Relevance of Objectives & Design: a. Relevance of Objectives: Objective : Substantial Nearly one-half of Haitian children do not have access to primary education and the country is not likely to meet the Millennium Development Goals in primary education enrollment . The public sector provides only 20% of the primary schools in the country, while the remaining 80% of schools are private. There is an undersupply of teachers, another 5,000-7,000 of which are needed to absorb out -of-school children (ICR, p. 9). The objectives of this project were highly relevant to the Government's Poverty Reduction Strategy which reoriented education supply towards poorer students and introduced a certification policy for teachers and school directors. It was also highly consistent with the Government's Operational Plan for Education 2010-2015, which emphasized basic education, among other areas . The project was designed to meet the supply -side need for more teachers with higher capacity . There was another project implemented by the Bank which addressed another supply-side issue, rebuilding schools (Education for All Haiti Emergency School Reconstruction Project, 2009-2012, US$5.1 million) and another that addressed demand -side intervention in the form of tuition subsidies and school nutrition (Education for All Adaptable Program Grant, 2007-2012, US$24.8 million). Neither of these projects as well as other donor projects addressed the supply of teachers (e.g. USAID addressed transitioning youth from school to employment; CIDA addressed basic education access and quality; EU supported teacher professional development and classroom renovation; and IDB financed early childhood development ) (PAD p. 11 and ICR p. 5). The project's objective was substantially relevant to the Bank's Interim Strategy Note (2012-2013), which emphasized "quick wins" to provide public social services and the Country Assistance Strategy (FY09-FY12), which focused on increasing equitable access to formal and non -formal basic education. However, the objectives focused on outcomes for which the project could not reasonably be held accountable and contained a higher level objective, contrary to OPCS recommendations . b. Relevance of Design: Design : Modest There is a causal chain between the components (accelerated teacher training program ) and the objective of access. With more teachers, access should increase, provided there are no other constraints (e..g buildings, school fees, culture). Design assumed the teachers who completed the accelerated program would be hired, since they were certified; however, there was no project activity to match project teachers with school openings . Additional activities would be needed beyond an accelerated teacher program to improve student learning (e.g. books, learning materials, effective school management, continued professional development of teachers ). Improved student learning would also likely require a longer time frame than that afforded by the project . The Results Framework did not provide outcome measures related to improved learning and only proposed tracking the outputs of the teacher training program, some of which would not be available until after project closure . Student-teacher stipends were not included as part of project financing, which reduced the incentive to participate in the program . The ICR (p. 11) reported another design weakness : the timing of the availability of the student-teachers did not coincide with the school schedule, and this resulted in decreased training time for the first cohort of teachers . 4. Achievement of Objectives (Efficacy): To develop and implement the Teacher Training Program to improve access to basic education : Modest Outputs: The teacher training curriculum was revised and new modules were introduced during the project . The draft of the whole curriculum was produced and distributed to students and teachers . Teacher educators were trained in the new curriculum. Two cohorts from 4,574 qualified candidates were selected by eight regional Departments for participation in an accelerated teacher education program, which exceeded the target (3,600). The program included the advertising of teacher training to recruit student -teachers; receiving applications; conducting a qualifying examination for the applicants; making the final selection of teachers; ensuring that the training curriculum was ready for the teachers at the beginning of the program; and providing remedial training for those who needed additional preparation prior to the program . 1,390 new teachers (cohort one) who had completed high school and then the three -year teacher accelerated preparation program (of which two years was student -teaching) became certified, which exceeded the target (900 new teachers and at least 90% certified). The project team reported that this tripled the number of teachers which were certified annually. The ICR did not report the employment status and location of the first cohort of certified teachers, since the data were to be collected in 2013 (ICR p. iii). It was estimated that 60% of the newly certified teachers worked in rural areas, which met the downwardly revised target (40%). The project team subsequently reported that 204 teachers who participated in the accelerated program were recruited by public schools and 95 were teaching in private schools out of 1223 surveyed teachers. 1,526 teachers completed the first year of practice teaching (cohort two), which exceeded the target of 1,400 student-teachers. Both cohorts of student-teachers could reach up to 2.3 million students, if the second cohort completes the training and all of teachers from both cohorts continue teaching for 20 years and have at least 40 students annually. Outcomes: The ICR did not discuss how the project contributed to increasing enrollment and meeting the MDG for basic education. The project team subsequently reported that the project has had a positive impact on access to basic education, but provided no data . At this point in time, any increase in enrollment could be attributed to the Government's policy of universal free primary education, which began in 2012-2013, rather than to the teachers trained by the project, especially in view of the fact that most of the project's newly certified teachers were not yet employed at project closure . It is reasonable to assume that a greater number of teachers will be absorbed into the system in the future with the creation of new schools . In addition, the project team reported other efforts employed by the Ministry and a subsequent Bank project (such as open posts being identified and filled with newly qualified teachers; and the obligation, for schools participating in the program financed by the Bank project, to employ qualified teachers ), which should facilitate recruitment of the newly certified teachers . To develop and implement the Teacher Training Program to improve learning in basic education : Negligible The ICR provided no evidence of improved learning, attributable to the accelerated teacher training program . The project team reported that, as part of the follow -up project, an impact evaluation of the accelerated teacher training program is planned which will provide information on student learning . 5. Efficiency: While the project expended 97.7% of appraised costs, the scope of actual training was significantly reduced from what the Government had committed itself to in the context of Haiti ’s participation in the Highly Indebted Poor Countries (HIPC) Initiative (8,250 student-teachers in three years), as well as the number planned at appraisal (7,500 student-teachers in three years). The actual output of the project was 1,390 new teachers (and 1,526 student-teachers who partially completed the program ). While the ICR suggests that the Internal Rate of Return for the project was 31.2%, this number is likely overestimated based on some of the assumptions utilized (for example, that the number of teachers trained was higher than the number who completed the three year program during the project; and there was no allowance for students dropping out of school or taking longer to complete school ). Efficiency : Modest ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate Yes 31.2% 100% * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Relevance of objectives was substantial, and that of design modest . Efficacy was rated modest for one objective and negligible for the other. Efficiency was modest. The project revised the accelerated teacher training curriculum and 1,390 new teachers who completed the program were certified . Few of the newly certified teachers have been recruited into public or private schools . a. Outcome Rating : Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: The teacher training program supported by the project is becoming more established within the Teacher Training Institutions. The curriculum was revised and improved during the project . A follow-up project (US$8 million) supports continued implementation of the program with two more cohorts . However, the Ministry of Education struggles to provide basic services and will depend on donor assistance to continue the program and pay student-teacher stipends, which were needed to attract and retain student -teachers. The Ministry is currently revising the payment scheme for the stipends to lower the cost and increase sustainability . The Ministry is also considering making other changes to the program such as decreasing the student -teaching time. The quality of the skill development of the student -teachers will be a factor in the continuation of the program . The Teacher Training Institutions continue to need strengthening to improve their monitoring of the student -teaching. The Ministry has not yet made the accelerated program the standard approach . Considering all of these factors, the risk to the development outcome is significant . a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: a. Quality at entry: The Bank team developed the project with urgency, since they faced a "use it or lose it" situation to comply with HIPC Initiative requirements (ICR, p. 2). As a result, there was not sufficient time during preparation to address issues in the curriculum or build ownership of the approach (ICR p. 5). Lessons from previous projects in Guinea and Senegal were incorporated (for example, requiring teacher candidates to have a high school diploma and providing resources for the supervision and monitoring of the student -teaching program). Preparation engaged with stakeholders to communicate about the training and certification program and used their feedback to establish the sequence of steps in the process . However, there was insufficient understanding of the project's end -goal among stakeholders, which created reservations and misunderstandings and resulted in implementation delays (ICR, p. 31). The risks were appropriately identified, but design and mitigation measures did not realistically address them (for example, the capacity of the teacher training institutions ). Student-teacher stipends were not included as part of project financing, which reduced the incentive to participate in the program and was a miscalculation on the part of the Bank . There was an issue of the timing of the training, which related to imposed requirements of HIPC . The training targets were optimistic and did not consider student drop -outs or examination failure. at -Entry Rating : Quality -at- Moderately Unsatisfactory b. Quality of supervision: The Bank team provided intensive supervision, since this project required correct sequencing of events . The Bank team monitored the performance of teacher training institutions . After the earthquake, the Bank team hired consultants to evaluate the Teacher Training Institutions' physical infrastructure, as well as managerial capacity. After a year of low supervisory ratings, the Bank recruited four consultants and an assistant project coordinator, which assisted in improving implementation and provided support with the curriculum reform and setting up the teacher practice program . The team also put in place a plan to support improved implementation. Two level two restructurings addressed implementation challenges such as providing more incentives to participate in the program with the provision of student -teacher stipends (and retroactive payment to 2011) and worked with the more capable teacher training institutions, as well as providing additional support to build the capacity of the institutions . An overall portfolio review was carried out by the Bank, which allowed the shifting of the third cohort to another Bank project, thus permitting the timing of the training for this cohort to coordinate with the school year, which was not the case with this project . The Government reported that the Bank team helped to improve project procurement and financial management, but said there were occasions where the response time for "no objections" was too long (ICR p. 34); however, the length of the delay was not specified . However, nothing was done during restructuring to address the impossibility of achieving the second objective (improved learning) during the project’s time frame. There was no attempt to collect data on learning . Quality of Supervision Rating : Moderately Satisfactory Overall Bank Performance Rating : Moderately Unsatisfactory 9. Assessment of Borrower Performance: a. Government Performance: To qualify for the HIPC funding, the Government had to commit to recruiting at least 2,750 teachers annually for three years. The Government and Ministry of Finance were supportive of the project even after all of the disruption in the aftermath of the January 2010 earthquake which had major repercussions on the project . There were three changes in Ministers of Education during the project, which had a negative impact on implementation, since each Minister had a different view towards key aspects of the training program such as allocation of stipends, duration of the training, and recruitment process . There was lack of coordination between Ministries, which negatively affected the payment of teacher -stipends and led to a decline in morale and a student-teacher strike. The Ministry of Planning constantly delayed approvals and was sporadic in approving the release of funds for the Ministry of Education to provide the stipends for which the Ministry of Finance had allocated resources in the budget . Government Performance Rating Moderately Unsatisfactory b. Implementing Agency Performance: The Ministry of Education was the implementing agency and utilized technical units within the Ministry to implement various components of the project, mainly the Department of Training and Professional Development, which was a departure from creating an external unit and was meant to strengthen capacity within the Ministry. The Ministry was responsible for consultations with non -public stakeholders and for providing sites for practice teaching . Administrative and fiduciary support was provided by the Project Technical Unit of another Bank project and was also supported by the Department of Administrative Affairs of the Ministry. The Department of Training and Professional Development Department was unaccustomed to managing such a large project. The steering committee and technical committees were not established and functioning, which led to a series of delays . The January 2010 earthquake negatively impacted the Implementing Agency and project implementation . The offices of the Ministry collapsed in the earthquake . The Project Coordinator, Director and Assistant Director of the Department of Training and Professional Development were killed in the earthquake . The teacher training institutions were responsible for training, assessing students, and monitoring placement of student-teachers. The Institutions were not able to provide adequate supervision for the student -teaching portion of the program. Efforts were made to strengthen the Teacher Training Institutions . Turnover of personnel in the Ministry made it difficult to maintain capacity building efforts . Project management did not work well in the beginning, but improved as new people were hired . The fiduciary environment was weak in relation to budget process and financial control (See 11b). However, the Ministry worked with the Project Technical Unit to ensure the contracts with the Teacher Training Institutions were implemented. Implementing Agency Performance Rating : Moderately Unsatisfactory Overall Borrower Performance Rating : Moderately Unsatisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: The PAD contained two sets of indicators for the Results Arrangements and Results Framework (ICR, p. 2). Design did not specify outcome measures related to improved learning . It tracked the outputs of the teacher training program, many of which could only be measured at the end of the student -training process -- after the project was completed -- as a means of measuring improved access . The project appraisal document (p. 39) specified baseline data for some of the indicators, as well as responsibility for data collection . A number of evaluations were planned (e.g. recruitment process, institutional training, practice teaching ). b. M&E Implementation: During implementation a number of targets were reduced, as restructuring changed the scope of activities (e.g. training two cohorts of teachers rather than three ). M&E functions were carried out by the Project coordinator and the assistant coordinator, who tracked progress related to key indicators . The number of teacher recruited, enrolled, graduated, and certified were all tracked by the Teacher Institutes . Teacher stipends were also tracked, but were not monitored as closely as the key results indicators (ICR p. 8). Planned evaluations were not implemented (e.g. recruitment process, institutional training, practice teaching ). c. M&E Utilization: Monitoring of key performance indicators showed that project targets needed to be downwardly revised, which was done during restructuring . The Bank utilized the information collected about teacher stipends (e.g. payment and non-payment) to allocate finances to cover them . M&E Quality Rating : Negligible 11. Other Issues a. Safeguards: This Category “C� project did not trigger any safeguard policies (PAD, p. 28). b. Fiduciary Compliance: The ICR reported satisfactory financial implementation of the Ministry of Education and Teacher Training Institutions contracts; however, the ICR also stated "despite some progress in building capacity the overall fiduciary environment remained risky, with weak budget processes and financial controls throughout government, and especially in the Department of Administrative Affairs in the Ministry of Education ." (ICR, p. 8) There was compliance with audit requirements and the audits were unqualified and submitted to the Bank in May 2012. Financial reports were provided as required for the release of each tranche . Software malfunctions prevented adequate tracking of disbursements and resulted in delays in the presentation of financial information (ICR p. 35). There was compliance with the procurement plan, although delays were noted . There were no reported cases of misprocurement. c. Unintended Impacts (positive or negative): The lack of funding for the student -teacher stipend was a serious impediment to the project . d. Other: 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Relevance of objective was substantial, Satisfactory Unsatisfactory while design was modest. Efficacy was rated modest for one objective and negligible for another. Efficiency was modest. Risk to Development Significant Significant Outcome : Bank Performance : Moderately Moderately There were significant shortcomings in Satisfactory Unsatisfactory Quality at Entry (see Section 8a above). Based on the Harmonization Criteria, the outcome rating determines the overall performance rating when there is a split between the ratings for Quality at Entry and Quality of Supervision. Borrower Performance : Moderately Moderately There were significant shortcomings in Satisfactory Unsatisfactory both Government and Implementing Agency performance (see Section 9 above). Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The ICR (p. 15) provided several lessons, which IEG has synthesized as follows : Accelerated teacher training can be effective when there is an undersupply of teachers . This program was found to be effective because student -teachers were in the classroom one year after pedagogical training. The program required candidates to be high school graduates and pass an examination, but would have been more effective if the training had been coordinated with the academic year . To be able to recruit, maintain, and motivate student -teachers in an accelerated teacher program, stipends were needed. Failure to disburse the stipend demoralized and disillusioned the student -teachers and jeopardized the teaching program . The Bank needs to ensure that timely payments are made . The follow-up project has utilized progressive implementation of stipends (e.g. no stipend in the training year, one-half stipend during the first year of student -teaching, and full stipend during the second year of student-teaching), which partially takes account of the fiscal constraints . The Borrower also provided several lessons, which IEG has synthesized as follows : Time and effort spent on seeking a consensus between the stakeholders is not wasted . In the Haitian education sector, any innovation quickly arouses suspicions and is met with significant resistance . An understanding of the causes of this resistance and the issues faced by the different stakeholders is needed to improve project outcomes . Projects need to be spearheaded by the public sector with the aim of strengthening the country's institutions in the long term. There is a high level of incompetence in the public sector, but this should not lead to the creation of parallel project management structures . The relationship between consultants and civil servants in this project was troubled, due to the fees earned by the consultants in comparison to civil servants' salaries. Bonuses for civil servants involved in projects are being evaluated . 14. Assessment Recommended? Yes No Why? A project performance report would provide another opportunity to examine the ratings and could follow up on learning outcome and access results with the passage of time . 15. Comments on Quality of ICR: The ICR is satisfactory. It provides a candid assessment of the implementation shortcomings, as well as Bank and Borrower performance. However, it did not provide data related to attainment of the project development objectives, and was constrained by a weak Results Framework . The ICR only described the indicators, rather than a broader discussion of attainment of objectives . The ICR used a weighting scheme to determine ratings, which is not required, since the project development objectives were not revised . The Borrower comments were very useful and used to supplement this Review . a.Quality of ICR Rating : Satisfactory