ICRR 14260 Report Number : ICRR14260 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 03/31/2014 Country : Guyana Project ID : P089324 Appraisal Actual Project Name : Guyana - Education US$M ): Project Costs (US$M): 4.0 32.92 For All - Fast Track Initiative L/C Number : Loan/ US$M): Loan /Credit (US$M): 0 0 Sector Board : Education Cofinancing (US$M): US$M ): 0 0 Cofinanciers : Fast Track Initiative Board Approval Date : 09/28/2004 Catalytic Fund Closing Date : 08/31/2007 12/31/2012 Sector (s): Primary education (100%) Theme (s): Education for all (100% - P) Prepared by : Reviewed by : ICR Review Group : Coordinator : Pia Schneider Denise A. Vaillancourt Lourdes N. Pagaran IEGPS2 2. Project Objectives and Components: a. Objectives: According to the Letter of Agreement dated September 20,2004 (Para 1.1, page 3), the purpose of the grant is "to support the Recipient in attaining the goal of universal primary school completion for girls and boys by the year 2015." This purpose was cited as the objective of the grant in the subsequent restructuring papers . The project was developed in the early stages of the Fast Track Initiative /Bank collaboration when a Project Appraisal Document (PAD) was not required. b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components: The project financed the following activities : (a) Improving the quality of the teaching force in the hinterland (at appraisal US$0.35 million, at completion US$ 5.50 million including Additional Financing), comprising: (i) train qualified teachers using the Guyana Basic Education Teacher Training Distance education approach; (ii) provide continuous professional development for all teachers; (iii) establish satellite learning centers for teachers within a school cluster; (iv) improve the conditions for primary school teachers by providing teacher housing and remote area incentives that lead to salary top-ups for hinterland teachers and similar bonus payments as paid to comparable professionals in the hinterland. b) Enhancing the teaching /learning environment in primary schools (at appraisal US$3.30 million, at (b completion US$12.20 million including Additional Financing), comprising: (i) establish Escuela Nueva Learning Model and rewrite textbooks using the guidelines for the Escuela Nueva approach; (ii) establish child-friendly classrooms in coastal schools; (iii) improve the status of utilities across all schools (water supply, sanitary facilities, and electricity); and (iv) provide primary school students with textbooks . The Escuela Nueva program was dropped in 2010 after a pilot experience because the Ministry of Education found the costs of converting hinterland schools to Escuela Nueva Schools to be prohibitive . However, some aspects of the program were retained . (c) strengthening school /community partnerships (at appraisal US$0.22 million, at completion US$13.89 million including Additional Financing), comprising: (i) implementation of school improvement plans financed by school improvement plan grants (SIP Grants) to primary schools that meet the criteria set forth in the Operational Manual; and (ii) upgrade the existing school feeding program in the hinterland through school feeding grants to schools that meet the criteria set forth in the Operational Manual . (d) project implementation (appraisal US$4.0 million, at completion US$1.33 million), including carrying out required audits, monitoring and evaluation and project operational activities . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project Costs and Financing : The project was funded by the Education For All Fast Track Initiative (EFA-FTI) Catalytic Fund (CF).The original amount pledged was $ 12 million, but only $4 million was available at the time of approval (Letter of Agreement, 2004). The first two Additional Financing were to fulfill the pledge of $ 12 million. The third Additional Financing was made in 2009 in the amount of $20.92 million- bringing the total project cost to $ 32.92 million. With this amount, the Catalytic Fund accounted for 46% of total donor contributions to primary education between 2004 and 2008 and 72% of donor contributions to the financing gap for education between 2009 and 2012. Borrower Contribution : This grant from the Fast Track Initiative Catalytic Fund contributed to financing the gap in the government's education budget . Dates : The project agreement was amended five times to add the additional financing and the closing date was extended to allow implementation by 12/31/2012. The 11 project development objective indicators and target values were not revised during these restructuring . The amendments took place as follows : 1. February 2006: additional funding of US$4.0 million toward the original pledge. 2. May 2007: additional funding of US$4.0 million toward the original pledge. . March 2009: additional funding of US$20.92 and to extend the closing date to October 1, 2012. 3. 4. March 2012 to reallocate US$2.4 million funds planned for the Escuela Nueva program to the School Feeding Program (SFP). 5. July 2012 to extend the closing date from October 1, 2012 to December 31 2012 to allow for the completion of training and construction activities . 3. Relevance of Objectives & Design: a. Relevance of Objectives: Modest . The objective of universal primary completion was relevant to country needs at the project design stage. Guyana was a low-income country with a small population . Disparities in education between coastal and hinterland areas were significant, and teacher attrition a problem, particularly in the hinterland . Although gross enrollments in primary education was high, primary completion and learning outcomes were still a challenge . The objective is consistent with the Bank's Country Assistance Strategy for 2009-2012, the Guyana Poverty Reduction Strategy, the Guyana Education Strategy Development Plan 2008-2013, and Education for All goals for primary education. However, the project objective was met in 2006, two years into project implementation and a full seven years before the project closed . While continued efforts are needed to maintain this result, the objective should have been revised when it was achieved . Thus, at the time of project closure, the relevance of the objective was modest. b. Relevance of Design: Substantial . The EFA-FTI Program was originally designed to fill in gaps around on -going donor projects, targeting hard to reach regions and providing finance for activities where government funding was lacking . Project activities logically supported the objective of improved primary completion rate through inputs that would affect access to and quality of primary education . Teacher training, community participation and improvements in the educational environment, together with teacher remote area incentives reinforced each other to strengthen quality schooling. As this was not requested, the project results framework has not defined at the onset and monitoring and evaluation of progress was not set up; however, surveys have been made to collect additional information (see Section 11). 4. Achievement of Objectives (Efficacy): Attain universal primary school completion for girls and boys by the year 2015. 2015 . Substantial . The nationwide primary completion rate increased and reached the target of 100% from 85.5% in 2000 to 103.8% in 2012 for boys and girls. from 87.4% in 2000 to 104.6% in 2012 for boys from 91.6% in 2000 to 104.4% 2012 for girls The primary completion rate in the hinterlands increased from a baseline of 64.8% to 118.8%. Project activities contributed to the increased completion rates . Progress toward improved quality of teaching has been modest. While the teaching environment has been improved with the infrastructure investments, modest progress has been made toward improved learning outcomes . Community partnerships improved and impact evaluation show improved nutrition status . However, baseline values are missing for several indicators, which limits any analysis on changed performance . Quality of the teaching force in hinterland The percentage of trained primary school teachers in hinterland areas increased from 31.5% in 2000 to 46.9% in 2012. If this trend continues, then the target of 74% by 2015 will not be met. The primary school student/trained teacher ratio in the hinterland decreased from 80:1 in 2000 to 55:1 in 2012. No target was set. Data on teacher retention rates were not available, but would have given an indication of the success of the teacher incentives on the teaching force in rural areas . 64% of teachers surveyed said they could apply new techniques in the classroom, 54% said their knowledge of the curriculum improved . Poor communications in remote regions and the lack of an adequate communication plan meant that Learning Resource Centers were sometimes not used, as communities (and schools) were not aware of their services (page 20, Para 99, ICR). Issues with staffing, equipment and hours of operation led to Centers not being fully utilized . Enhanced teaching /learning environment 22 Learning Resource Centers were originally included in the project design . This target was revised to 11 during project restructuring. By 2012, 13 Centers were built surpassing the target . 90% of teachers said students were more active 91% of teachers said students performed better . Share of students nationwide scoring 50% or more on the National Grade Six Assessment improved from 2007 to 2012 from 20% to 35% in English and from 29% to 40% in Math. These fell far short of target set for each of these subjects . Learning assessment scores specifically for the hinterland were not available . Community partnerships and nutrition status improved The school feeding program reached 138 schools in the hinterland with training and technical support to provide one meal a day, benefiting 16,303 students (84%) of the target population. Studies conducted in 2009 and at the end of the project found that parent participation had doubled during that time from 39% in 2009 to 80% in 2012. Community participation in school affairs increased from 25% to 75% at the end of the program. The ICR attributes these results to the School Feeding program and the School Improvement Planning and Grants activities . 435 students and 1,930 parents and community members had served on School Improvement Advisory Committees at 342 primary schools (85% of hinterland schools and 75% of coastal schools). The 2007-2009 School Feeding program impact evaluation found that students in treatment schools had improved nutritional status and height by round three of the program . 5. Efficiency: Substantial . No economic analysis was carried out for the project at appraisal . The ICR (pg 13) compares the School Feeding Program with a Day Meal Program in India, and finds lower unit cost in Guyana . Similarly an analysis of unit costs for construction states that these were of satisfactory quality and competitively priced . The project strengthened the decentralization of education services . School staff gained experience in school improvement planning, grant writing, project management and implementation, which contributed to efficient use of project funds. Communities were also engaged in the financial management of projects and the school feeding programs, contributing to transparency . The integration of project activities into mainstream Ministry of Education departments helped strengthen those departments and build skills and a sense of accountability . These factors contributed to efficient implementation . Shortcomings included implementation inefficiencies including the rotation of regional officers which curtailed continuity in their relationships with communities and led to implementation delays, and inadequate capacity to complete all planned teacher training . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The relevance of objective at project closure is modest . The objective was achieved two years after project effectiveness, and could have been revised to accurately reflect the expected outcomes to be attained during the remaining time of project implementation . Relevance of design is substantial . The objective of achieving primary school completion for girls and boys was met and rated Substantial . Efficiency is rated Substantial . a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: The Government committed to a 30% budget increase for 2013 for the education sector, which helps to expand support for the school feeding program by fourfold . The Government continues to pay the remote area incentive to teachers. While donor support continues to the program, there are uncertainties to sustain the completion rate objective. The funding gap for 2014-2018 remains for primary education, and there has been a sharp decline in external sources . Limited capacity and priorities of Regional Education Departments may hinder progress. However, the completion rate objective has been sustained now for about eight years . Therefore the risk is rated moderate. a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: a. Quality at entry: The Bank supported the government in preparing the country proposal, including costing of resource needs . It helped the government getting the buy -in from other donors to bridge the financial gap . Bank technical assistance helped Guyana becoming one of the first countries to benefit from this FTI instrument . Risk factors identified during preparation included the migration of trained teachers from rural to urban areas, insufficient financial accountability and inadequate monitoring and evaluation . However, the Bank team did not develop a monitoring and evaluation framework for the project with a clearly defined results chain and underlying assumptions . Over-ambitious targets suggest the Bank team did not have a clear understanding of the scope and scale of the project . at -Entry Rating : Quality -at- Moderately Satisfactory b. Quality of supervision: The Bank supervised the project regularly, trained local staff in fiduciary and education matters, helped the government in the preparation of implementation reports, and provided technical assistance on the education strategy which enabled the country to receive additional financing from the Catalytic Fund . The Bank team ensured collaboration with local governments and communities and thereby built capacity . There were shortcomings in supervision . Bank team did not prepare a Project Document in 2008 during the re-endorsement of the country proposal and when additional financing was added, although this was required a the time. While the relevant operational, and fiduciary documents were prepared, the Bank still did not prepare a results framework for this grant . Even so the project objective had been achieved two years into effectiveness, the Bank did not rephrase objectives and targets . Quality of Supervision Rating : Moderately Satisfactory Overall Bank Performance Rating : Moderately Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: The government created a strategic planning committee at the Ministry of Education to government process in Guyana. The Ministry with other agencies and donors commissioned several studies to prepare the project. An Education Thematic Group, chaired by the MoE and with representatives from the Bank and other donors oversaw the project, which was also coordinated with other EFA -FTI activities on teacher training . The Government maintained education expenditures, and demonstrated ownership of project activities . The Ministry of Finance also fulfilled its obligations in terms of reporting, and there was a constant dialogue with the Bank team. Government Performance Rating Satisfactory b. Implementing Agency Performance: The project was implemented by a project team within the Ministry of Education . The team was highly committed, it conducted frequent supervision visits to the field, and monitored progress as required . It was resourceful in overcoming obstacles during implementation and built capacity among beneficiaries . The appointment of a qualified coordinator for the learning centers in 2009 improved performance of these centers. The project was administered in a sound fashion with no significant procurement or financial management issues reported, which helped Guyana receive additional financing . Implementing Agency Performance Rating : Satisfactory Overall Borrower Performance Rating : Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: The project did not have a standard World Bank monitoring and evaluation framework since EFA -FTI projects were not required to follow Bank guidelines at that time . Indicators were identified in the 2002 country proposal. However, for most indicators baseline values were missing and several targets set for 2015, beyond the closure of the project, or missing entirely . The design should have examined the effect of project activities on teacher retention and absenteeism. b. M&E Implementation: The MoE strengthened the Education Management Information System prior to the start of the grant . Monitoring & Evaluation was carried out by the Ministry staff, Regional Education Officers and School Administrators . Data were used by the Bank and other donors . Quarterly reports were delivered on progress to the Bank and the Education Thematic Group. A review of project outcomes was conducted in 2008 for the endorsement of further funding . Parent surveys were conducted in 2004,2009, and 2012 and a teacher survey was carried out in the hinterland in 2012. In addition the Project and the Bank financed an impact evaluation of the School Feeding Program between 2007 and 2009. In 2009 a survey assessed the usage and function of the Learning Resource Centers . Independent evaluations were conducted of the quality of civil works between 2011 and 2012. There were gaps in data collection on key-indicators, particularly on teacher retention and student performance, which affected analysis . c. M&E Utilization: The information collected formed the basis of project reports and informed about progress . Based on this information, additional EFA-FTI funding was approved. Data were also used to inform project and program activities. The evaluation of Learning Resource Centers led to a greater emphasis on the hiring and training for LRC Coordinators. Findings from the evaluation of the School Feeding Program and its impacts enabled donors to justify reallocating unspent funds for the Escuela Nueva program to an expanded SFP . M&E Quality Rating : Modest 11. Other Issues a. Safeguards: The project was given a Category C categorization for environmental safeguards and was not required to produce an environmental management plan . An Amerinidian People's Strategy was prepared as a condition for Grant Agreement signature. No safeguard issues were reported during the implementation period . In 2009, the Third Amendment and Re-Statement of the Grant Agreement document included specific wording to ensure that the project complied with Bank standard conditions for grants, including references to OP 4.12 relating to involuntary resettlement. b. Fiduciary Compliance: Financial management was satisfactory . Fiduciary compliance was monitored through quarterly interim financial reports. Quarterly audits were carried out for activities requiring cash transfers including the School Improvement, RAI and SFP activities. No major issues were identified by the Bank or external auditors . Annual procurement plans were prepared and submitted for review in a timely fashion and post reviews conducted by the Bank were in accordance with Bank procurement guidelines . Requirements for fiduciary compliance were also specified in the 2009 Third Amendment and Re-Statement of the Grant Agreement. c. Unintended Impacts (positive or negative): d. Other: 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Satisfactory Satisfactory Risk to Development Significant Moderate The development outcome has been Outcome : achieved two years into the project and since then maintained. Substantial increases in domestic financing to education and donor commitment to continue funding suggest a moderate risk. Bank Performance : Moderately Moderately Satisfactory Satisfactory Borrower Performance : Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The ICR notes a number of mostly generic lessons learned (pages 19 and 20). IEG agrees with the lessons and finds: - Projects should always have a solid monitoring and evaluation framework to ensure progress monitoring, even when procedures do not require this explicitly . - When project objectives are achieved and substantial funding is added, these objectives need to be adjusted along with the M&E framework to document what the additional financing is buying . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: As there was no official PAD for this project, the ICR used a set of outcome indicators drawn from the Indicative Framework of the original Grant, supplemented by indicators from teacher and parent surveys . The latter were included as key performance indicators in the ICR . The ICR was systematic and clear in its reporting of implementation progress. It also offered interesting lessons . The ICR was well written and concise . a.Quality of ICR Rating : Satisfactory