Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review STP- Quality Education for All Project (P146877) Report Number: ICRR0022044 1. Project Data Project ID Project Name P146877 STP- Quality Education for All Project Country Practice Area(Lead) Sao Tome and Principe Education L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-H9030,IDA-H9770,TF-16391 30-Jun-2017 4,945,916.85 Bank Approval Date Closing Date (Actual) 20-Dec-2013 28-Jun-2019 IBRD/IDA (USD) Grants (USD) Original Commitment 900,000.00 1,100,000.00 Revised Commitment 5,374,912.60 1,080,362.85 Actual 4,945,916.85 1,080,362.85 Prepared by Reviewed by ICR Review Coordinator Group Rocio Manchado Judyth L. Twigg Joy Behrens IEGHC (Unit 2) Garabito 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The project development objective was to improve the system of in-service teacher training and to strengthen the management of human resources in the education sector in São Tomé and Príncipe (Financing Agreement, 09/23/2014, p. 5). Page 1 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review STP- Quality Education for All Project (P146877) Some outcome targets were increased at a June 2014 Additional Financing (AF), and one outcome indicator was revised at a 2017 restructuring. However, because the project's scope increased commensurately with the AF and did not change substantially at the 2017 restructuring, a split rating is not necessary. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 27-Jun-2014 c. Will a split evaluation be undertaken? No d. Components Component 1: Improving the System of In-Service Teacher Training in Primary Education (Appraisal: US$1.1 million; Revised Cost with AF: US$4.2 million; Actual: US$3.4 million) This component had two subcomponents:  Setting up of an in-service teacher training system, including: (i) the preparation and implementation of a competence-based training framework; (ii) the development of education training plans and modules, and carrying out of related training of approximately 400 primary education teachers; (iii) the setting up of a teaching certification process and carrying out of related academic and pedagogical competence evaluations; and (iv) the setting up of a proper institutional structure within the Ministry of Education (MoE) to manage teacher training, including the establishment of a certification body and the provision of equipment, capacity building, management tools and technical assistance. In parallel, an evaluation of all primary education teachers' academic and pedagogical competence, covering 1,113 teachers, was to be carried out.  Development of a Student Learning Assessment, including: (i) the development and implementation of a learning assessment road map and provision of related training, management tools, and materials to MoE staff and teachers for the administration of tests; (ii) the administration of the Early Grade Reading Assessment (EGRA) for the 3rd grade of primary education; and (iii) the pilot of an early-grade small scale impact evaluation assessment in approximately 20 schools. Component 2: Strengthening Management of Human Resources in the Education Sector (Appraisal: US$0.9 million, Revised Cost with AF: US$1.3 million; Actual: US$1.6 million). This component had two subcomponents: Page 2 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review STP- Quality Education for All Project (P146877)  Development of a Management Information System within the MoE, including: (i) provision of technical assistance, training, and equipment at the central and decentralized levels; and (ii) strengthening of MoE’s human resources policies and management capacity for the allocation, management, and supervision of teachers, including the provision of management tools, re-definition of roles and responsibilities of education professionals, and re-launching of the pre-existing supervision structure. The new Education Management Information System (EMIS) was to be installed in the Directorate of Planning and Education Innovation.  Support for project management. Revised components: All the new activities under the 2014 AF were added to Component 1 to expand the training activities to: (i) allow for the delivery of a more complex distance learning program using technology; (ii) develop training programs not only for primary school teachers but preschool teachers and teacher trainers as well; and (iii) reach more teachers and especially those in rural areas far from the capital. Several PDO-level indicator targets were scaled up: the number of primary teachers to be trained (from 400 to 667), and the total number of direct project beneficiaries (from 2,000 to 2,500). The learning assessment was revised to include an additional grade. The certification process was scaled up to include preschool teachers, and development of a preschool curriculum was added. The 2017 restructuring provided additional time for project implementation and conformed with a MoE decision to reduce the training target (from 667 to 515) to include only unqualified primary school teachers who were not already enrolled in other training, and teachers who had more than three years until retirement. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project costs and financing: According to the PAD, (p.ii) the project was to be co- financed by an original IDA grant of US$ 0.9 million and a Global Partnership for Education (GPE) grant (TF-16391) of US$1.1 million, for total originally planned costs of US$ 2 million. IDA resources were found to be insufficient at the time of appraisal (ICR, p. 12), and therefore one month after both grants were effective, an AF IDA grant of US$ 3.5 million was approved (ICR, p. 52), bringing total planned project costs to US$ 5.5 million. The ICR did not explain why it was discovered only one month after project effectiveness that additional funds were required. At the 2017 restructuring, the Financing Agreements for the original IDA and GPE grants were not extended. However, the project description for the AF was revised to finance all remaining activities. Actual project costs were US$ 4.95 million; the ICR did not explain the difference between planned and actual costs. No Borrower contribution was planned or made. Page 3 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review STP- Quality Education for All Project (P146877) Dates: The project was approved on December 20, 2013 and became effective on May 19, 2014. A mid-term review was undertaken on May 2, 2016. The project was restructured two times:  June 27, 2014, to accommodate AF, add new project activities, and amend the results framework accordingly.  May 16, 2017, to extend the closing date two years from June 30, 2017 to June 28, 2019 and allow the project to complete all planned activities, to introduce modifications to project indicators and target values, and to change the implementation schedule. Funds were reallocated between disbursement categories.  The project closed on June 28, 2019. 3. Relevance of Objectives Rationale The PDO were relevant to the country context at appraisal. The education sector has been and remains a priority. However, despite a decade of robust growth, poverty was a serious concern in the country, and lack of education appeared to be a significant determinant of poverty. In 2010, the poverty rate for heads-of- household with no formal education was 73.8 percent. The proportion of public expenditure in the sector increased from 2.7 percent of GDP in 2002 to 8.8 percent in 2010; however, government funding and donor support to the education sector failed to address important aspects of the country's Education Sector Plan (ESP, 2012-2021). By providing additional resources, the proposed project complemented those interventions and contributed to fill the financing gap. The project focused on quality of primary education and the efficiency and management of the system. Although São Tomé and Príncipe had achieved universal primary education, and with that, gender parity, the education sector still faced many challenges with respect to efficiency, quality, and governance. The quality of education was low due to lack of provision of adequate academic and pedagogical competencies leading to a high number of primary education teachers unqualified (60%). The management of the education system was weak because there was no reliable human resources management system to oversee recruitment, assignment, and career development. There had not been systematic planning and implementation of a complete learning assessment with a mechanism of feedback for policy analysis and decision-making. There was no reliable education management information system to link the center with decentralized structures. Inspectors and pedagogical advisors rarely visited schools and classrooms, and when they did visit, they lacked management and reporting tools to effectively support teachers’ development. The project addressed quality of teaching, which is one of the most important determinants of quality of learning in the longer perspective. Also, improvements in learning outcomes would positively impact sector efficiency through reduction of repetition and dropout rates, and their associated costs. Page 4 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review STP- Quality Education for All Project (P146877) The project’s PDOs were relevant to the government's education strategy, as outlined in the Letter of Education Policy (2012-2021) and operationalized in the ESP, as they focused on the quality of primary education, a declared priority. They were also relevant to the Bank's Country Partnership Strategy (CPS) at appraisal (2014-2018) in supporting implementation of the Second Poverty Reduction Strategy Paper 2012- 2016, targeting the promotion of quality education for all through its pillar III (human capital development and improvement of social service delivery). The 2014-2018 CPS is the Bank's most recent strategy document for the country. The project was aligned to the key priority areas of GPE, as it proposed to target issues of quality of education and learning outcomes. At closing, the project continued to be relevant as it responded to the main challenges that the sector is facing today. The current ESP (2019-2023) focuses on the improvement of the efficiency and equality of the education system by addressing low learning outcomes at all education levels. The priorities of this ESP are: (i) improving teacher standards, (ii) strengthening the use of evidence-based decision-making, and (iii) improving efficiency and the overall management of human resources. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To improve the system of in-service teacher training Rationale To achieve improvement in the system of in-service teacher training, the project’s theory of change proposed creating a teacher qualifications framework to guide training content and establishing a certification process for primary school teachers. Additionally, the system for pre- and in-service teacher training was to be based on a common framework that would include minimum content knowledge and competencies of teachers, as well as standards as to how teachers were to be evaluated. This project also supported under this objective the development of a learning assessment system, to use the results as inputs to guide teacher training and professional development. Outputs Page 5 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review STP- Quality Education for All Project (P146877)  An assessment of critical competency needs of teachers was finalized. It served as the basis for the content included in the teacher training program.  Competency-based training plans for in–service teacher training were developed, including a distance learning program. Nine distance learning training centers were established. The original project focused primarily on face-to-face training, but as many teachers lived outside the capital and were unable to participate due to distance and cost, the AF supported distance learning.  A certification process for pre-primary and primary school teachers was developed and approved, including the official adoption of a competency test according to the teacher qualifications framework. Associated regulatory/legal frameworks were developed, but due to a change in the government, they were not approved.  312 pre-school teachers were trained, exceeding the revised target of 175. A total of 26 preschool teachers traveled to Brazil to receive training on the new preschool program, while 100 were trained on the new preschool curriculum at the Institute of Higher Education and Communications (ISEC), and 186 participated in face-to-face sessions on Portuguese and Mathematics. The project also purchased various teaching materials to accompany the new program, and printed manuals for the new curriculum.  123 people were trained through "training of trainers" (included ISEC staff, supervisors, instructional leaders and inspectors), exceeding the target of 100.  A student learning assessment system was established. A new assessment unit was created within the Ministry of Education, the Department of Assessment and Accreditation. This unit carried out assessment exercises, and put a plan in place to perform the assessment every 3 years.  There was a total of 2579 direct project beneficiaries (50% female), exceeding the revised target of 2500, including: 123 trainers/supervisors; 312 pre-service and in-service preschool teachers; 528 in- service primary school teachers; 528 administrative staff and school directors; 110 inspectors, instructional leaders and supervisors; 20 ISEC administration staff and teacher trainers; 708 additional in-service primary school teachers; and more than 250 department leaders and technical staff within the central MoE. Outcomes  528 primary teachers participated in teacher training, exceeding the revised target of 515. Of the 515 teachers who participated in the distance learning program, 499 passed the course, with a minimum passing grade based on attendance and performance on class assignments. 90% of teachers who participated in the distance learning program were highly satisfied with its content (ICR, p. 16).  Two learning assessments were conducted for grades 2, 4, and 6. The ICR did not provide information on the results of those learning assessments, and it did not describe whether the initial intent of using those results to inform teacher training content and practices was realized. Although a distance learning system was established and the number of teachers trained through this system exceeded the target, the planned certification process for teachers was not put into place. Information on the extent to which system improvements were demonstrated, through improved skills of teachers, application of improved teacher skills in the classroom environment, or improved student outcomes, was not provided. In addition, it is not known whether learning assessment data were used to Page 6 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review STP- Quality Education for All Project (P146877) improve the system of teacher training. The ICR (p. 19) noted that "most of the trainers delivering the training at ISEC lacked any higher degrees in education, and the courses delivered were not accompanied by a rigorous end-of-course evaluation to determine learning gains as a result of the training." Due to lack of information on how much teachers actually learned and whether any learning gained was put into practice, achievement of this objective is rated Modest. Rating Modest OBJECTIVE 2 Objective To strengthen the management of human resources in the education sector Rationale The theory of change for this objective was based on the premise that improving the capacity to produce timely and reliable evidence for better resource allocation and management would strengthen the management of human resources in the education sector. The interventions were designed to improve the supervision system, including inspection, school data collection capacity, and analytical capacity at the MoE. Outputs  The percentage of primary education schools that benefit from four annual school-level supervisions and a mentoring program increased from 0% in 2014 to 100% in 2019. This includes visits made by supervisors, inspectors, and instructional leaders. The old inspection system was analyzed, a revised model was proposed, new tools were developed and aligned with the new system, inspectors were trained, and over 300 inspections took place using these new tools.  A new EMIS was established, and initial data from all schools has been collected and entered into the system. School directors have been trained, and all schools with electricity have been resourced with a computer and internet connection. Outcomes  Daily working hours by teachers increased from a baseline of 3.50 in 2014 to 4.5 in 2019, meeting the target. A change in policy, as well as increased supervision, helped to increase the overall number of daily teaching hours. Rating Substantial Page 7 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review STP- Quality Education for All Project (P146877) OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale Supervision capacity and infrastructure developed by the project have substantially improved the management of human resources in the education sector. The ICR did not provide sufficient information on the results of teacher training for a robust assessment of improvement in the teacher training system. With one objective substantially achieved and the other modestly achieved, but strong achievement of outputs on teacher training making it likely that positive outcomes could be demonstrated with additional data, overall efficacy is rated Substantial. Overall Efficacy Rating Substantial 5. Efficiency The PAD (p.18) provided an economic and financial analysis that assessed the macroeconomic context, as well as an economic analysis that found that improvement of learning outcomes would have a substantial impact on student’s earnings and economic growth. Also, it presented evidence about a positive relationship between teacher training and learning outcomes, especially when enhanced with the strengthening of teaching practices and the EMIS. In addition, the PAD provided information about the cost per unit of in service-training under the project and the sustainability of this training system. The cost-benefit analysis (CBA) in the ICR focused on the quantifiable parts of the project’s total investments, primarily the benefits expected from improving the quality of preschool and primary school teachers. The teacher training program corresponded to 60% of actual disbursement. The CBA demonstrated that the project was economically viable, with a net present value (NPV) of US$2.112 million for the costs and benefits of the training (lower than the NPV of the quantifiable benefits), an internal rate of return of 12 percent (higher than the official discount rate of 9 percent), and an overall cost-benefit ratio of US$2.1 returned for every US$1 invested. The cost-effectiveness analysis showed that the project was modestly cost-effective. Although most of the activities under the project were kept under budget, project management costs represented 108% more than was estimated at appraisal, and 15% of the total project expenditure. Even though the overall estimated annual cost per student receiving training at ISEC decreased from US$673 in 2016 to US$ 114 at the end of the project, the distance learning program costs per primary school teacher and teacher trainer were significantly higher (US$2,354), corresponding to 70 percent of the annual cost of sending a Page 8 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review STP- Quality Education for All Project (P146877) tertiary student abroad (US3,407). The cost of the equipment for the nine distance learning training centers as well as the cost per hour of distance training was significantly much higher than the cost per hour of face-to-face training, and higher than similar programs in comparator countries; this led to the reduction of the number of training hours under the project from 450 to 375 due to limited resources. There were many factors that contributed to these higher costs, including the small size of the potential market limiting the pool of applicants for the training contract (ICR, p. 19). There were also some important shortcomings in cost-effectiveness during implementation. For instance, a change of Task Team Leaders during the first year of the project's life led to delays. Also, a lack of technical and financial decision coordination among institutions led to tensions between the agencies involved, and consequent delays in project implementation (ICR, p. 24). The lack of a reliable internet connection impacted the delivery of the distance learning program and EMIS (ICR, p. 26). This problem was solved by giving control of the routers to a service provider, connecting the internet directly to computers, and making an offline version of the EMIS was available to allow schools to continue work when having connectivity issues. Challenges with procurement (see Section 10b) were also indicative of implementation inefficiencies. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 60.00 ICR Estimate  12.00  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The relevance of objectives is rated high across the entire project, as the objectives continued to be consistent with Bank and government strategies, and responsive to the needs of the country. Overall efficacy is rated substantial, with one objective substantially achieved and the other modestly achieved, but strong achievement of outputs on teacher training making it likely that positive outcomes could be demonstrated with additional data. Finally, the efficiency is rated modest based on high training costs and inefficiencies during implementation. These ratings are consistent with moderate shortcomings in the project's preparation and implementation, producing an Outcome rating of Moderately Satisfactory. Page 9 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review STP- Quality Education for All Project (P146877) a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome The project's contributions are expected to be consolidated through the next education project in the country, a proposed Girls' Empowerment and Quality Education for All Project (P169222), which will include teacher professional development, additional distance learning training courses, support for teacher/school management and accountability, further development of the national learning assessment system. The continuity of the development outcomes is also supported by the technological infrastructure that was developed, as well as the staff who were trained to use the new EMIS and online platforms for data collection and reporting. Institutional strengthening, such as the implementation of the National Large-Scale Assessment in Basic Education, the establishment of an official MoE unit responsible for learning assessments, and the adoption of an overall assessment strategy, as well as plans in place to continue to implement national assessment on a regular basis, will also consolidate the project's achievements. However, there are continued risks. There is ongoing high turnover of MoE staff, as well as challenges in financing recurrent costs for internet connectivity and capacity required to maintain the equipment to manage the EMIS and deliver the distance learning program. São Tomé and Príncipe recently officially joined the Bank’s Human Capital Project and is developing a multisectoral plan to guide all investments in this area; this commitment is accompanied by a significant increase in resources for education that will mitigate many of the potential risks to sustainability (ICR, p. 33). 8. Assessment of Bank Performance a. Quality-at-Entry To be implemented in a three-year timeframe, the project had a simple design that was based on evidence provided by the Education Sector Analysis Report 2012. The PDOs were well defined and aligned with the ESP (2012-2021). However, there were important shortcomings regarding project design. At appraisal, IDA resources were insufficient to fund the activities proposed. The timeline for the AF, prepared in just two months and approved only one month after project effectiveness, was unrealistic given the proposed scale-up of project activities and the addition of new ones (ICR, p. 12). Some diagnostic studies required to define project activities during the design and planning phases were not completed until project implementation. These studies included: (i) the assessment of teachers’ competencies, (ii) the analysis of the inspection system, (iii) a feasibility study regarding information technology requirements for distance learning, and (iv) an in-depth study of ISEC in terms of its staffing, budget, and training programs. These delays negatively impacted the implementation of the project, in particular the distance learning and teacher certification processes. Page 10 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review STP- Quality Education for All Project (P146877) Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision The Bank team worked proactively and closely with the government, ISEC, and the Fiduciary and Implementation Agency (AFAP) across the project implementation period. During the mid-term review, an in-depth project analysis to solve issues related to the design and implementation of scaled-up and new activities added with the AF led to a restructuring. Although there were 11 ISRs, the ICR (p. 31) considered the submission and filing of aides-memoire weak during project preparation and the first stages of implementation, through there was gradual improvement over time. Immediately after project effectiveness, there was a change in the MoE and TTL that led to some delays. Also, there were confusion and initial implementation delays due to inconsistencies in keeping the PDOs the same after the AF but not fully articulating and describing project activities in project documentation. The team spent a significant amount of time working with the MoE to secure funds for these activities, and to reach joint agreements about different aspects that had not been clearly defined during the design and planning phase (ICR, p. 25). The team provided fast solutions to the challenges that arose during the life of the project. For instance, during a supervision visit to the teacher training centers, they found out that although the AFAP had previously reported that all pending payments had already been processed, the teachers had not received the latest payments. To solve the problem, the team established a system to file grievances/complaints and conducted a detailed review of the processes to determine the cause of the delays. An action plan was implemented to resolve pending payments in a timely manner and to ensure the delivery during the final two remaining months of the training (ICR, p. 31). Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The M&E design of the project included lessons learned from the previous GPE-financed project. The MoE, through the Directorate of Planning and Education Innovation (DPEI), was responsible for collecting, consolidating, compiling, and publishing system and institutional data used to inform education policy and investments. Also, the DPEI was to provide consolidated monitoring data, including status reports on project implementation by component, with a detailed summary of project activities at the national, regional, and school levels. In collaboration with other relevant structures within the MoE, the DPEI developed the Page 11 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review STP- Quality Education for All Project (P146877) systems, tools, and practices needed for an effective assessment of students' achievements. An evaluation of the MoE and ISEC activities, achievements, and constraints was to be carried out semi-annually and inform the preparations for the following year. The General Inspectorate of Education was responsible for ensuring administrative and pedagogical monitoring of schools, principals, and teachers to identify training needs by using report cards and computer tools. A Teacher Training Institution Steering Committee was in charge of the evaluation of the training activities implemented by ISEC. The AFAP was to assist with M&E of the fiduciary implementation of the project. The ICR (p. 27) discussed the alignment of the PDO-level indicators with the objectives of the project. The three PDO-level indicators related to the first objective were considered to capture the improvements required to establish a robust in-service teacher training system, and as a consequence, improve education quality. The ICR questioned the selection of the two indicators chosen to evaluate the performance of the second objective. It pointed out that the total number of direct project beneficiaries should have been included as an intermediate rather than an outcome indicator. Also, it proposed the intermediate indicator “percentage of primary schools that benefit from four annual school-level supervision visits and mentoring program offered to teachers” as a better option than the actual PDO-indicator to measure improvements in the management of human resources. The reasons presented were that the increase in the number of daily working hours could be related to a simple change in policy, while the alternate proposed indicator would have captured actual supervision and mentoring. b. M&E Implementation The AF was added at the beginning of the project. For this reason, the PDO indicators were not changed, but with more funds available, the targets of some PDO-level indicators were revised to correspond with the scaled-up project scope. The changes were as follows: (i) training of 267 additional primary education teachers from 400 to 667 (equivalent to the total number of unqualified primary teachers in the system), (ii) the number of direct beneficiaries from 2,000 to 2,500, and (iii) the learning assessment to include an additional grade. The following intermediate outcome indicators were also scaled up: (i) the development and approval of a certification process was scaled up to include preschool teachers; and (ii) an assessment for 6th grade was included in addition to the 4th-grade student learning assessment already planned under the original project. Two new intermediate indicators were added: (i) training of 100 preschool teachers; and (ii) training of 100 teacher trainers. A detailed review of the entire results framework, conducted during the mid-term review, led to a project restructuring in 2017 that included the change of one PDO-level indicator and intermediate outcome indicators. Progress reports were submitted periodically during the project's life, except for the last year of implementation due to changes in the government after an electoral process. The EMIS, once implemented, allowed for collection of timely and reliable data. Project indicators and the expected outputs and outcomes were systematically monitored. To ensure adequate donor coordination, conduct Page 12 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review STP- Quality Education for All Project (P146877) additional education sector analysis, and monitor implementation of the government’s ESP, the Bank helped organize two Joint Sector Reviews of the implementation of the ESP, including all key education stakeholders in the country. The conclusion of the Reviews was that the ESP indicators should be more selective and more outcome-based, so that the information collected would be more meaningful. c. M&E Utilization The institutional data from the education sector collected by the DPEI was available regularly and used for the coordination with relevant structures at the MoE, ISEC, and AFAP. The findings from the 2018 Education Sector Analysis Report and National Large-Scale Assessment in Basic Education I and II were used to update the ESP (2019-2023). M&E Quality Rating Substantial 10. Other Issues a. Safeguards Even though the project was classified as Category C, and did not trigger any social and environmental safeguards policies, the ICR (p. 29) stated that factors related to gender and vulnerable groups were considered during consultations and project implementation. b. Fiduciary Compliance Financial management The AFAP, responsible for managing fiduciary aspects under the project, had previous experience with operations and procedures of Bank-financed projects. From the onset of the project, the AFAP was adequately staffed, the procedures manual was reviewed before effectiveness, and independent auditors were recruited. The ICR (p.30) described the financial accounting, auditing, and reporting of the project in terms of adequacy and consistency with the Bank's financial management guidelines. The ICR also stated that external auditors were in place, audits were completed on time, and interim financial reports were submitted regularly and in accordance with Bank procedures. Annual work plans and budgets were submitted for those areas that required additional attention based on annual audit reports. To ensure greater transparency in the use of funds, a Project Steering Committee reviewed and approved of all financial reports. Procurement Page 13 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review STP- Quality Education for All Project (P146877) The AFAP managed procurement processes under the project, which was overall compliant with Bank rules. Most of the contracts under the project were post reviewed, as the amounts were below the Bank’s thresholds (ICR, p 30). At the time of the mid-term review, the Bank’s new procurement tracking tool – Systematic Tracking of Exchanges in Procurement (STEP) -- was adopted to help overcome initial delays in implementation, better monitor project performance, and increase transparency. Overall, according to the ICR (p. 30), procurement functions demonstrated lapses and weak coordination between the government and AFAP. During the procurement review of the project for FY19, noncompliance with procurement rules and procedures was detected and led to a formal letter of reprimand to the Minister of Finance highlighting this noncompliance and recommending that AFAP be reminded of Bank procedures (ICR, p. 30). c. Unintended impacts (Positive or Negative) The project increased the use of technology for the improvement of education system quality. The MoE took advantage of the greater access to the internet provided by the Central Africa Backbone Project (P117652) to connect all primary schools with a reliable energy source. In addition, the online platform development and data collection developed for this project has supported online training across the country (ICR, pp. 22-23). d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory Satisfactory Moderately Bank Performance Moderately Satisfactory Satisfactory Quality of M&E Substantial Substantial Quality of ICR --- Substantial 12. Lessons The ICR (p. 33-35) offered several useful lessons, including the following adapted by IEG: In projects with several different implementing agencies, the precise definition of roles helps to reduce tensions. In this case, the Ministry of Education needed to rely on an outside agency to manage project funds because it did not have the necessary financial management systems. The AFAP’s role was clearly defined to only overseeing fiduciary aspects, while all technical leadership Page 14 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review STP- Quality Education for All Project (P146877) was the responsibility of the Ministry; however, some tensions arose during project implementation. To mitigate this situation, it was necessary to clearly define each agency's roles and competencies. Having dedicated personnel within a project's main ministry is critical for smooth implementation and coordination among all agencies and entities involved in a project. There was no one dedicated to work full-time on this project, and the resulting lack of coordination between different agencies was a significant barrier to implementation effectiveness. After the mid-term review, a Project Coordinator from within the Ministry of Education was appointed, but this staff member had many other responsibilities. Technology is a powerful tool to improve education quality, but risks related to limited capacity in information technology and weak infrastructure need to be accounted for in project design. This project included several initiatives that introduced information technology to make improvements in the education sector. The costs associated with purchasing the equipment to set up these programs and recurrent costs are not insignificant. The Ministry of Education needs more staff with information technology backgrounds to support the maintenance and implementation of these activities. Merit-based selection brings sustainability to a project. In the case, many of the positions across the entire education system are politically appointed, including Ministry of Education leaders, department heads, school directors, school supervisors, etc. This poses significant risks to sustainability when there is a turnover in government. After the elections in 2018, and during the final six months of implementation, nearly every person in these positions changed, which compromised investments made in capacity-building and institutional strengthening. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR provided a comprehensive narrative of the project’s experience. It was concise, results-oriented and candid. The evidence presented was well referenced, with annexes that included relevant information to support the description and analysis of the project's achievements. The ICR was internally consistent and followed guidelines. The theory of change was clear and well presented, and the annexes complete and exhaustive. While the ICR's lessons were insightful, they were not tightly linked to the description of project experiences and the narrative of the challenges that the project faced, and there was not clear differentiation between the lessons and the recommendations provided. In some areas, the ICR made definitive claims (such as the last sentence of paragraph 91) that were not supported by evidence. a. Quality of ICR Rating Substantial Page 15 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review STP- Quality Education for All Project (P146877) Page 16 of 16