Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG State Edu. Program Inv. Project (P122124) Report Number: ICRR0022524 1. Project Data Project ID Project Name P122124 NG State Edu. Program Inv. Project Country Practice Area(Lead) Nigeria Education L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-52200,IDA-58480 31-Oct-2017 230,176,709.09 Bank Approval Date Closing Date (Actual) 26-Mar-2013 30-Jun-2020 IBRD/IDA (USD) Grants (USD) Original Commitment 150,000,000.00 0.00 Revised Commitment 243,162,126.07 0.00 Actual 230,176,709.09 0.00 Prepared by Reviewed by ICR Review Coordinator Group Hjalte S. A. Sederlof Judyth L. Twigg Joy Maria Behrens IEGHC (Unit 2) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The Project Development Objective (PDO) for the Nigeria State Education Program Investment Project as set out in Schedule 1 of the Financing Agreement and page 11 of the Project Appraisal Document (PAD) was to support: (a) need-based teacher deployment; (b) school-level management and accountability; and (c) measurement of student learning in Participating States. The PDO was revised in June 2017 during Additional Financing, AF (see “Significant Changes” below). The revision emphasized that the reforms aimed at strengthening the education system, while at the same time Page 1 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG State Edu. Program Inv. Project (P122124) expanding geographic coverage from the original three states to include five additional states. The revised PDO read as follows: the PDO was to strengthen the educational system by supporting: (a) need-based teacher deployment; (b) school-level management and accountability; and (c) measurement of student learning in the Participating States. While the project was focused on strengthening the education system, notably by improving teaching and learning, it stopped short of setting targets for learning outcomes or measuring the impact of interventions on learning; rather, the project’s objectives were expressed in terms of intermediate development outcomes. For purposes of this review, the analysis will be based on those intermediate outcomes: (a) need-based teacher deployment; (b) school-level management and accountability; and (c) measurement of student learning. The formulation of the project objectives remained essentially the same for the AF as for the original project, albeit with expanded geographic coverage and an increase in commitment, so in this case the revision of the PDO does not warrant a split rating. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 24-May-2016 c. Will a split evaluation be undertaken? No d. Components The original project covered three States (Nigeria has 43 states) that had been selected based on strong government commitment at the sub-national level, strong fiduciary capability, complementarity with other donors (i.e. gap-filling while avoiding duplication), and the States' willingness to use results-based approaches (see Component 1). The project had two components: Component 1: Results-based Support to Education Sector Reform Program (estimated cost at appraisal US$125 million; actual cost US$104.50 million). The component was to apply performance-based approaches with strong in-built monitoring and evaluation (M&E) to improve learning outcomes in schools in participating States. It was to do so by emphasizing the use of better information flows to trigger accountability supported by decentralized school-level decision-making; and provision of teachers through assigned deployment or recruitment through incentives. For this, the component used a results-based Page 2 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG State Edu. Program Inv. Project (P122124) financing modality (RBF) with disbursement-linked indicators (DLIs) to finance participating States’ eligible expenditure programs focusing on the achievement of tangible and measurable results over the project period. DLI-linked areas of focus included: improving teacher effectiveness through better deployment of teachers based on need; regular measurement of student achievement; stronger school management and accountability through effective school-based management committees (SBMCs); and public-private partnerships to raise relevance of technical training. According to the PAD (p. 14), the choice of these areas of focus responded to key constraints in each of the States’ education systems, confirmed by a rigorous benchmarking system (SABER) developed by the Bank to strengthen performance in education. The DLIs were as follows: DLI 1 – improving teacher availability in rural areas to address the imbalance in the supply of teachers in rural and hard-to-staff schools through improved deployment policies and teacher incentives (mandatory postings; bonus programs). DLI 2 – improving teacher availability in core subjects through a better matching of teacher skills with student needs, including addressing the lack of teachers in core subjects – English, Mathematics, Science - - again through improved deployment policies and teacher incentives. DLI 3 – improving measurement of learning and achievement at primary and secondary levels, focusing on introducing a regular system for collecting standardized information on student learning outcomes and effectively reporting on results to support improved teaching and learning. DLI 4 – stimulating partnerships and program accreditation for relevance in technical and vocational schools to improve the quality of teaching, research, and skills development in senior secondary technical schools, and increasing their relevance for the job market. Measurable outputs were to include the number of partnerships established, improvements in course accreditation, and capacity building in technical and vocational training. DLI 5 – strengthening school-based management committees’ participation and capacity to influence schools’ effectiveness, management, and accountability through grants to primary and secondary schools based on their school improvement plans (SIPs). Component 2: Technical assistance (TA) (estimated cost at appraisal US$25 million; AF US$19.7 million; actual cost US$40.25 million). The component was to finance advisory, technical, and capacity- building support to: (i) the federal level for overall project coordination and to help ensure sustainability of successful activities; and (ii) the State level to achieve the DLIs and strengthen related capacity. TA under the component focused on the following four areas: educational development; developing budget frameworks; monitoring and evaluation; and strengthening fiduciary processes. Significant Changes During Implementation Several changes were made to the project during implementation, mainly reflecting the Boko Haram insurgency, but also to clarify project aims. None of the changes materially affected the educational Page 3 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG State Edu. Program Inv. Project (P122124) priorities (the theory of change) that were being pursued under the original project, i.e. supporting the priorities of participating States in improving educational access, quality, and efficiency. Additional Financing. AF totaling US$100 million was approved by the Executive Directors in June 2017 to support the government’s emergency program for the North East region that was being affected by the Boko Haram insurgency. Since the project already was operating in some North Eastern states, it was reasonable to scale up the project rather than introducing a separate project, especially in consideration of the relative urgency of stabilizing the education system in the threatened areas. The Bank was able to process the AF under OOP10.00 paragraph 12, referring to projects in situations of urgent need of assistance. Project coverage – originally consisting of three States with a population of 15.2 million – expanded to cover an additional six States and 23.4 million people. (Nigeria’s total population is 205 million people.) The PDO was marginally revised with the introduction of the AF (see Section 2a), and two outcome indicators were added to measure the impact of the project in the new areas. Components revised with the introduction of AF. Component 2, TA, was revised, and a new Component 3 was introduced: Revised component 2 (Technical Assistance): (original financing US$25 million; AF US$19.7 million; total planned US$44.7 million; actual cost US$44.7 million). The component was scaled up to accommodate the new States, and three TA sub-components were defined: Sub-component 2.1 (a): TA – improving education quality in the three original States, as well as strengthening their institutional capacity to implement DLIs; Sub-component 2.1 (b): TA to the North-East States (introduced under AF) for institutional capacity building in terms of teacher management and deployment, school-level management, and psycho-social and pedagogical training for teachers; Sub-component 2.2: TA to the Federal level for overall project coordination and for the establishment of an enabling environment in line with national policies. Component 3: Improving access to quality education and learning environment in the North East States (AF US$80.3 million; actual cost US$67.05 million). While similar to the original Component 1, the results-based approach was dropped from this component with the urgency of intervening in the North East due to the insurgency. Likewise, the “quality” focus was somewhat different from that envisaged under Component 1. The emphasis shifted to addressing damage from the insurgency, i.e. the physical rehabilitation of schools and improving access to the schools. The new Component 3 had two sub-components: Sub-component 3.1: Teacher incentive grants through salary top-ups after psycho-social and pedagogical training (to counteract the effects of the insurgency on teachers and school children). Page 4 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG State Edu. Program Inv. Project (P122124) Sub-component 3.2: School grants to school management committees that produced approved SIPs, including physical rehabilitation plans. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost. At appraisal, total project cost was estimated at US$150 million. Actual project cost at closing was US$211 million. The totals express spending on the original project and AF. The discrepancy between allocations reflects changes in the US$/Naira exchange rate and an unspent amount of US$6.8 million due to disturbances from the insurgency. Financing. The project was financed by a US$150 million IDA Credit and a US$100 million Additional IDA Credit. A total of US$169.70 million and US$67.05 million, respectively, were disbursed. Remaining funds were cancelled, as they could not be used in the insurgency-affected areas. Dates. The project was approved on March 26, 2013 and became effective on August 27, 2013. AF was approved on June 12, 2017, and the closing date was extended to October 31, 2019. Three Level II restructurings were noted. A first one in September 2018 involved the reallocation of funds from preparation of the new Component 3 to the project support unit for project coordination; a second one in October 2019 extended the closing date for the project to June 30, 2020 to allow insurgency-affected States to complete one more school year; and a third one allowed for cancellation of US$6.8 million due to limited access in one participating State. The project closed on June 30, 2020. 3. Relevance of Objectives Rationale The project (in its original and revised version) addressed key challenges in the education system, notably the persistence of poor learning outcomes driven by limited availability of qualified teachers, affecting access and quality of learning especially in rural areas and in core subjects, and weak governance and management capacity. In doing so, it continued the Bank's engagement in the education sector, this time focusing on the North East where the Boko Haram insurgency makes the provision of quality basic education particularly challenging. The project drew on that longer-term reform effort and aligned with findings of the Bank’s 2020 Systematic Country Diagnosis (SCD) and the current Country Partnership Framework (CPF) for FY21-25, which draws on the SCD. The CPF explicitly underlines the importance of supporting increased access to quality basic education; empowering girls and women; and promoting youth employment and skills. The project supported this by providing better learning tools (well- trained teachers and better governance), but it stops short of examining the implications of these interventions on beneficiaries, given the challenging environment (the insurgency). The project also served as a building block for the Bank’s ongoing Partnership for Education project, notably with regard to strengthening governance and public sector management, and lessons learned from coordinating federal- and State-level actions and the use of DLIs are being drawn on for other Bank projects to come on stream. The AF fit into the above framework. It drew on the teacher deployment function in the original project to deploy teachers to insurgency-affected areas, and it formed part of a coordinated multi-sector package of support to the North East and the World Bank’s North East Emergency Transition and Stabilization Program Page 5 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG State Edu. Program Inv. Project (P122124) in support of the government’s efforts to restore stability and create economic opportunities for the most vulnerable. Other projects include five human development operations and an agriculture operation. Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Need-based teacher deployment Rationale Needs-based teacher deployment was to be achieved by improving the availability of qualified teachers in conflict and displacement-affected project states, where in many areas teachers as well as students risked being subject to violence (DLI 1 and DLI 2). This involved the mandatory deployment of teachers, as well as the application of incentives, including bonus programs, and conditioning incentives on pedagogical training and training in psycho-social skills to better cope in their postings. Outputs/Outcomes Success under the original project was measured in terms of the percentage of planned teachers who were actually deployed. This was a straightforward and practical indicator that was measurable, monitorable, and understandable, and it was pertinent to the stated objective. Teachers deployed by core subjects and to rural areas exceeded planned levels (as determined in the education management information system (MIS) and by local education authorities) in two of the three original States. The third, insurgency-ridden Bauchi State, fell short on deployment to rural areas. In other insurgency-affected areas introduced under the AF, teacher deployment targets were achieved or exceeded in four States and not achieved in one case, Adamawa (in addition to the original Bauchi). No baselines (pools of potential teachers) had been determined in AF States, and no distinction was made between need on core subjects and need in rural areas. The focus was on getting trained teachers into the field and appropriately equipped with pedagogical and socio-psychological skills. In general terms, from a deployment angle, project interventions appear to have been relatively successful overall, with teacher deployment exceeding targets in both original and AF States, and while it is not obvious by which means targets were achieved – through deployment or incentives, or both – in core subjects, total deployment reached 90 percent, surpassing a target of 80 percent of pre-project levels; and in rural areas total deployment reached 108 percent, surpassing a target of 81 percent of pre-project levels. Teachers deployed by core subjects:  Anambra, original target 50%, revised target 90%, achievement 117% Page 6 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG State Edu. Program Inv. Project (P122124)  Bauchi, original/revised target 70%, achievement 81.4%  Ekiti, original target 86%, revised target 90%, achievement 104% Teachers deployed to rural areas:  Anambra, original target 60%, revised target 90%, achievement 167%  Bauchi, original target 60%, revised target 70%, achievement 60%  Ekiti, original target 70%, revised target 95%, achievement 136% Teachers deployed in insurgency-affected areas (under the AF):  Adamawa: target 9,450, achievement 9,050  Bauchi: target 9,000, achievement 7,724  Borno: target 16,070, achievement 25,000  Gombe: target 4,500, achievement 9,739  Taraba: target 5,400, achievement 5,400  Yobe: target 2,370, achievement 5,090 The ICR summarized a set of plausible factors that may have influenced teacher availability: government engagement in performance management, extra teacher training, and better equipped schools (the latter is also believed to have promoted student retention and re-engagement, lowering absenteeism). Although beyond the scope of the PDO, the ICR (p. 16) speculated about higher-level improvements in learning outcomes that may have resulted from the needs-based teacher deployment specified in the objective. It made the reasonable argument that greater access to skilled teachers and to ones specialized in core subjects may have improved the learning experience of children and raised morale among teachers; it is not, however, obvious how actual learning progressed. Financial incentives (top-ups to salaries) and improvements in school system management, introduced under PDO 2, are likely to have had a positive impact on the teaching and learning environment, in attracting teachers and maintaining teachers and students in school. Achievement of this objective is rated substantial. Rating Substantial OBJECTIVE 2 Objective Improved school-level management and accountability Rationale Page 7 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG State Edu. Program Inv. Project (P122124) The objective was to be achieved by: (i) improving the quality of secondary technical schools by raising the relevance of their programs through public-private partnerships and program accreditation (DLI 4); and (ii) strengthening school-level management at primary and secondary levels (DLI 5). Such improvements were expected to result in an increase in the number of schools meeting agreed service standards, as well as an increase in the number of enrolled students. In AF, the bulk of funds for school-level management and accountability were focused on physical rehabilitation of schools that had been damaged as a result of the conflict. Outputs/Outcomes Key indicators included the total number of beneficiaries (annual estimates for students and teachers in the catchment area), enrollments in technical schools, schools meeting service standards, and schools receiving grants against improvement plans proposed by local school boards. All were relevant in measuring results from the project interventions.  The overall number of direct project beneficiaries, 2.4 million, fell almost 30 percent short of the target of 3.3 million; only two States met or exceeded the target.  The target for students enrolled in technical and vocational schools was exceeded overall but was not achieved in two individual cases (Ekiti, Anambra). Of students enrolled in technical and vocational schools, the number of girls increased by 27 percent from the baseline, while the number of boys remained flat. The share of girls in the system increased from 24 percent at the baseline to 30 percent.  The target for schools receiving grants against improved plans (concerning only AF) was exceeded by some 75 percent.  None of the three original States achieved the target for schools meeting agreed service standards. According to the ICR, measurement of this indicator was modelled on a survey that had been completed but was not being repeated in Nigeria, and its monitoring by the project was stopped. Results partly reflect the likely influence of more effective SBMCs and a better educational environment resulting from training provided to SBMCs under the project. Additional information provided by the task team in the Region comments to the ICR Review draft indicates that such training has resulted in a gradual increase in the quality of SIPs and in their implementation. Surveys indicate a steady rise in SBMCs that produce quality SIPs that in turn plausibly influence service standards and enrollments. While the measurable outcome indicators in most cases showed targets as being only partly achieved, the strengthened SBMCs are likely to constitute a platform of school-level management and accountability that can serve as a springboard towards better school performance in the future. On that basis, the project plausibly contributed substantially toward achievement of this objective. Rating Substantial Page 8 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG State Edu. Program Inv. Project (P122124) OBJECTIVE 3 Objective Measurement of student learning Rationale The objective was to be achieved by introducing standardized, state-level testing (DLI 3) to collect standardized information on student learning outcomes, helping to determine what, if any, efforts would be needed to improve student performance. Outputs/Outcomes The PDO-level indicator – availability of a system for learning assessment at the State level – was achieved in all three States (in Bauchi English and Mathematics for Primary 5, and in the other two in English and Mathematics for Senior Secondary 2). The ICR (p. 19) rated achievement of this objective substantial instead of high, pointing to the use of an un- nuanced “yes/no” indicator, rather than a more complex one that might offer further guidance. This ICRR review agrees with this assessment, noting that a more in-depth assessment might have been more in line with the spirit of the project’s intentions of providing insights into the teaching, learning and school management process. Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale The project included three objectives - all three were rated substantial. The first objective included PDO indicators that all were valid ones measuring changes in need-based teacher deployment. The second objective in large part reflected the results of improved SBMCs, albeit that quantitative results in some instances fell short of targets. The single target for the third objective was met, although a more in- depth analysis would have been useful. Overall Efficacy Rating Substantial 5. Efficiency Page 9 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG State Edu. Program Inv. Project (P122124) The ICR undertook a benefit-cost analysis (BCA), estimating benefits generated by each component of the project on an incremental basis, and the project's economic feasibility was determined based on calculation of the net present value (NPV) as well as the internal rate of return (IRR). As no BCA was conducted at appraisal, the ICR applied assumptions used for similar projects in Nigeria as comparators. Aggregating benefits across project beneficiaries and considering direct project costs and project management costs, the NPV of the project was estimated at US$ 12,557,702, and the IRR was 19.6 per cent, comparable to results for such analyses in other similar countries. Assumptions and methodology for the calculations appear reasonable. Factors that favorably influenced implementation efficiency: (i) the government’s strong commitment; (ii) the use by the project of local structures, including by drawing on a significant TA component; (iii) the identification of key risks and mitigation measures; and (iv) a consultative approach to both clients and donors. Delays in implementation were driven by the need to constantly reassess project activities in context of the Boko Haram insurgency; a loss of momentum in setting up DLI-related verification for disbursements and in converting project accounts to a Treasury Single Account (see Section 14 on Financial Management); and general elections. A change in project scope – the introduction of AF – also contributed to slowing down the project as originally envisaged. Altogether, project closing was extended by 32 months. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of Objectives is rated substantial: they were relevant to country conditions and to government and Bank strategy, noting however, that objectives were pitched at a relatively modest level, in part because of the restlessness in the project States. Achievement of the three objectives was rated substantial, with most targets being achieved or exceeded. Efficiency was rated substantial. On this basis, the project is considered to have had minor shortcomings, and outcome is rated satisfactory. a. Outcome Rating Satisfactory Page 10 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG State Edu. Program Inv. Project (P122124) 7. Risk to Development Outcome The federal government as well as participating State governments appear committed to sustaining the institutions that were put into place under the project, especially the school-based management mechanisms, including by institutionalizing their school-based funding. Still, building sustainable levels of teaching quality can take time and will require continuing efforts. Here, the Bank’s continued presence in the education sector should alleviate concerns. The country program points to continued education sector engagement; already, two projects are underway – AGILE Project (P170664) and Edo Basic Education Sector and Skills Transformation Operation (P169921); and the ICR (p. 34) drew attention to the recent CPF and its support for further increasing access to and quality of basic education. 8. Assessment of Bank Performance a. Quality-at-Entry The project was strategically relevant, being linked to higher-order objectives of achieving better learning outcomes through improvements in such aspects as access, quality, and efficiency that were basic tenets of Nigerian development planning in education, and consistent with the Bank’s CPS that also flagged these three aspects. At the same time, project design was mindful of the uncertainties surrounding life in North East Nigeria, and project objectives were modestly set, but still with an aim to move the education agenda forward. That involved focusing on teacher availability, pedagogy and socio-psychological training for teachers, and governance, which together provided a platform for better learning but did not set targets for learning outcomes. The project concentrated on States in the North East, giving it a strong poverty focus, and with the subsequent introduction of the AF it also came to support areas that were falling behind as a result of the Boko Haram insurgency. In pursuing equity, gender was introduced, as the project sought to correct gender imbalances among teachers and inclusion of women in school committees. (However, among indicators only one – students enrolled in technical and vocational schools – made a gender distinction.) The results-based design of the project gave it a coherent results framework, focusing on institutional changes as sources of sought-after quality improvement. At the same time, project design complemented other more traditional community-level Bank-financed projects supporting schools and health facilities (P090644, P126964, and a State Health Investment Project, P120798). The pre-project risk assessment focused on essential issues in the Nigerian environment, and mitigating measures proved adequate (within the framework that the Bank team could influence). M&E indicators were robust, albeit output-oriented. Design was underpinned by extensive data analysis (including through the Bank’s SABER initiative for examining education systems) and lessons learned from earlier engagements in Nigeria and further afield. In addition, the project drew on participating States’ own education sector plans in a consultative and participatory approach. Project preparation was designed to complement the work of other donors, adding a more systemic approach to their more child- and teacher-centered interventions. Quality-at-Entry Rating Satisfactory Page 11 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG State Edu. Program Inv. Project (P122124) b. Quality of supervision Project supervision was facilitated by the government’s commitment to the project and by the choice to use existing institutional structures in implementation, which strengthened client ownership. This may have been especially useful when facing the challenges of Boko Haram and in furthering implementation, as the DLI procedures (and especially the hiring of a firm for verification of DLIs) took longer than initially expected. Likewise, fiduciary challenges would delay the AF. On these, and other lesser implementation issues, the project was able to benefit from consistent Bank team support and active follow-up on issues in terms of technical support and capacity building. Despite changes in TTLs and the project team over the life of the project, consistency in approach appears to have been maintained and adjustments to the project from security concerns and field-level operational challenges handled. Supervision inputs appear to have been adequate, and performance reporting was straightforward. Technical support from the Bank was an important element for project implementation, and here the project team was effective (ICR, p. 27). Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The project’s M&E framework was based on the government’s established education monitoring system, including some adjustments to allow selected outputs to be measured during implementation. Fully engaging local government authorities was a strong point of the design, which tended to reinforce lessons, even under the uncertainties created by the Boko Haram insurgency. Objectives were clearly set out, and indicators focused on institutional changes (for instance public-private partnerships, service standards, learning assessments) as sources of increased access and improved quality of the included school systems. Indicators were measurable and included baseline and quantitative targets where relevant. While output-oriented ones, they allowed monitoring of the kind of institutional reforms being sought. Achievement of the third objective could have been monitored by more complex, nuanced indicators in additional to the simple yes/no indicator that was adopted. b. M&E Implementation M&E was the responsibility of the State Ministries of Education, and data collection was a routine function of their operations. This ensured that data collection would be (and was) carried out, and indicators included in the results framework were measured and reported. Use of established systems strengthened the likelihood that M&E functions and processes, including adjustments, were to continue after project Page 12 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG State Edu. Program Inv. Project (P122124) closing. Indicators would prove to be robust, although as noted above, their emphasis was more on measuring outputs than outcomes. Still, implementation of M&E suffered from some delays in third party verification of DLIs, in part due to the importance given to the function, and in part due to the bureaucratic process required in establishing a Treasury Account for channeling project funds, resulting in additional consultant support being drawn into the process. In part, national elections also interfered. For the AF, separate consultants were recruited to verify progress. c. M&E Utilization The M&E system was utilized by the project team, the federal government, and education partners at the sub-national level and in the donor community to inform themselves about the project and how it was contributing to the overall education sector plan. Most indicators were consistently tracked, and supervision missions and stakeholder meetings received timely updates on how the project was achieving its objectives. This permitted course corrections to be made which, according to the ICR, were minor, albeit contributing to more reliable data. The project also contributed to analytical work, including student learning assessments, as well as several studies of the project’s impact on targeted beneficiaries. To the extent that such studies were completed during the project, or available to the ICR team, the information might have supported a more robust consideration of project outcomes than that presented in the ICR. M&E Quality Rating Substantial 10. Other Issues a. Safeguards The project was classified as Environmental Category B and triggered the Environmental Assessment OP/BP 4.01 safeguard policy. Activities that triggered the policy were related to Component 1, in which grants were to be provided to education services, including for school maintenance and repairs. In response, the project developed an Environmental and Social Management Framework (ESMF). These safeguards classifications were retained over the life of the project. To ensure safeguard compliance in line with the project’s ESMF, the project required that site-specific Environmental and Social Management Plans (ESMPs) be prepared and disclosed in-country and subsequently at the World Bank InfoShop before the commencement of any minor rehabilitation works. This was completed as planned. Further, the task team recommended that each State prepare a monitoring plan on compliance with environmental and social safeguards guidelines in the ESMF. Safeguards compliance was rated as Satisfactory throughout the entirety of the project’s lifespan. The project financed some minor rehabilitation and maintenance of schools in participating States. All physical works were completed by the project’s closing date of June 30, 2020. The safeguards rating at project closing thus stands as Satisfactory. Page 13 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG State Edu. Program Inv. Project (P122124) b. Fiduciary Compliance Financial Management (FM). Budgeted expenditure programs and actual expenditure were captured in a credible and timely manner including through tracking and recording. The aggregated disbursements and Statements of Expenditures were promptly submitted to the Bank for review and reimbursement. Account records were adequate, with timely preparation of accurate financial reports. Overall, funds flows, reporting, and accounting system(s) were deemed sound, and there were no major audit observations. Nevertheless, some challenges regarding FM were encountered, specifically, delayed submission of the audit report for 2016 and the delayed account opening and activation under the AF; other minor challenges were related to some delays in addressing unretired balances and audited reports submission. Recognizing the need for further capacity building, in July 2016, core personnel of State units participated in a refresher training in procurement, FM, and disbursements. Other fiduciary clinics were also provided throughout project implementation. Procurement. Procurement was rated satisfactory throughout the life of the project. Procurement activities focused on the provision of TA according to World Bank procurement guidelines. To accelerate implementation, client teams had technical discussions with the Bank team, focusing on implementation start-up activities. Identified procurement staff also benefited from other appropriate training and capacity building programs. For instance, the World Bank Fiduciary team provided fiduciary clinics, and other presentations on procurement, financial management, and safeguards were given by World Bank specialists throughout project implementation. Generally, there were no major issues on procurement. Internal controls, record keeping, and implementation of the project were in accordance with the PAD. There were some delays related to the selection process and delayed deployment of a newly selected independent verification firm to undertake DLI validation. This was also negatively affected by the introduction of a Treasury Single Account, as the project's account at the level of the Federal Ministry of Education took a while to get activated by the Central Bank. Nevertheless, necessary action was taken to address issues as they came up, and as a result, at the close of the project, procurement was rated Satisfactory. c. Unintended impacts (Positive or Negative) None reported. d. Other Gender. The project focused on promoting equity, including by setting goals to close the gender gap through correcting gender imbalances among teachers; identifying out-of-school children, including gender considerations, and considering strategies to get girls back into school. It sought to ensure that women were represented in school management committees and thereby have opportunities to engage in decision making at the community level. At the time of the project, Baluchi State was implementing a policy of improving gender parity imbalances by 2018 through a set of measures aimed at: (i) increasing the number Page 14 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG State Edu. Program Inv. Project (P122124) of female graduates at its colleges of education from 632 in 2012 to an expected 720 in 2013 and 900 by 2016; (ii) providing more space for girls in secondary schools; (iii) replacing all retiring/exiting teachers by female teachers; and (iv) providing incentives to attract and retain girls in schools, as well as providing conditional cash transfers focusing on female students. In response, the project provided TA to support Bauchi State's plan to attract, recruit, and retain female teachers to improve the gender balance among the teacher force. 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory Bank Performance Satisfactory Satisfactory Quality of M&E Substantial Substantial Quality of ICR --- Substantial 12. Lessons IEG draws several lessons from the ICR: Output-type objectives may be stronger when direct links are established to higher level outcomes. A major objective for the Nigerian education system is providing access to quality basic education. The project supported this by providing better learning tools: teachers and the school environment. However, it stopped short of examining the implications of project interventions on the teaching-learning process. Building formal relationships between the school system and the community can revive and mobilize both in the face of adversity. The project advocated for community (and increased female) participation in school committees, and this active engagement on a broad front was intended to re-engage teachers and students in the schools, increase a sense of security in exposed communities, and lead to better monitoring of the school community. 13. Assessment Recommended? No 14. Comments on Quality of ICR Page 15 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG State Edu. Program Inv. Project (P122124) The ICR is extensive at 36 pages (83 pages with Annexes) and partly because of that, and also because it is well-written, it mostly offers a robust evidence base to support its ratings. That said, the discussion of efficacy and in particular the results orientation could have been stronger, and here shortcomings - concerning both objectives and indicators - are to be found on the side of the project team. Other parts are more satisfying – in particular those on Efficiency and M&E, as well as the overall impressions one can draw from the text on Relevance and Bank performance. a. Quality of ICR Rating Substantial Page 16 of 16