Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG:Youth Employment & Soc Support Operat (P126964) Report Number: ICRR0022652 1. Project Data Project ID Project Name P126964 NG:Youth Employment & Soc Support Operat Country Practice Area(Lead) Nigeria Social Protection & Jobs L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-52120,IDA-58390 30-Jun-2020 170,307,082.61 Bank Approval Date Closing Date (Actual) 26-Mar-2013 30-Sep-2020 IBRD/IDA (USD) Grants (USD) Original Commitment 300,000,000.00 0.00 Revised Commitment 199,976,849.00 0.00 Actual 170,307,082.61 0.00 Prepared by Reviewed by ICR Review Coordinator Group Hjalte S. A. Sederlof Salim J. Habayeb Eduardo Fernandez IEGHC (Unit 2) Maldonado 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The Project Development Objective (PDO) for the Youth Employment and Social Support Operation (YESSO) as set out in Schedule 1 of the Financing Agreement was to increase access of the poor to youth employment opportunities, social services, and strengthened safety net systems in Participating States. The PAD (page 8) had the same formulation for the objective. The PDO was revised with the introduction of Additional Financing (AF) in 2016 to read as follows: to increase access of the poor and vulnerable, using improved social safety net systems, to youth employment Page 1 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG:Youth Employment & Soc Support Operat (P126964) opportunities in all Participating States and to provide Targeted Cash Transfers to the poor, vulnerable and internally displaced people (IDPs) in the North East States. The third objective in the revised PDO – provision of Targeted Cash Transfers - replaced the objective “social services” of the original PDO, which was withdrawn from the project without being started (see below also Section 2d – Significant Changes During Implementation). Achievement of the objectives will be assessed separately for (i) strengthened/improved social safety net systems; (ii) increased access of the poor and vulnerable to youth employment opportunities in all Participating States; (iii) provision of Targeted Cash Transfers to the poor, vulnerable and IDPs in the North East States. The objective “strengthened/improved safety net systems”, underpins the other objectives in each of the two PDOs, i.e., the selection of beneficiaries of objectives (ii) and (iii) was to be determined by a common targeting system and a unified registry of potential beneficiaries that was to serve as a platform for targeting safety net programs. This ICR Review will not undertake a split evaluation, as the changes to the project (described in Section 2d) predominantly addressed changes in the scope of the project. The realignment of cash transfers to target IDPs rather than potential beneficiaries in all participating states could warrant a split evaluation. However, the original cash transfer component was withdrawn without being implemented under the project to be included in another Bank-financed project (National Social Safety Net Project, P151488) and replaced by the Targeted Cash Transfer program for IDPs, later expanded to include the elderly and physically challenged in the North East states. Moreover, while outcome targets were either reduced or increased, most original and revised targets were achieved or exceeded, and a split evaluation would not have added much value. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 07-Jun-2016 c. Will a split evaluation be undertaken? No d. Components Component 1: strengthening social safety net systems (estimated cost at appraisal US$20.0 million; actual cost US$45.70 million). The component was to define institutional responsibilities and Page 2 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG:Youth Employment & Soc Support Operat (P126964) implementation arrangements for effective coordination between federal and state governments and other stakeholders involved in implementing safety nets in Nigeria: it was to strengthen federal capacity for coordination, monitoring, policy development and developing co-financing mechanisms for safety net programs; and strengthen capacity at the state level to implement a safety net system and scale up key safety net interventions. Component 2: public workfare program (estimated cost at appraisal US$200 million; actual cost US$66.66 million). The component was to support participating state governments to provide labor intensive work opportunities for unskilled youths between the ages of 18 and 35 from poor households. Component 3: skills for jobs programs (estimated cost at appraisal US$30 million; actual cost US$27.04 million). The component was to pilot a new approach to skills training to better combat unemployment among youth. The approach emphasizes demand-driven skills training that can facilitate entry into local labor markets through trainee arrangements, including institutional capacity building and public-private sector partnerships to enhance employability (such approaches have produced positive results in other countries). Component 4: conditional cash transfer (CCT) program (estimated cost at appraisal US$50 million; actual cost US$31.13 million). The component was to assist participating state governments in designing and operating the CCT program, including providing cash to parents investing in their children’s health and education. SIGNIFICANT CHANGES DURING IMPLEMENTATION The project received one Level 1 Additional Financing (AF) credit and was restructured four times during implementation (three were Level 2 restructurings). Additional Financing amounting to US$100 million was approved by the Executive Directors in June 2016. The credit was introduced as a crisis recovery response to the ongoing Boko Haram insurgency in the country’s six North East states. At the same time, the project went through a first restructuring replacing the provision of CCTs to eligible beneficiaries in participating states by targeted cash transfers (TCTs) to support return and resettlement of Internally Displaced Persons (IDPs) in the North East States that were included in the unified registry. The PDO was reformulated to reflect this change, and targets were adjusted accordingly. Page 3 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG:Youth Employment & Soc Support Operat (P126964) In addition to the restructuring under Additional Financing, the project was restructured three more times (Level 2 restructurings): A second restructuring in 2018 was to address the slow pace of implementation – only some 13 percent of project proceeds had been disbursed as of end-April 2018. Delays had initially been driven by slow implementation of the targeting system (the registry), but subsequently the pace had been slowed by capacity constraints in the project coordinating unit and frequent changes of project counterparts, difficulties with return and resettlement of IDPs due to the ongoing conflict, and often insufficient counterpart funding. The restructuring raised disbursement amounts to IDPs, and simplified the design of the public works and skills components. A third restructuring in 2019 expanded the TCT component to include Special Cash Transfers (SCTs) to eligible elderly and disabled beneficiaries in the North East. Targets for some outcome and output indicators were again adjusted to better reflect implementation progress. Following revisions of cost estimates, US$200 million were cancelled and redeployed to other projects. A fourth restructuring in 2020 extended the closing date. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost. At appraisal, total project cost was estimated at US$500 million. Actual project cost at closing was US$170.31 million. Financing. The project was financed by a US$400 million IDA credit and a government contribution of US$100 million. US$200 million of the IDA credit was cancelled during the third restructuring in 2019. Actual project costs were fully met by the IDA credit. Dates. The project was approved on March 26, 2013, with an original closing date of June 30, 2020. The closing date was extended once, with the project closing on September 30, 2020. 3. Relevance of Objectives Rationale The original and revised PDOs were aligned with country priorities, government policy on youth employment and social support services, and the Bank’s Country Partnership Framework for FY21- FY25. At the time of appraisal, Nigeria was characterized by widespread poverty and low human development indicators, as well as security threats in the northern parts of the country because of the Boko Haram insurgency. Together with poverty and vulnerability, and security risks, the country was Page 4 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG:Youth Employment & Soc Support Operat (P126964) experiencing a demographic youth bulge and the perspective of continued high, and possibly rising, levels of youth unemployment. The social sector contained an array of safety net and labor programs to address shocks and risks faced by the poor and vulnerable, but these were often poorly targeted and their implementation was weak, suffering from low capacity and uncertain financing at both federal and state levels. Government policy, through its Economic Recovery and Growth Plan and its successor, the Economic Sustainability Plan, includes scaling up social assistance, creating jobs and stimulating micro and small enterprise activity, and promoting investments in infrastructure. The Bank is well-placed to assist the government in developing an effective safety net system: the Bank’s social protection team has been engaged in the sector since 2004, mainly in providing information and technical assistance. This is reflected in country strategies. The Bank’s Country Partnership Strategy (CPS) FY10-13 was aligned with government priorities in social protection at project preparation and implementation, and it continues to be so in the FY21-FY25 Country Partnership Framework (CPF). Both documents signal the Bank’s continued commitment to supporting the government in increasing access to and use of human development services, make them more efficient and equitable, and to improve their quality. This orientation is also in line with the Bank’s Africa Regional Strategy. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Strengthening the safety net system in Participating States Rationale The objective was to be achieved by implementing a community-based targeting system (a unified registry of poor and vulnerable households) in Participating States. The registry was to serve as a platform for targeting potential beneficiaries for safety net programs. Under the project it served as the platform for identifying participants in project-supported youth employment, skills development and cash transfer programs in Participating States. Outputs/Outcomes The project introduced unified registries for potential participants in public works schemes, skills development programs and cash benefit transfers in 15 states comprising 1.2 million households compared to an original Page 5 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG:Youth Employment & Soc Support Operat (P126964) target of 12 states. (This includes 360,000 households from the North East states). In addition, by project closing, the registry was being utilized for social and health interventions in Nigeria by over 40 national and international agencies, including in the Bank’s National Social Safety Nets project (P151488). An evaluation in 2020 indicated that 97 percent of beneficiaries under this project were satisfied with the targeting process; the ICR (page 17) indicates that mistargeting was very low. The targeting mechanism also demonstrated its effectiveness in identifying COVID-affected needy poor and vulnerable families. Based on this performance, the government (i) established an institutional framework for expanding the registry beyond the project states, and (ii) the registry was being drawn on as a coordinating platform for developing a national social protection policy framework (ICR, paras. 33 to 35). Rating High OBJECTIVE 2 Objective Increased access of the poor and vulnerable to youth employment opportunities in the Participating States Rationale The objective was to be achieved by providing labor intensive work opportunities for unskilled youths from poor households, as well as skills for jobs training in life, vocational and entrepreneurial skills. Work opportunities were to be accompanied by orientation sessions at the beginning of each public works job cycle and livelihood skills training during the course of the scheme to ensure that participants understood what was expected of them and to prepare them for life in the labor market. Skills for jobs training was to focus on internship programs. Outputs/Outcomes Public workfare was provided to 242,000 youths (59 percent female) in 16 states, against an original target of 250,000, a subsequent target of 300,000 introduced with AF, 200,000 introduced under the first restructuring, and 100,000 under the second one. The declining targets reflected implementation bottlenecks at the State government level, both in terms of insufficient funds and inadequate staff capacity (Restructuring Papers, 2018, 2019). Measures introduced during the restructurings restored momentum to the public works schemes, almost achieving the original target but short of the 300,000 that had been envisaged because of AF. Page 6 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG:Youth Employment & Soc Support Operat (P126964) Of the 242,000 participants in public workfare schemes, 183,000 received orientation and training in life skills, exceeding the original target of 100,000, but falling short of the target of 205,000 set during the 2018 restructuring. Beneficiaries reported increased consumption, assets, financial inclusion and increased self-esteem as a result of participating in the public works schemes. Spending on food increased by an average of 49 percent, asset ownership saw an average 66 percent increase in the value of assets held, while ownership of bank accounts increased from 22 percent to 74 percent. Over 95 percent of community members in benefiting communities saw the project having a positive effect on their communities. In the skills for work program, 28,185 trainees (41 percent female) graduated from the internship programs compared to a target of 40,000 – 72 percent into wage-paying jobs or as self-employed, compared to 21 percent before the intervention; 70 percent reported an increase in income. The discrepancy between actuals and targets does not necessarily measure the full impact of the program, as such programs often have a stronger effect over the medium to long term than the one-year perspective that the numbers reflect. Tracer studies post-internships indicated that almost all beneficiaries appreciated the quality of the training. Rating Substantial OBJECTIVE 3 Objective Provision of targeted cash transfers to the poor, vulnerable and IDPs in the North East States Rationale The objective was to be achieved by the implementation of the TCT program in the North East, introduced under AF, and through special grants to vulnerable individuals over 65 or with disabilities, introduced with the 2019 restructuring. Outputs/Outcomes Targeted Cash Transfers were provided to 138,900 IDP households in the North East, compared to an original target of 50,000 households, set in the AF. Special Grants were provided to 99,000 beneficiaries compared to a target of 25,000 set at the 2019 restructuring. As a result of the Targeted Cash Transfers, the value of food consumption among beneficiaries increased by 39 percent, and the value of assets increased by 69 percent; ownership of bank accounts increased by 23 percentage points. Special Grant recipients saw increases in consumption by 57 percent, and the value of assets by an average of 36 percent. Page 7 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG:Youth Employment & Soc Support Operat (P126964) The number of IDPs resettled in safe communities in the North East numbered a low 220 compared to a target of 250,000 set in the AF. It is likely that this modest number reflects continued restlessness in the North East that hindered resettlement. Over 90 percent of beneficiaries of these programs considered that they promoted improved access food, health care, education, shelter and participation in community activities. Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale The three objectives were rated either fully achieved (Objective 1) or almost fully achieved (Objectives 2 and 3), and their outcomes can be predominantly attributed to project activities. Overall Efficacy Rating Substantial 5. Efficiency Neither the PAD nor the ICR included a benefit-cost analysis of the project. Instead, the efficiency analysis was qualitative. By investing in a platform that allows targeted safety net programs to be coordinated (the registry), and later supporting the financing of the targeted programs (cash transfers, public works, skills), the project introduced a widely recognized effective way of addressing poverty. For instance, the ICR, in assessing the efficiency of wage transfers to unemployed youth, estimates a benefit of US$1.00 at a cost of US$0.24. Implementation efficiency had both favorable and unfavorable aspects: delays in implementation were significant (as noted in Section 2b) and did lead to subsequent cancellation of US$200 million. The start of implementation of public workfare and skills for jobs programs was delayed by about four years (ICR, p. 25). Other factors that negatively impacted implementation included non-release of government contributions, varying state government commitments, inadequate staff capacity in terms of both quantitative and qualitative aspects, complexity of project design, and a difficult political and security environment (ICR, pp. 30- 31). Conditioning the payment of grant transfers on IDPs relocation resulted in the payment of only one tranche to nearly all IDP beneficiaries (ICR, p. 35). On the other hand, efficiency was enhanced by continuity in terms of Page 8 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG:Youth Employment & Soc Support Operat (P126964) Bank staff working on the project; and by maintaining management costs at reasonable levels (around 15 percent) despite the uncertain operational environment in the northern parts of the country. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance was rated high based on its alignment with the country situation, government policy and the Bank policy framework for Nigeria. Efficacy was rated substantial – the project put into place an operational and effective targeting mechanism under a common targeting system. Efficiency was rated modest, reflecting significant shortcomings in the efficiency of implementation. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome The ICR assesses the risk to development outcome as substantial. Still, the institutional setup developed under the project is likely to be sustained. The registries introduced under the project have been integrated into the Nigerian social safety net coordinating office and have been extended to additional states - a transition that is being undertaken with Bank support. Youth employment and social protection are considered priorities by the government and works undertaken under the project are likely to be maintained considering the high degree of ownership of the results that have been noted in the various evaluations. On the uncertain side is the availability of continued government co-financing for the safety net programs: this is of some concern and may be even closely tied to oil revenues and fluctuations in oil prices. Page 9 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG:Youth Employment & Soc Support Operat (P126964) 8. Assessment of Bank Performance a. Quality-at-Entry The project was strategically relevant with its focus on government priorities in building safety nets: cash transfers and employment creation through public works schemes and skills development, mainly aimed at youth (young men and women). This approach also aligned with the Bank’s country strategy for Nigeria. Its design drew on lessons from other similar Bank-supported initiatives in Nigeria and elsewhere, and the results framework was straightforward, as was monitoring and evaluation. While safeguard policies were triggered, safeguard instruments were appropriate. Regarding fiduciary compliance, procurement procedures were initially slow but became satisfactory over the course of project implementation. The risk assessment did recognize institutional constraints at the federal and state levels, as well as political risks. To address them, project design included capacity building that focused on both participating federal and local authorities, as well as the project team’s efforts to collaborate with stakeholders at both levels during preparation (and implementation). Nevertheless, risks still may not have been sufficiently anticipated, contributing to a slower-than-anticipated start-up with only 13 percent of project funds disbursed after five years of implementation. This is particularly surprising, considering that the Bank had a lengthy experience supporting social protection programs in Nigeria dating back to 2004. However, other factors also would influence implementation of the operation: its design as a support operation seeking common ground with ongoing state-level programs would prove challenging; as would the conditioning of grants on IDPs relocating. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision The project benefited from proactive supervision and a TTL who remained with the project throughout the project cycle, from preparation and implementation to closing. With four restructurings (including the AF), and frequent revisions to targets, the Bank team was alert and active to addressing issues as they arose and finding working solutions. This helped ensure that momentum was maintained in project implementation despite security concerns, fiscal financial constraints, and limited local implementation capacity. As a result of the restructurings, the PDO gained in relevance as it responded to the fragility and insecurity in the North East. At the same time, while risks and capacity constraints may have been anticipated at preparation and addressed at implementation, project implementation started in earnest only in 2018, during the second restructuring when the slow pace of progress was addressed. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Page 10 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG:Youth Employment & Soc Support Operat (P126964) Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The project’s results framework was straightforward, with measurable indicators allowing continued monitoring of progress towards meeting the PDO. M&E design was preceded by an assessment of existing M&E systems, which was to form a basis for the project M&E manual. Monitoring systems were to collect baseline data from primary surveys and records with follow-up and progress data drawn from routine data collection mechanisms. The ICR raises an important consideration on measuring outcomes: in a country with a large informal or semi-formal employment system, the impact of a skills training program may be better measured by pre- and post- training wages rather than the pre- and post-job situation. b. M&E Implementation Third-party monitoring provided quarterly feedback, notably on beneficiary selection and payment of grants. Tracer studies were put in place for the skills for jobs component. An intermediate outcome evaluation of all project components was undertaken before project closing. However, in some locations staff capacity was insufficient and reports were not always up to quality. c. M&E Utilization Bank supervision missions tracked progress, and drew on the M&E data to determine restructurings. M&E Quality Rating Substantial 10. Other Issues a. Safeguards The project and the AF were both classified as Environmental Category B, six safeguard policies were triggered for potential impacts of public workfare and livelihood activities, and the appropriate safeguard instruments were disclosed in-country (Environmental Assessment (OP/BP 4.01); Natural Habitats (OP/PB 4.04); Pest Management (OP/BP 4.09); Physical Cultural Resources (OP/BP 4.11); Involuntary Resettlement OP/BP 4.12; Forests (OP/BP 4.36)). No Resettlement Action Plans were prepared as project implementation did not result in land acquisition or restriction of access. The Environmental and Social Management Framework (ESMF) prepared under the parent project in accordance with OP 4.01 was revised and disclosed for the AF. An Integrated Pest Management Plan (IPMP) was prepared as well Page 11 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG:Youth Employment & Soc Support Operat (P126964) during project preparation. Initial public works sites were prescreened for environmental and social risks in line with the ESMF. The overall safeguards rating was satisfactory from 2018 to closing. b. Fiduciary Compliance Procurement. Procurement was slow at the beginning and procurement performance was rated moderately satisfactory at the end of 2017 and became satisfactory in May 2019 since almost all the agreed procurement activities had been fully implemented. Local coordinating and implementing units benefited from training organized by the Bank in January 2018, and all the procurement plans were reviewed (Restructuring paper 2018). The procurement process continued to improve until the second restructuring in 2019. All State Operation Coordination Units had procurement plans approved and were following agreed processes and documentation. Financial management. Financial management (FM) under the project was rated moderately satisfactory early during implementation and became satisfactory at the end of the project. Existing Federal Project Financial Management Division (FPFMD) and State Project Financial Management Unit (SPFMU) structures were used, leveraging existing capacity and experience within the Government. To strengthen the SPFMUs, states deployed additional FM staff and resources, enabling adequate monitoring and supervision of fiduciary-related issues. Flow of funds to states was smooth for the most part, with some delay in the flow of funds to the Federal Project Implementation Unit (FPIU). The project also struggled with late submission of Statement of Expenses (SOEs) by some states. External audits were conducted on regular basis during the entire lifetime of the project and submitted to the Bank in a timely manner, with content and quality acceptable to the Bank. FM under the project was rated Satisfactory at project closing. c. Unintended impacts (Positive or Negative) As a positive impact, the ICR (p. 28) reported that certain beneficiaries of Special Grant Transfers and Public Workfare for unskilled youths were able to save part of their stipends and to start businesses. According to the ICR, this was unanticipated (in contrast to the expected benefits generated by the piloted Skills for Jobs Program). d. Other None mentioned 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Page 12 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG:Youth Employment & Soc Support Operat (P126964) Moderately Outcome Moderately Satisfactory Satisfactory The ICR Review rating reflects design shortcomings that have subsequently hampered implementation, including an original design producing a capital intensive and complex Bank Performance Satisfactory Moderately Satisfactory process, high dependence on government contributions similar to a support operation setup, and conditioning grant transfers on the relocation of internally displaced persons. Quality of M&E Substantial Substantial Quality of ICR --- Substantial 12. Lessons The following lessons are drawn from the ICR: A long project implementation period can be an opportunity to gradually introduce more complex approaches into a fragile environment. This project introduced single registries at scale into an essentially fragile environment, initially allowing the project to keep it simple and gradually add complexity when the mechanism and the context it was operating in was better understood. Care should be taken in conditioning livelihood support on IDPs returning home. Under this operation, research showed that preferences of IDPs - stay, return, or go somewhere else - varied, in this case with some 40 percent wanting to return home, indicating that IDPs themselves might be given the choice of what to do depending on their own judgement of the circumstances they were in. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR was reporting on a project with some complex nuances – the introduction of numerous revisions, changes in scope, etc., with some risk to the quality of the evidence and the coherence of the analysis. Still, Page 13 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG:Youth Employment & Soc Support Operat (P126964) while the main body of the text was long, and similar, albeit nuanced, arguments were made in several places, the results orientation in the document came through even on the few occasions where internal consistency did not seem to be altogether obvious. a. Quality of ICR Rating Substantial Page 14 of 14