Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GN Productive Social Safety Net Project (P123900) Report Number: ICRR0022117 1. Project Data Project ID Project Name P123900 GN Productive Social Safety Net Project Country Practice Area(Lead) Guinea Social Protection & Jobs L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-D1210,IDA-H7820,TF-18255,TF- 02-Oct-2017 41,457,906.57 A2454 Bank Approval Date Closing Date (Actual) 19-Jun-2012 30-Sep-2019 IBRD/IDA (USD) Grants (USD) Original Commitment 25,000,000.00 6,215,000.00 Revised Commitment 43,204,228.89 6,204,228.89 Actual 41,457,906.57 6,204,228.58 Prepared by Reviewed by ICR Review Coordinator Group Hjalte S. A. Sederlof Salim J. Habayeb Joy Behrens IEGHC (Unit 2) P145117_TBL Project ID Project Name P145117 GN Additiional Financing to PSSNP ( P145117 ) L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) 0 Bank Approval Date Closing Date (Actual) Page 1 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GN Productive Social Safety Net Project (P123900) 11-Mar-2014 IBRD/IDA (USD) Grants (USD) Original Commitment 0.00 0.00 Revised Commitment 0.00 0.00 Actual 0.00 0.00 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The Project Development Objective (PDO) as set out in the Financing Agreement was to provide income support to vulnerable groups and to lay the foundations of a social safety net strategy by testing some of the building blocks necessary for a larger system. The Project Appraisal Report (PAD) had the same formulation for the PDO. The ICR (p. 8) defines building blocks for a safety net strategy as operational tools to improve beneficiary identification and registration; monitor the [safety net] program; and ensure that cash is transferred to eligible beneficiaries in a secure, transparent and timely manner. In fact, these are essential components of a functioning safety net scheme. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? No c. Will a split evaluation be undertaken? No d. Components The Project originally had three components: Component 1: Labor-intensive public works program (LIPW) in urban areas with a focus on youth and life skills development (estimated cost at appraisal US$16.5 million; actual cost US$12.05 million). The component had two sub-components: Sub-component 1.1: Temporary employment opportunities. It was to temporarily increase revenues of the underemployed and unemployed by providing employment opportunities through labor intensive public Page 2 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GN Productive Social Safety Net Project (P123900) works (LIPW) to about 24,000 unskilled/semi-skilled workers in the five communes of the special zone of Conakry and seven regional capitals. Projects were to be drawn from local municipal development plans and selected in consultation with the local populations. In order to promote the participation of women, 30 percent of all LIPW participants were to be women. An estimated 2.4 million person-days of labor (or 100 days of work per beneficiary) were to be created through the project. Local contractors were to be recruited to execute projects and facilitate payments to beneficiaries. Eligible neighborhoods were to be selected by the Project Coordinating Unit (PCU) in consultation with municipal authorities based on poverty and vulnerability criteria that will be specified in the Project Implementation Manual (PIM). Beneficiary selection was to be conducted by neighborhood leaders in consultation with local authorities and local contractors under the supervision of the PCU. Sub-component 1.2: Life skills development for youth. This sub-component was to provide 5,000 youths with trainings in non-cognitive skills. The youth were to be selected from the LIPW beneficiaries based on demand, a minimum educational level, and regular attendance at the LIPW programs. Training content was to focus on topics such as civic education, the rights and responsibilities of workers, conflict prevention and resolution, HIV/AIDS prevention and treatment, entrepreneurship, etc. A total of 100 training sessions were to be offered over the five years of project implementation. Component 2: Pilot cash transfer program to protect human capital (estimated cost at appraisal US$4.5 million; actual cost US$4.5 million). The component had three sub-components: Sub-component 2.1: Design of the pilot project. UNICEF, as the only agency with a national network of local infrastructure and the technical capacity to rapidly facilitate one of the first cash transfer interventions for nutrition and education in the country, was to be contracted to implement this sub-component. The sub- component included the elaboration of a technical annex on targeting, beneficiary selection, registration, and payment methods for the pilot cash transfer as a first step towards the establishment of cash transfer systems. Sub-component 2.2: Cash transfer for nutrition. This sub-component was to test the effectiveness of cash transfers in reducing the malnutrition rate of targeted children. It was to benefit at least 5,000 children aged 0-24 months that were affected by chronic malnutrition in at least two of the regions with some of the highest chronic malnutrition rates in the country. Beneficiary selection was to be based on poverty and chronic malnutrition levels of children and will involve community participation. Cash transfers to the beneficiaries were to be made by payment agencies based on a list of selected beneficiaries. The transfers were to be provided publicly to the mothers of the malnourished children, once they present their beneficiary IDs. The payments were to be made once every two months to reduce the transaction cost but at the same time, to allow beneficiary households to purchase food as needed. The amount of transfer per child was determined based on household consumption levels and best international practice. Training on health, nutrition, and sanitation have been included in the package to maximize the impact of the cash transfer on nutrition. Sub-component 2.3: Cash transfer for girls’ education. This component aimed at testing the efficiency of cash transfers in increasing the primary school completion rate among beneficiaries. The intervention was to target at least 5,000 girls in the fifth and sixth grade in at least three regions where the girls‘ completion rate is among the lowest. A committee including representatives from the Parent–Teacher Association (PTA), the Ministry of Primary Education, and the Rural Development Communities was to be established to select the schools that were to benefit from the project based on the highest ratio of girls‘ dropout. Beneficiary households were to be selected at the community level by the community based on Page 3 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GN Productive Social Safety Net Project (P123900) poverty criteria. The amount of transfer per child was to be determined based on household consumption levels; and all households in the targeted villages were to be sensitized to the importance of girls’ education. Component 3: Project coordination and institutional capacity building (estimated cost at appraisal US$4.0 million; actual cost US$4.32 million). The component was to ensure that the unit coordinating project implementation, the CFS (Cellule Filets Sociaux, Safety Net Unit) was operational, managing contracts with partners involved in implementation, and conducting supervision. The component also was to finance institutional capacity building, to support the development of the social protection strategy. Significant Changes During Implementation The Project was restructured five times. Significant changes included the following: During a first restructuring, undertaken in June 2014, US$ 2.00 million of (joint) co-financing from the United Nations Peace Building Fund (UNPBF) was provided to scale up the LIPW scheme, and the design of the cash transfer program was expanded to include a school feeding program. A second project restructuring, in October 2014, coincided with a first Additional Financing. A new project component – Response to the Ebola Epidemic – was introduced, supported by a reallocation of US$6.00 million of LIPW funds to the new component. The component was to provide logistical support to the treatment teams on the ground, support community mobilization, and provide training for teachers and officials in emergency responses to the epidemic. Additional Financing of US$1.865 million was also provided by UNPBF, raising total spending on this component to US$7.87 million. A third project restructuring in May 2016, included a second Additional Financing of US$16.35 million. It included an IDA grant of US$12.00 million and an Ebola Recovery and Reconstruction Trust Fund Grant of US$4.35 million. A fourth and fifth restructuring were undertaken in October 2017 and May 2018, including marginal changes in project design and an extension in the Closing Date. The first four restructurings were accompanied by changes in the results framework, notably several revisions to the PDO indicators – by project closing all eight key indicators introduced at project approval had either been dropped or revised, eventually being replaced by six key indicators. All of the revisions to key indicators were either modifications of targets or elaboration of the original indicators to more appropriately reflect sought-for results (for instance, from “beneficiaries of safety net programs” to “direct project beneficiaries of which female”). Other changes included several changes in implementation arrangements as authority over the project coordinating unit CPS shifted, cash transfer design changed from unconditional to conditional and back to unconditional, allocations between disbursement categories were modified, with the introduction of the Ebola component; and extension of closing dates. None of the changes were of a kind to justify a split evaluation. Page 4 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GN Productive Social Safety Net Project (P123900) e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost. At appraisal, the total cost of the project was estimated at US$25 million. Actual project cost at Closing (funds disbursed) was US$43.22 million. Financing. The project was initially financed by a US$25.00 million IDA credit of which US$23.23 million was disbursed over the project period. A second IDA credit of US$12.00 million was approved in May 2016 and was fully disbursed. At the same time, US$4.35 million was drawn from the Ebola Recovery and Reconstruction Trust Fund and fully disbursed. In addition, joint financing from the United Nations Peace Building Fund (UNPBF) of US$3.87 million was provided in June 2014 and October 2014 (US$3.85 disbursed). Differences between amounts awarded and disbursed reflect exchange rate changes. Borrower contribution. There was no Borrower contribution. Dates. The project was approved on June 19, 2012 and became effective on February 14, 2013. The original Closing Date, October 2, 2017, was extended on May 9, 2016, to September 30, 2019, at which time it closed 3. Relevance of Objectives Rationale The PDO was relevant to country needs, government policy and Bank strategy. With over half the population living in poverty, and a broader segment vulnerable to economic and environmental shocks, risk coping strategies had been taking the form of informal family and community safety nets and discrete crisis interventions by donors. The international economic crises towards the end of the last decade, and the return to civil government at around the same time, provided an opportunity for introducing pro-poor economic and social policies as part of the public policy toolbox. These were guided by the government’s Poverty Reduction Strategy Paper and an accompanying Interim Strategy Note (ISN) for 2011-2012. The documents advocated a two-stage approach towards social protection: in the short term assisting the poorest and most vulnerable, while at the same time moving towards developing a comprehensive longer- term safety net. The approach is reflected in the Project objectives; it is consistent with the main themes in the Bank’s Social Protection Strategy for the period 2012-2022 developed at the time; and with the Bank’s Africa Social Protection Strategy, which emphasizes the productive aspects of safety nets. The approach is also reflected in the government’s National Policy on Social Protection, developed under the Project and approved by the government in 2017. It is consistent with the Country Partnership Framework (CPF) for FY2018-2022. A distinctive feature of that CPF is that it continues the focus on strengthening resilience to shocks by maintaining the Bank’s support for the further development of social protection systems and labor-intensive technologies. Human development, including improved basic education, health, and social protection, forms a second pillar in the thee-pillar CPF, and the focus in social protection is on continued Page 5 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GN Productive Social Safety Net Project (P123900) strengthening of the main safety net institutions, introduced under the Productive Social Safety Net Project. . While the operation’s objectives are consistent with country strategy at project start-up and remain in the post-project period, the ICR raises the concern that the Project may have been too complex for a country with the limited administrative capacity of Guinea, and that may be reflected in the numerous restructurings that took place. At the same time the situation on the ground was likely to be such that a project that introduced essential building blocks necessary to operate a cash transfer system was appropriate, while recognizing the risks of doing so in a fragile environment. Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To provide income support to vulnerable groups Rationale The objective was to be achieved by the provision of LIPW and direct cash transfers – either conditional or unconditional (CCT, UCT), and by linking them to accompanying measures intended to leverage the effects of the LIPW and cash transfers in improving consumption and asset holding as well as human capital among beneficiary families. Accompanying measures included learning sessions on financial education, productive activities, hygiene practices, nutrition, and early childhood development. Outputs (baselines and targets are provided in parentheses):  The Project provided 1.75 million (0; 1.78 million) person days of LIPW benefiting 61,112 people (0; 41,118), including 36,668 women (0; 25,000).  9.33 million Guinean Francs (GNF) in wages were distributed to LIPW workers (0; 7.24)  408 microprojects implemented under LIPW schemes were completed and registered with local authorities (0; 322). Page 6 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GN Productive Social Safety Net Project (P123900)  1,800 households benefited from the CCT program, with 69 percent of beneficiaries complying with the conditions – attending school and seeking regular check-ups at health centers (0;9,000; 90); and 6,900 households benefited from UCTs  Nutrition, parenting, early learning, stimulation education was provided to 25,000 beneficiaries. Outcomes Altogether 75,338 households (0: 84,000) benefited from LIPW and cash transfers. An ex-post evaluation of the programs indicated high poverty targeting with an estimated 80 percent of beneficiary households classified as poor. While there is no information on changes in household wellbeing (consumption, asset holding), the ICR (p. 15), drawing on field observations and evidence gathered by the project coordinating unit (CFS), suggested that beneficiaries spent their additional income on food and investments in productive assets (animals, land). The impact evaluation also assessed the effects of the cash transfers on school attendance, notably the CCT phase where conditional cash was distributed to 1,800 households. According to the ICR (p. 16), the effects were positive, with school enrollment 11 percent higher in treated villages and households. In terms of learning outcomes, the results were not significantly different between treated and control villages. It is worth noting that only 69 percent of beneficiary families actually complied with the conditionalities: the pilot found that application of the CCTs was constrained by the lack of services, as well as by reluctance to monitor compliance (not unusual in CCTs at local levels). The Project did not measure the impact of the accompanying measures. However, the ICR noted that anecdotally many beneficiaries appreciated the value of saving and reinvesting. In the absence of stronger, more results-oriented indicators – indicators were essentially output-oriented - Objective 1 is rated modest. Rating Modest OBJECTIVE 2 Objective To lay the foundations of a social safety net strategy Rationale The objective was to be achieved by developing a national policy document and testing key building blocks for a safety net, including beneficiary identification and registration (targeting), a payment system, a grievance redress mechanism, and a management information system (MIS). Page 7 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GN Productive Social Safety Net Project (P123900) A National Policy on Social Protection providing the context for developing a broad social protection strategy was approved by the Council of Ministers in 2017. Key building blocks for a safety net system were tested:  Systems for beneficiary identification using geographical targeting and proxy means testing were successfully tested (based on targeting accuracy of 82 percent and community verification);  An electronic beneficiary registration system (applying a new MIS designed for the safety net) was successfully used in registering beneficiaries of LIPW and transfer programs;  Payment mechanisms were tested – cash in hand through a bank, and digital payments in urban areas;  A grievance redress mechanism was developed, and grievances were tracked through the MIS from 2017 onward. By late 2019, all complaints registered in the system had been addressed. Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale The Project introduced a framework for a national social protection policy, as well as key building blocks of a safety net system in addition to addressing immediate poverty needs by the introduction of LIPW and cash transfers. The strong output orientation of the project – impact evaluations did not allow for assessing the changes among beneficiaries of measures introduced under the Project – specifically under Objective 1 – should not detract from the otherwise relatively successful performance, especially under Objective 2. Overall Efficacy Rating Substantial 5. Efficiency The efficiency analysis in the ICR built on the one in the PAD. The analysis covered the LIPW and cash transfer components and was based on reasonable assumptions about beneficiary identification and registration, and costs and benefit levels. The analysis indicated that the LIPW was cost effective compared to similar programs in the region, primarily due to high labor intensity and targeting efficiency: 60 percent of program costs went to wage transfers, while targeting efficiency was estimated at 82 percent. In terms of cost effectiveness, it cost US$1 to deliver US$2.87 to beneficiaries. A similar calculation for cash transfers indicated that it cost US$1 to deliver US$2.56 to beneficiaries. Page 8 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GN Productive Social Safety Net Project (P123900) The ICR cautions against reading too much into the analysis. There were inconsistencies in monitoring during implementation because of frequent changes in project indicators, and logistical challenges due to difficult field- level data gathering. Additional factors that adversely influenced efficiency were implementation difficulties requiring project restructurings and extending the project period, low administrative capacity in Guinea, as well as frequent changes in the Bank team. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of objectives was rated substantial but did not align well with absorptive capacity. Efficacy was rated substantial, consistent with almost fully achieving project objectives. Efficiency was rated modest reflecting concerns about the economic analysis due to data inconsistencies. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome At project start-up, overall risk was rated high, reflecting a still uncertain political environment, as well as low administrative capacity, and modest experience with safety nets. While administrative capacity remains modest, the situation has improved – the political environment is more stable, the project has helped build safety net institutions, and administrative responsibilities are better defined. Moreover, the Bank has a continuing engagement with the country in the social sectors, including safety nets. Page 9 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GN Productive Social Safety Net Project (P123900) 8. Assessment of Bank Performance a. Quality-at-Entry The project was strategically relevant, addressing both government and Bank concerns about the absence of a permanent safety net in Guinea. Preparation was based on analytical work, field-level consultations, and policy dialogue with the new government and its development partners. Design drew on Bank experience in addressing social safety net issues in other countries in the region and addressed immediate welfare concerns as well as longer-term planning of a safety net. Environmental and social safeguards were appropriate; and a fiduciary team was hired to ensure adequate financial management and procurement practices. While the team recognized overall project risks, including risks relating to governance and absorptive capacity, the project would still turn out to be complex and result in several restructurings: implementation would be affected by a dearth of local sector experience and lessons that would influence CCT application; by an awkwardly formulated PDO that would result in several changes in project indicators and an emphasis on output-related indicators rather than outcomes (Section 9); and implementation arrangements that were to require several adjustments in project oversight. Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision Implementation placed significant challenges on the supervision effort as the project adapted to the implementation challenges: project scope, structure, and contents were changed several times connection with the restructurings and accompanied by adjustments in the formulation of key indicators and targets. The introduction of an Ebola crisis component added to the challenges. Moreover, task team leaders changed five times on a project that by its nature – piloting and testing unfamiliar approaches in a weak environment – could have benefited from greater attention to continuity. To address the changing circumstances, the team undertook 17 supervision missions accompanied by telecommunications and correspondence. As a result, the major elements of the project came to be tested and later adapted to circumstances, and the findings now form a basis for building a national safety net. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization Page 10 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GN Productive Social Safety Net Project (P123900) a. M&E Design M&E as well as an impact evaluation was to be coordinated by the CFS with the assistance of a contracted M&E specialist. An M&E database was to be established for each component to collect information on all beneficiaries, and subsequently a more in-depth survey was to be undertaken of a sample of participants. A results monitoring framework was to assess progress towards the PDO using key indicators. An evaluation was to be undertaken to further inform the establishment of a safety net system. The ICR (p. 26-27) points to several weaknesses in project design that influenced M&E design. The ICR found the wording of the PDO, especially the second objective, ambiguous: did it relate to testing, or to institutional development? The IEG found it less ambiguous than the ICR, interpreting it as an elaboration of a social protection strategy (of which safety nets were an element); and development of the key components of a safety net system. It did find that the outcome-oriented indicators could have been strengthened with more focus on changes in household well-being. However, this may be due to the nature of the activities – introducing new and complex concepts into an unfamiliar environment where outcomes may have been difficult to define. Beyond the question of indicators, the ICR noted that CFS capacity, even with the assigned technical assistance, was insufficient for the extensive geographic area it was to cover. b. M&E Implementation With indicators and targets changing several times, and with the introduction of the Ebola component, tracking progress was difficult. Combined with the project’s complexity, this resulted in fluctuations in reporting and uncertainty over what progress was being made. c. M&E Utilization Data was used to monitor progress on components and better focus interventions and recognize revisions when needed. M&E Quality Rating Modest 10. Other Issues a. Safeguards The Project’s environmental assessment category was rated B given small-scale and localized public works. An Environmental and Social Management Framework (ESMF) was prepared and applied to assess the impact of the LIPW on the environment. Assessments involved consultations with key stakeholders and informed the design of specific Resettlement Action Plans (RAP) and Environmental and Social Management Plans (ESMP) when necessary. Public consultations were held on action plans, and Page 11 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GN Productive Social Safety Net Project (P123900) community systems were established to handle grievances. All key stakeholders were sensitized and trained on the mitigation measures. At project closing, safeguard risk was rated moderate. b. Fiduciary Compliance The CFS hired a fiduciary team including a financial management specialist, an accountant, and a procurement specialist. It set up adequate mechanisms to verify payments to beneficiaries and manage complaints. Throughout implementation, FM performance was rated as Satisfactory and Procurement performance was rated as Moderately Satisfactory, except in the initial period due to the effectiveness delay. Given the volatility of the exchange rate, the fiduciary risk rating was increased from Moderate to High. No waivers were required of the World Bank’s fiduciary policies, no ineligible expenditures were presented, interim unaudited financial reports were presented in a timely fashion and the final annual audit report for the period ended January 31, 2020 provided an unqualified/clean opinion. c. Unintended impacts (Positive or Negative) None noted d. Other Gender. The LIPW activities sought participation of women by targeting poor female-headed households, emphasizing female participation and sensitivity to cultural aspects of female spaces. The number of female participants exceeded original targets, and meaningful participation of women in public works activities was achieved through: (a) training the selection committees on female empowerment; (b) introducing informal nurseries for mothers to drop off their children; (c) identifying appropriate work for women, including in setting work norms; (d) working hours that took into account women’s household obligations; and € separate toilets for female and male beneficiaries. 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory Satisfactory Moderately Bank Performance Moderately Satisfactory Satisfactory Quality of M&E Modest Modest Quality of ICR --- Substantial Page 12 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GN Productive Social Safety Net Project (P123900) 12. Lessons The following lessons are drawn from the ICR: Projects in low capacity environments are best kept simple. With a complex design, project implementation faced institutional challenges that could have been reduced with greater attention to administrative capacity and the pacing in the introduction of new systems. In the project, that pacing was only introduced as challenges were encountered. While the urgency of implementation may have been well-founded, the reality on the ground ultimately sets the pace. Continuity in Bank teams can facilitate implementation. Closing of the project was delayed by two years. While the delay may reflect a more realistic timetable for the establishment of a social protection strategy or a functioning safety net system, a high turn-over of team leaders is also likely to have contributed, and the ICR (p. 33) points to the possibility of consequent misunderstandings and modifications as factors contributing to slowing down implementation. 13. Assessment Recommended? Yes ASSESSMENT_TABLE Please Explain The Project put in place key elements of a needed and functioning social protection system in a very weak environment. While there will be follow-up operations, it would be useful to look at some results – because of both the challenge of the environment and the absence of bona fide outcome measures in the project. 14. Comments on Quality of ICR The project provided enough information to allow the project to be evaluated. The evidence drew on project documentation – ISRs and field visits, as well as CFS documentation. The analysis was able to bring out salient features of implementation and allowed drawing of links between implementation and results. To some extent this was affected by the absence of strong outcome indicators and, more generally, by several changes in indicators. The ICR presented a clear theory of change which guided the analysis, which was internally consistent. The analysis was internally consistent and responded to guidelines. At 33 pages, it was on the longer side. a. Quality of ICR Rating Page 13 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GN Productive Social Safety Net Project (P123900) Substantial Page 14 of 14