Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Social Protection System Support(P151279) Report Number : ICRR0021648 1. Project Data Country Practice Area(Lead) Rwanda Social Protection & Jobs Programmatic DPL Planned Operations: 0 Approved Operations: 0 Operation ID Operation Name P151279 Social Protection System Support L/C/TF Number(s) Closing Date (Original) Total Financing (USD) IDA-55800 31-Mar-2016 66,240,910.00 Bank Approval Date Closing Date (Actual) 22-Jan-2015 31-Mar-2016 IBRD/IDA (USD) Co-financing (USD) Original Commitment 70,000,000.00 0.00 Revised Commitment 70,000,000.00 0.00 Actual 66,240,910.00 0.00 Prepared by Reviewed by ICR Review Coordinator Group Hjalte S. A. Sederlof Judyth L. Twigg Eduardo Fernandez IEGHC (Unit 2) Maldonado PHPROJECTDATATBL Operation ID Operation Name P155024 RW-Second Social Protection Sys (SPS-2) ( P155024 ) Page 1 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Social Protection System Support(P151279) L/C/TF Number(s) Closing Date (Original) Total Financing (USD) IDA-55800,IDA-57460 30-Sep-2016 93,962,861.00 Bank Approval Date Closing Date (Actual) 01-Dec-2015 30-Sep-2016 IBRD/IDA (USD) Co-financing (USD) Original Commitment 95,000,000.00 0.00 Revised Commitment 95,000,000.00 0.00 Actual 93,962,861.00 0.00 PHPROJECTDATATBL Operation ID Operation Name P158698 Third Social Protection System (SPS-3) ( P158698 ) L/C/TF Number(s) Closing Date (Original) Total Financing (USD) IDA-55800,IDA-57460,IDA-59100 31-Dec-2017 91,399,594.00 Bank Approval Date Closing Date (Actual) 31-Oct-2016 31-Dec-2017 IBRD/IDA (USD) Co-financing (USD) Original Commitment 95,000,000.00 0.00 Revised Commitment 95,000,000.00 0.00 Actual 91,399,594.00 0.00 2. Program Objectives and Policy Areas a. Objectives Page 2 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Social Protection System Support(P151279) The Social Protection System operation (SPS) consisted of a series of three Development Policy Operations (DPO). The program development objective (PDO) for the series was set out in the Program Document (PD) of the first operation, SPS-1. It was to “support the Government of Rwanda to improve the efficiency, accountability and coverage of its social protection system.” Achievement of the PDO will be assessed on the basis of three objectives: (i) improved efficiency of the social protection system; (ii) improved accountability of the social protection system; and (iii) improved coverage of the social protection system. The PDO remained unchanged throughout the series. b. Pillars/Policy Areas The series included three pillars: Pillar 1: efficiency of the social protection system to improve administrative efficiency of the SPS and help generate value for money by updating the database of poor and vulnerable households; improve the management information system (MIS), including a registry of beneficiaries; and increase harmonization across social protection programs. Pillar 2: accountability and transparency to improve transparency and accountability in SPS budget reporting, and strengthen citizen engagement. Pillar 3: coverage of the poor and vulnerable to increase social protection coverage of the poor and vulnerable population, and improve targeting. c. Comments on Program Cost, Financing, and Dates Program cost and financing. The series consisted of three IDA credits of US$70 million, US$95 million, and US$95 million, respectively. The full amounts for each credit were disbursed. Borrower contribution. None. Dates. SPS-1 was approved on January 22, 2015 and closed on March 31, 2016; SPS-2 was approved on December 1, 2015 and closed on September 30, 2016; and SPS-3 was approved on October 31, 2016 and closed on December 31, 2017. There were no postponements of closing dates of any of the credits. Page 3 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Social Protection System Support(P151279) 3. Relevance of Objectives & Design a. Relevance of Objectives The SPS series’ objectives were fully aligned with country needs, the Bank’s Country Partnership Strategy (CPS) FY2014-2018 in effect at series closing, and Rwanda’s National Social Protection Strategy. Despite impressive reductions in poverty, Rwanda remains one of the poorest countries in the world with high levels of vulnerability. The National Social Protection Strategy aims at continued improvement in the efficiency and effectiveness of the country’s social protection system, notably the safety net, and further reductions in poverty and inequity. Improving productivity and incomes of the poor is a central theme (Theme 2) of the CPS, which explicitly mentions social protection as a mechanism for doing so. The series was also consistent with the Bank’s broader social protection strategy for Africa aiming at eliminating extreme poverty and boosting shared prosperity. Rating High b. Relevance of Design SPS design – a three-operation series – was relevant to its objectives. The PD provided a clear rationale for the choice of pillars/policy actions, and the theory of change as presented in the policy and results matrix linked prior actions and triggers to policy actions that were likely to achieve set series objectives. Key outcome indicators were defined, albeit with an output orientation, given the initially short project timeframe. The operation drew on the experience of two earlier DPO series, and implementation was supported by analytical work and hands-on technical assistance. Rwanda’s macro-fiscal framework was robust at the start of the operation and remained so throughout the series. At the time of project preparation, the International Monetary Fund had completed a second review under a three-year Policy Support Instrument and the Article IV Consultation. The short- to medium-term economic outlook was positive, with the economy expected to manifest strong growth, and the risk for debt distress to remain low. The level of financial support under the SPS series was considered consistent with the government’s financial commitment to social protection. Rating Substantial Page 4 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Social Protection System Support(P151279) 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective To improve the efficiency of the social protection system Rationale Rationale: The objective was to be achieved by: (i) strengthening the national poverty classification system and database (Ubuhede) to improve inter-operability across safety net and other poverty-targeted poverty program databases; (ii) improving the MIS, including a registry of social protection program beneficiaries; and (iii) harmonization across social protection programs to prevent duplication in benefit provision. To strengthen the poverty classification system, • The Ministry of Local Government (MINALOC, the main project implementing agency) adopted a road map for updating Ubuhede (SPS-1 prior action); signed a memorandum of understanding with the national ID agency, NIDA, on the technical interface between the two agencies (SPS-2 trigger), which became fully functional under SPS-3 (SPS-3 trigger). As a result, 90 percent of the population over 16 (those with a national ID) were matched to the Ubuhede database. The baseline was 25 percent and the target 60 percent. To improve the MIS, • Social protection MIS objectives for the short-, medium- and long term were approved by a social protection MIS committee convened by MINALOC (SPS-1 prior action); MINALOC finalized, validated, and adopted MIS technical specifications (SPS-2 trigger); and MINALOC developed priority MIS modules and issued protocols for MIS use and maintenance (SPS-3 trigger) For harmonization, • MINALOC issued guidelines on benefit allocation when a household was eligible for more than one benefit (SPS-1 prior action); developed guidelines for harmonizing income-generating activities across two core social protection programs linked to the MIS, meeting the target (SPS-2 trigger); and implemented Page 5 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Social Protection System Support(P151279) the guidelines (SPS-3 trigger). To ensure correct implementation, all local government staff were trained in applying the guidelines, exceeding the target of 70 percent. Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 2 Objective To improve accountability of the social protection system Rationale Rationale: The objective was to be achieved by: (i) improving transparency and accountability in social protection budget reporting and discussion; and (ii) strengthening citizens’ engagement. For improved accountability and transparency, and to strengthen citizens’ engagement, • The Ministry of Economy and Finance (MINECONFIN) produced a consolidated social protection budget (SPS-1 prior action), accessible to the public through MINALOC’s website, through joint sector reviews and sector working groups, and district and sector level public accountability days (SPS-2 trigger). The latter – the accountability day events – served as a key indicator of transparency and accountability for the operation: against a target of such events taking place in 60 percent of districts in 2018, 85 events were recorded. • Staff was increased and trained in citizens’ engagement at district and local levels in promoting citizens’ engagement at those levels (SPS-1 prior action, SPS-2 trigger). • An appeals and complaints system was implemented (SPS-3 trigger), is operational and used: numbers in the ICR (page 19) covering the period 2014-2018 indicate that 68-75 percent of appeals/complaints are settled in favor of the complainant. Here, the baseline was 80 percent and the end-target over 80 percent. Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 3 Page 6 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Social Protection System Support(P151279) Objective To improve coverage of the social protection system Rationale Rationale: The objective was to be achieved by: (i) expanding coverage in line with Rwanda’s long term strategy; and (ii) testing improved poverty targeting and introducing gender and child sensitive social protection programs to promote child development. For expanded coverage, • MINALOC and MINECONFIN approved plans and budgets for FY15 (SPS-1 prior action), FY16 (SPS-2 trigger), and FY17 (SPS-3 trigger). • MINALOC adopted procedures to enhance community participation in the Ubuhede household classification process and launched testing of poverty scorecards (SPS-1 prior action). • The program exceeded sector coverage in terms of both access to direct support and public works, with access to direct support increasing from 240 geographic sectors to all 416 sectors of the country (against a target of 395 sectors); and access to public works increasing from 150 to 240 sectors (equaling the target). For gender and child sensitive programs, • MINALOC identified policy options for improving gender and child sensitivity of social protection programs (SPS-1 prior action); introduced a plan for introducing selected options (SPS-2); and incorporated the plan into the FY17 budget (SPS-3). • The program substantially achieved its targets for the number of households and female-headed households with access to direct support and public works. Direct support households increased from 44,000 in FY2013/14 to 95,000 in FY2017/18, against a target of 96,000 households (which was achieved the previous year); 69 percent were female-headed, against a target of 66 percent. Coverage of public works increased from 89,000 households in FY2013/14 to 135,000 in FY2017/18, against a target of 130,000; the share of female-headed households increased from 46 percent to 51 percent, against a target of 48 percent. Page 7 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Social Protection System Support(P151279) For better poverty targeting, • There are no indicators of changes in poverty targeting errors. However, elements of restructuring of the poverty classification system and its application to geographical targeting – notably a broadening of categories likely to reduce errors of exclusion, and prioritizing geographical sectors within the poorest districts - should plausibly have increased the poverty focus and reduced both errors of exclusion and inclusion. Rating Substantial PHREVDELTBL PHREVISEDTBL 5. Outcome Relevance of objectives is rated high and relevance of design is rated substantial. The former is fully in line with Bank and government priorities in social protection, and address country needs. The latter is in line with objectives, but more consideration could have been given to more outcome-focused key indicators. Efficacy is rated substantial for all three objectives. These ratings indicate only minor shortcomings, producing an Outcome rating of Satisfactory. a. Outcome Rating Satisfactory 6. Rationale for Risk to Development Outcome Rating The risk to development outcome is low. Over the years, the government has consistently pursued poverty reduction and a strong social protection system; this is reflected in policy documents setting out strategic priorities for poverty reduction. The policy actions under this operation are being further developed through technical assistance financed under an investment project (Strengthening Social Protection), and parallel technical assistance provided by other donors. a. Risk to Development Outcome Rating Page 8 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Social Protection System Support(P151279) Negligible 7. Assessment of Bank Performance a. Quality-at-Entry The SPS series was prepared on the basis of a sustained commitment on the part of the Bank to help build up social protection. The team drew on the experience of earlier Bank development policy operations in Rwanda, and on related analytical studies as well as collaboration with other donors, in close interaction with the client. The choice of development policy lending appears appropriate in the presence of a strong sector strategy and a solid results framework. M&E design was adequate. Quality-at-Entry Rating Satisfactory b. Quality of supervision According to the ICR and judging from the supervision budget, supervision was strong, results focused, and characterized by a joint Bank-client effort, including development partners, to move the agenda forward. Regular supervision was provided from the country office, which facilitated timely hands-on support in providing advisory services and technical assistance. All prior actions and triggers were met in a timely fashion, and each of the DPOs was delivered successfully. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory 8. Assessment of Borrower Performance a. Government Performance The government appears to have been fully committed to the SPS series and the improvements that were being put into place. MINALOC, the implementing agency, provided effective support in policy formulation and oversight of implementation and monitoring processes that allowed smooth and timely implementation. Government Performance Rating Satisfactory Page 9 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Social Protection System Support(P151279) b. Implementing Agency Performance See Section 8a. Implementing Agency Performance Rating Not Rated Overall Borrower Performance Rating Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The policy matrix and results framework for SPS-1 provided a logical sequence of actions that were to lead to desired outcomes. They included prior actions and triggers for each of the three operations, as well as seven key outcome indicators. An impact evaluation and regular household surveys of living conditions were to be conducted every three years b. M&E Implementation A sector working group provided an overall framework for monitoring performance on policy, outputs, and outcomes on a semi-annual basis, making recommendations for the following six months. c. M&E Utilization Data from the household surveys and the impact evaluation was used to guide the design of each subsequent phase of the SPS and the government’s social protection program. M&E Quality Rating Substantial 10. Other Issues a. Environmental and Social Effects Page 10 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Social Protection System Support(P151279) No safeguard policies were triggered. b. Fiduciary Compliance Fiduciary issues were not discussed in the ICR. c. Unintended impacts (Positive or Negative) None noted. d. Other None recorded. 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory --- Risk to Development Negligible Negligible --- Outcome Bank Performance Satisfactory Satisfactory --- Borrower Performance Satisfactory Satisfactory --- Quality of ICR Substantial --- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 12. Lessons Page 11 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Social Protection System Support(P151279) The lessons are drawn from the ICR. National vision can be an overriding factor for success. Teams need to be alert to government commitment in recommending reforms. In Rwanda, the government’s sustained policies and financial commitment to social protection has facilitated dialogue with the Bank and development partners and created government-led policies that have facilitated ambitious forward movement. Early action on accountability and transparency can enhance the credibility of public programs. Broad support is often necessary if a social protection initiative is to be successful. In Rwanda, the introduction of an appeals and complaints mechanism, and subsequently broader accountability initiatives (social accountability, citizens engagement tools) helped ensure that beneficiary and other stakeholder views were mainstreamed into project planning and implementation. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR provided sufficient evidence-based information and analysis to adequately complete the ICRR. It provided a reasonable balance between results-orientation and narrative. In some instances – especially the discussion surrounding policy area 1 -- the ICR seemed to assume in-depth knowledge of Rwanda’s safety net structures in order to easily follow the discussion. The document was internally consistent and written according to guidelines. At 29 pages, it was fairly long. a. Quality of ICR Rating Substantial Page 12 of 12