Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JM Social Protection (P105024) Report Number : ICRR0021436 1. Project Data Project ID Project Name P105024 JM Social Protection Country Practice Area(Lead) Additional Financing Jamaica Social Protection & Labor P146606 L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-75550,IBRD-83290 30-Sep-2013 80,000,000.00 Bank Approval Date Closing Date (Actual) 13-May-2008 30-Mar-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 40,000,000.00 0.00 Revised Commitment 80,000,000.00 0.00 Actual 80,000,000.00 0.00 Prepared by Reviewed by ICR Review Coordinator Group Hjalte S. A. Sederlof Judyth L. Twigg Eduardo Fernandez IEGHC (Unit 2) Maldonado 2. Project Objectives and Components a. Objectives According to the Loan Agreement (page 5), the project’s development objective (PDO) was “to support the Borrower in strengthening its social protection system by: (a) improving the effectiveness of the PATH to foster investment by poor families in human capital accumulation; (b) developing a structured system for assisting working age members of PATH Eligible Households to seek and retain employment; (c) enabling the formulation of a reform program for public sector pension schemes; and (d) developing a social protection strategy.” PATH denotes the Program of Advancement through Health and Education. Page 1 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JM Social Protection (P105024) The Project Appraisal Document (PAD) had the same objective, albeit by specifying the “development of a holistic social protection strategy." A holistic social protection strategy was defined in the PAD (page 10) as a strategy that would inform decision-making on the appropriate policies and programs to address social risks and vulnerabilities within the population. Substantively, this did not alter the PDO in the Loan Agreement. The ICRR will assess efficacy based on the four specific objectives (a) to (d) in the Loan Agreement. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 18-Dec-2013 PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? No d. Components The project had four components: Component 1: Improving effectiveness of the PATH (estimated cost at appraisal US$36.36 million; Additional Financing US$37.26 million; actual cost US$ 75.64 million). The component had two sub- components: Subcomponent 1.a.: Co-financing of PATH Grants. Under this sub-component, the Bank was to reimburse the Government a share of the cost of PATH grants to children age 0 – 19 years. Higher rates of reimbursement by the Bank were tied to performance milestones linked to institutional strengthening of the Program (Subcomponent 1.b). Reimbursement was to begin at 28 percent, increasing to 35 percent when a first set of milestones had been met, and to 40 percent when a second set had been met. Subcomponent 1b: Institutional strengthening of PATH. This subcomponent included three activities: (i) improving administrative systems; (ii) improving accountability, monitoring, and evaluation; and (iii) improving capacity to implement PATH in the Ministry of Labor and Social Security (MLSS). Administrative systems were to be improved by: (a) better procedures for targeting, enrollment, and recertification of beneficiaries; (b) implementing a Content Management System (CMS) for greater accuracy in data entry and records management; and (c) training social workers and other relevant stakeholders. Accountability, monitoring and evaluation were to be improved by: (a) introducing a strategic management information system (MIS)-based monitoring system; (b) drawing on two process evaluations on program implementation; (c) developing service standards and compliance monitoring; (d) strengthening financial Page 2 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JM Social Protection (P105024) management and internal auditing of PATH; (e) drawing on an impact evaluation to assess PATH’s impact on school enrollment, attendance, and high school completion rates; (f) assessing PATH targeting accuracy; and (g) developing public education and social marketing programs. MLSS’ capacity to implement PATH was to be ensured by the financing of consultant services, goods, and operating costs. Component 2: Building capacity for a Steps-to-Work (StW) Program (estimated cost at appraisal US$2.14 million; Additional Financing US$1.17 million; actual cost US$1.64 million). This component was to support capacity building in the MLSS to implement a new StW Program that referred working age members of PATH-eligible households to support services for seeking and retaining employment. The StW Program was to be launched as a two-year pilot in six parishes targeting approximately 12,000 members of PATH households, and aiming at a subsequent nation-wide rollout. Specifically, the project was to finance: (a) capacity building; (b) incremental operating costs; and (c) monitoring and evaluation, including impact and process evaluations. Component 3: Improving public sector pension system administration and building capacity for reform (estimated cost at appraisal US$0.90 million; Additional Financing US$0.43 million; actual cost US$1.79 million). The component had two subcomponents: Subcomponent 3a: Preparation of a reform program. This subcomponent was to support the preparation of a White Paper outlining a multi-year reform program for the public sector pension schemes. To this end, the project was to finance: (a) training on financial modeling of pension systems, cash flow forecasting, and the design and operations of defined benefit, notional defined contribution, and defined contribution systems; and (b) technical assistance on policy analysis. Subcomponent 3b: Improving public sector administration and information systems. Activities under this subcomponent were to improve the pension administration for timelier pension calculation, award, and payment. The project was to finance: (a) development of a computerized database for storing work histories of public sector workers; (b) redesign of the current pension payment process and supporting information technology (IT) infrastructure; and (c) a review of existing pension legislation for public sector workers to harmonize the pension schemes. Component 4: Development of a holistic social protection strategy (estimated cost at appraisal US$ 0.50 million; Additional Financing US$0.43 million; actual cost US$0.73 million). This component was to support the development of a holistic social protection strategy to inform decision-making on the appropriate policies and programs to address social risks and vulnerabilities within the population. To this end, the project was to finance: (i) diagnostic studies, including but not limited to vulnerability and social safety net assessments; (ii) consensus-building on the reforms required to address the gaps identified; and (iii) formulation of a Social Protection Strategy outlining recommended reform options. Significant changes during implementation Additional financing. Additional Financing (AF) in the amount of US$40 million was approved by the Board in January 2014. At the time, the Government was experiencing the fiscal aftereffects of the Page 3 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JM Social Protection (P105024) international economic and financial crisis that had hit Jamaica particularly hard, and AF was a means to maintaining momentum already achieved under the original project. By that time, most of the targets and all but one of the milestones of the original project had been met. The AF was to continue and further strengthen mechanisms that had been introduced: spending levels for the conditional cash transfers would be maintained, and graduation and recertification procedures, and case management, would be further strengthened; StW was to be expanded and capacity and information systems improved; the public sector pension reform was to be operationalized; and operational support was to be provided for the implementation of the social protection strategy. According to the Restructuring Paper (page 4), additional financing was chosen instead of a follow-up project given the urgency of the situation – an ongoing International Monetary Fund economic program called for safeguarding and strengthening the social protection programs – and the alignment of the AF intervention with the original project as well as the existence of ready implementation mechanisms. While the PDO remained unchanged, some key intermediate results indicators and output targets were adjusted to better reflect implementation at the time of the AF. None were of a nature to require a split evaluation (pre-post AF) of the project. Level 2 restructurings. The project underwent three Level 2 restructurings, in July 2012, June 2013, and November 2017. The first restructuring changed intended impact and process evaluations of the PATH and StW programs, in both cases focusing on implementing only one process evaluation for each activity. Here, the original design may have been too ambitious: in the former case, the intended activities were dropped due to the complexity of the impact evaluation and the lack of suitable candidates to undertake the tasks; in the latter, the cost of an impact evaluation for a pilot phase was not considered worth the investment (Restructuring Paper, pages 7 and 8). In the June 2013 restructuring, the Closing Date was postponed to allow the public sector pension MIS to be tested. In the November 2017 restructuring, key indicators and their outcomes were adjusted to take into account slow progress in implementing the PATH graduation strategy and the scaling-up of the StW. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost. The total project cost at appraisal was estimated at US$115 million, including a front-end fee of US$0.1 million (PAD, p 55). Additional Financing of US$40 million was introduced on December 18, 2013. Actual project costs at closing were US$155 million. Financing. The Bank provided an initial loan of US$40 million to launch the project, followed by an Additional Financing of US$40 million, bringing Bank financing to US$80 million. According to the ICR (p. 2), the Borrower provided US$ 75 million. Dates. The original project was approved on April 9, 2008 and became effective on October 10, 2008, with a Closing Date of September 30, 2013. The AF was approved on December 18, 2013. At that time, Page 4 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JM Social Protection (P105024) the Closing Date of the original project was extended to March 30, 2018 to accord with the AF. Both projects closed on that date. 3. Relevance of Objectives Rationale At project closure, the objectives remained appropriate to the country context and to Bank strategy. They addressed social protection issues that continue to be relevant, and where the Bank has a comparative advantage and is able to draw on broad international experience as well as on previous in-country work. The objectives are among the central features of the current country strategy agreed between the Bank and the Government (Country Partnership Strategy, CPS FY14-17). There they are raised as major objectives in two of the three areas of Bank engagement: (i) stronger government capacity to manage debt, the public pension system, and public sector investment programs under the public sector modernization area, while (ii) stronger social protection programs and improved institutional capacity for their management, as well as increased opportunities for the poor and vulnerable, are part of social and climate resilience area. The objectives are also well aligned with the Performance and Learning Review of the CPS, an extension of the FY14-17 CPS, which is anchored in Jamaica’s long-term development plan “Vision 2030 Jamaica” (ICR, page 12). The CPS noted that the Bank will continue its support for PATH. Specifically, it will: (i) ensure continued investment in human capital; (ii) strengthen PATH institutional capacity and programming; and (iii) invest in opportunities for “graduation” of PATH beneficiaries, by improving their skills and employability through the StW program. It will seek increased opportunities for the poor and vulnerable by continuing to support improvements in social protection programs, protecting and improving human capital, and fostering more inclusive and sustainable urban and rural development. Building on the successes of the Jamaica Social Investment Fund in improving quality of life and reducing violence in vulnerable communities, the Bank will scale up interventions in support of the Government's program of “community renewal." About half of the resources allocated under the FY14-17 CPS were to support integrated community development, social protection, and early childhood projects. Rating High 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective Page 5 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JM Social Protection (P105024) Improve the effectiveness of PATH to foster investment by poor families in human capital accumulation Rationale The objective was to be achieved by building institutional capacity in PATH to channel conditional cash transfers to poor beneficiary families, encouraging them to build human capital through education, better health, and greater employability. It also channeled unconditional cash transfers to elderly and disabled beneficiaries. Outputs a. Changes in co-responsibility behaviors among PATH beneficiary families: Beneficiaries overall. In 2018, the PATH included 338,500 beneficiaries, compared to 277,000 in 2008, of which female beneficiaries numbered 186,000. There was no 2008 baseline for women. The original target was to include 520,000, but was subsequently adjusted to 360,000 as an indicator that took into account natural attrition and new entrants. The share of children in the first quintile (Q1) registered for PATH was 72 percent in 2018, compared to a starting point of 60 percent in 2006. This exceeded the original target of 64 percent, and equaled the revised target of 72 percent. Enrollment. Enrollment at secondary level for poor children (Q1) aged 15 to 16 rose from a baseline of 77 percent in 2006 to 89 percent in 2018, exceeding the original target of 81 percent and the revised target of 85 percent. Attendance. Average daily attendance of children in Q1 at primary level rose from a baseline of 17.8 days per month in 2006 to 19.2 days in 2018, exceeding original and revised targets of 18.5 days and 18.1 days, respectively. For Q1 children at the secondary level, attendance rose from 17.5 days per month to 18.5 days over the same period, equaling the original target and exceeding the revised target of 18.1 days. The share of PATH primary school children attending school 85 percent of the time declined slightly over the 2006-2018 period: attendance went down from 85 percent to 79 percent for boys, and from 85 percent to 83 percent for girls. The original target for both boys and girls - 86 percent - was missed by both. The boys also missed their revised target of 85 percent, while the girls achieved their revised target of 83 percent. At the secondary level, the share of boys attending school 85 percent of the time declined from 80 percent to 79 percent, while it rose for girls from 80 percent to 86 percent. The original target for boys – 84 percent – was not achieved, while the significantly lower revised target of 76 percent was exceeded. Health center visits. The share of children under 1 visiting heath centers declined from 90 percent to 79 percent, and for children between ages 1-3, it remained at 75 percent. The original targets were 92 percent and 80 percent respectively, and the revised targets 90 percent and 75 percent, meaning that the only target reached was the revised target for children ages 1-3. The absence of progress reflects a not uncommon Page 6 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JM Social Protection (P105024) problem of uncertain compliance at the facility level. Contributing factors seem to have been that the demands for health compliance under PATH were not in line with protocol designed in the Ministry of Health, and co-responsibilities contributed to overcrowding in health centers. Case management. MLSS introduced case management to provide PATH households with support beyond co-responsibilities, including links to the StW program. At project closing, 822 PATH families were receiving such support, compared to a projected 925 families. Graduation. A graduation strategy was approved in 2017. The number of PATH families due for a first recertification (and graduation) was 30,500 in 2018, against a target of 37,500. b. Institutional changes as a result of advances in PATH core processes: Targeting. Between 2006 and 2016, the share of beneficiary PATH households in Q1 increased from 45 percent to 64 percent, above the original target of 53 percent and the revised target of 62 percent. The share of children in Q1 registered for PATH increased from 60 percent to 72 percent, against an original target of 64 percent and a revised target of 72 percent. Benefit amount. The benefit amount was periodically reviewed for inflation; reflecting this, and possibly drops in other sources of income, the share of the benefit in the beneficiaries’ budget rose from 20 percent at the beginning of the project to 41 percent at project closing. Payment. 58 percent of new beneficiaries received a first payment according to service standards in 2018, compared to 63 percent in 2013, and an original target of 75 percent. This reflects continuing weaknesses in PATH operational processes that were not solved over the project period. Monitoring. A system of regular process evaluation, internal audit, and quality control was established, as well as service standards for MLSS parish offices. Outreach and social marketing. A focused outreach and social marketing program was regularized to reinforce positive behaviors in terms of PATH beneficiaries’ compliance with health and education co- responsibilities. Outcomes The objective had three formal outcome indicators, two relating to secondary school attendance and graduation, and one related to exit (households who did not seek recertification) and graduation (households no longer eligible) from PATH. The net change in school attendance by PATH children at the secondary level increased by 0.5 days compared to a control group, equaling the original and revised target. Page 7 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JM Social Protection (P105024) The net change in completion rates by PATH children at the secondary level increased by 50 percent compared to a control group, exceeding an original and revised target of 8 percent. 100 percent of families identified for graduation and families identified for exit by the reapplication process have started receiving their exit payments, compared to an expected 70 percent of such families. Summary. The issues – insufficient focus of poor families on human capital development, and low coverage of poor and vulnerable families by PATH – were partly addressed, and are likely to further improve over time as institutional changes are integrated throughout the system. Already, there were functional improvements in PATH, notably beneficiary identification and an MIS. The introduction of co-responsibilities had mixed results. While outcome indicators were achieved, examining intermediate indicators changes the picture somewhat. Then, targets for attendance were only met by secondary school girls. Likewise, intermediate targets for health co-responsibilities were unmet for children under age one, while remaining flat for children between ages 1 to 3 for reasons explained in Section 3a. Then, of two PATH functions - education and health - one was partly met and the other almost not at all. As a result, achievement of this objective is rated modest. Rating Modest PHREVDELTBL PHEFFICACYTBL Objective 2 Objective Develop a structured system for assisting working-age members of PATH eligible households to seek and retain employment Rationale The objective was to be achieved by implementing the nationwide Steps-to-Work (StW) program, an active labor market program to promote economic self-sufficiency among participating working-age PATH beneficiaries. Outputs The StW has been rolled out and is currently functional nationwide. It provides: (i) intermediation services designed to meet the particular challenges of working-age beneficiaries of PATH; and (ii) labor market Page 8 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JM Social Protection (P105024) analysis on demand and supply, and social services to reduce barriers to employment of PATH beneficiaries. Outcomes 822 PATH families were receiving StW services at project completion, falling short of the original and revised target of 925 families; 458 working-age members of PATH households completed skills training, below the original target of 1,000 and the revised target of 600; 28 percent of persons in PATH households that had completed StW training were employed within 12 months, nearly meeting the target of 30 percent; and 78 percent of PATH households that had received StW business training were still in business after 6 months, exceeding the original target of 50 percent and the revised target of 60 percent. StW’s intermediation services directly support PATH’s graduation strategy. The program’s operating handbook incorporates the graduation strategy, including a team of trained social workers that work with households identified for graduation. At project closing, they were working with the 26 households identified for graduation. In summary, the integration of StW into PATH services should significantly strengthen the program’s graduation function over time. However, current outcomes in most instances still fall short of targets. Moreover, the StW continues to face some institutional issues, namely: (i) the absence of a proper information system for monitoring the further careers of StW beneficiaries and for supporting strategic management of the program, and hence the monitoring of results beyond the project period; and (ii) consistent service standards for labor market intermediation services. Rating Modest PHREVDELTBL PHEFFICACYTBL Objective 3 Objective Enable the formulation of a reform program for the public sector pension schemes Rationale Page 9 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JM Social Protection (P105024) The objective was to be achieved by the provision of training and technical assistance on financial modeling of pension systems and technical assistance on policy analysis, as well as improvements in pension administration. Both of these elements were to be used to generate a position paper on public pension reform. Outputs A position paper on public pension reform was submitted to the Cabinet. A public employees pension administration system (an online system integrating data collection, processing, reporting, and use of information) was developed, and 98 percent of the pension administration’s personnel were trained in the use of the system. Outcomes The position paper became law on October 23, 2017. It became effective on April 1, 2018, shortly after project closing. The time for processing applications for pension benefits was reduced from 8 months to 4 months, against a target of 3 months. In summary, the main issue in public pensions appears to be a rapid increase in expenditures (PAD, page 1- 2), which points to problems with the benefit formula and eligibility, i.e. parametric problems. Instead, the project addresses deficiencies in business processes and information systems, the need for better policy analysis, and a stronger database on participants. These are essentially process-type interventions that are unlikely to adequately address the current crisis in the public pension system, except perhaps over a very long period of time, and even then only in combination with parametric reforms. Still, one should recognize that parametric reforms are difficult to initiate, and the approach taken by the team - not to tackle the "elephant in the room" head on, but rather piecemeal -- may have been the appropriate one at this time. This approach will bring about improvements in the system, and together with a position paper on reform, it is likely to provide a platform for a sustained discussion on reform. Considering that it is plausible that the project may achieve its intended outcomes over time, rather than having actually partly or fully achieved this objective, achievement of this objective is rated substantial. Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 4 Objective Page 10 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JM Social Protection (P105024) Develop a holistic social protection strategy Rationale The objective was to be achieved by conducting comprehensive reviews of the social protection system and socioeconomic risks and vulnerabilities in order to create a viable and sustainable social protection strategy that would address such risks and vulnerabilities. Outputs A national social protection committee was established, meeting regularly to coordinate programs and policies in social protection, drawing on stakeholder consultations by the Planning Institute of Jamaica (PIOJ). Outcomes A national social protection strategy was submitted to the Government on September 15, 2017, and was disseminated to stakeholders in March 2018. The strategy seems to have had catalytic effects on the organization of social protection: it is actively used to coordinate cross-sectorally, and the PIOJ, which served as the implementing agency in developing the strategy, has become a central clearing house for social policy, facilitating a more coordinated inter-agency approach to poverty reduction, providing a monitoring and evaluation framework for poverty strategy, and informing targeting and service delivery mechanisms to the poor. It has, moreover, become a reference point in the Bank's support for poverty reduction in the country. Rating High PHREVDELTBL PHOVRLEFFRATTBL Rationale Efficacy for improved effectiveness of PATH to foster investment by poor families un human capital accumulation (Objective #1) is rated modest since project failed to meet its targets for health co-responsibilities and partially achieved its education related targets. Efficacy for Objective #2 (Develop a structured system for assisting working-age members of PATH eligible households to seek and retain employment) is rated modest since the project missed most of its targets with respect to this objective. Efficacy for Objective #3 (Enable the formulation of a reform program for the public section pensions schemes) is rated substantial since the project improved the management and administration of the pension schemes and instigated a reform process that contributed towards addressing their main challenges. Finally, efficacy for Objective #4 (Develop a holistic social protection Page 11 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JM Social Protection (P105024) strategy) is rated high since the project either achieved or exceeded its targets. In conclusion, overall efficacy is rated substantial. Overall Efficacy Rating Substantial PHREVISEDTBL 5. Efficiency No formal economic cost-benefit analysis was performed either at appraisal or at project completion. However, the PAD did simulate the likely effects of changes in the grant scheme on household behavior, using data from the Jamaica Survey of Living Conditions (JSLC) of 2002 and 2006, and applying the proposed reforms to the PATH under the project (corresponding to 80 percent of project costs). The simulations, which were described in detail in the PAD (pages 79 – 92), determined that reforming the benefit level structure for PATH would generate better outcomes for the poor population in Jamaica, slightly increasing enrollments and attendance, and improving schooling outcomes. A similar analysis was not possible at project closing in the absence of 2016 JSLC data for PATH. Nonetheless, comparing the simulated figures in the PAD with available outcomes from the 2016 JSLC did indicate that the reformed PATH benefit structure would generate better outcomes than under the counterfactual. On the other hand, the doubling of Bank financing as a result of the AF does not appear to have increased outcome targets commensurately, with adverse implications for efficiency. Moreover, some implementation aspects may also have reduced efficiency: the complexity of organizing, implementing, and ensuring service standards in a multi-sector operation; weak financial management arrangements and slow procurement processes; and the inability to mobilize and maintain skilled human resources on a consistent basis. Other aspects are likely to have increased efficiency, notably the introduction of graduation and exit from the PATH rolls. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable Page 12 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JM Social Protection (P105024) * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The relevance of the PDO was rated high, as it was appropriate to the country context and to Bank strategy. Efficacy for the first two objectives was rated modest; while actions under them can be expected to result in improved outcomes for PATH families over time, they fell short of immediate targets. Achievement of the third objective, while not immediately addressing the crucial affordability issue in the public pension system, did set the basis for structural reform by supporting a pension reform strategy and through process reforms to the pension agency. It was therefore rated substantial. Efficacy for the fourth objectives is rated high, displaying solid outcomes across the board. The overall efficacy rating was substantial, emphasizing the fact that in most instances the objectives (or intended outcomes) were partly achieved or achieved. Efficiency is rated modest. These ratings indicate moderate shortcomings in the project's preparation and implementation, producing an Outcome rating of Moderately Satisfactory. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome The ICR noted that the Government across administrations has remained highly committed to the project objectives in political as well as economic terms, with a particular emphasis on PATH and pension reform. This was already reflected in annual increases in PATH’s spending allocations throughout project implementation and continues to be so in the regular dissemination of information about the social protection strategy, pension reform, and the PATH program, and the introduction of a graduation function. It is also reflected in the project’s positive effects on the institutional capacity of the MLSS: the introduction of service standards, stronger case management, improved StW training, and second chance education. Still, the ICR pointed to potential risks in the future: the bureaucratic demands to process program activities and take decisions, and uncertainties about further developing and maintaining some key systems, including the management information systems for PATH and the pension administration. 8. Assessment of Bank Performance a. Quality-at-Entry Page 13 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JM Social Protection (P105024) The operation drew on extensive Bank experience in developing social protection strategies and operations internationally, and from its longstanding collaboration with the Government in reforming its social protection system. The results framework was coherent, and the monitoring and evaluation (M&E) system designed to capture relevant outcomes, albeit within an overly ambitious time frame. The risk analysis was thorough. While the MLSS was the appropriate mechanism for implementing the reforms envisioned under the project, taking into account its broad responsibilities for social protection, the institutional placement of the StW program within the MLSS – in social assistance instead of a seemingly more appropriate location in the labor area (ICR, page 27) – may have reduced its effectiveness. On the other hand, the inclusion of the PIOJ to coordinate the planning of the social protection strategy across ministries and agencies appears to have significantly facilitated implementation of the strategy, generating results on time and under budget (ICR, page 28). Other key factors that might have warranted more attention during project preparation were the limited readiness of health and education structures to take on the management of co-responsibilities in their respective sectors; the overall project coordination of the multi-sector elements of the project, notably interactions between MLSS and the Ministries of Education and Health, which appear to have progressively weakened during project implementation; as well as issues of financial management (audits and financial reports), graduation strategy, and procurement processes that delayed implementation. While the financial and procurement issues strictly speaking lay outside the Bank’s control, they might have been recognized and project plans revised accordingly. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision The project had five task team leaders, spread out over a ten-year period. The ICR (page 32) noted that transitions were “seamless,” with ample overlap, and co-leadership in two instances. Supervision had the appropriate skill mix and was pro-active, providing technical assistance and advisory services in addition to regular supervision tasks. It supported new initiatives, notably more comprehensive case management to address the diverse challenges of PATH beneficiaries, and leveraged pension reform with expertise on disability and unemployment insurance. At the same time, key factors that should have been dealt with during preparation – for instance coordination between government agencies, and notably seeking adequate means to address the supply side of co-responsibilities – also affected implementation, and do not appear to have been sufficiently addressed through supervision. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory Page 14 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JM Social Protection (P105024) 9. M&E Design, Implementation, & Utilization a. M&E Design The theory of change, as reflected in the original results framework, was sound. Objectives were clearly specified, and six of the seven PDO-level indicators in the results framework provided a sufficient basis to determine outcomes. The output indicators (intermediate results) were formulated to capture the contribution of project activities towards achieving PDO-level outcomes. All were specific, measurable, achievable, relevant, and time-bound; baselines (where appropriate) and targets were defined and available. However, the number of indicators – in addition to the seven PDO indicators, the project included 20 intermediate indicators – was large and would become cumbersome to monitor during implementation. With the introduction of AF, adjustments were made to the results framework, but the number of indicators remained large; still, a PDO indicator on changes in the poverty gap was dropped as being outside the scope and timeframe of the project, and some other indicators were subject to minor revisions or put aside altogether when their targets had been met under the original project. Finally, the StW strategy was revised to support PATH’s graduation strategy, introducing a monitorable indicator. Project outcomes were assessed using an M&E system to track progress on the indicators specified in the results framework. The system drew on multiple information sources: (i) the MIS developed for the PATH; (ii) biennial JSLC reports; (iii) regular assessments, evaluations, and audits under PATH; and (iv) process and impact evaluations of the StW. b. M&E Implementation The sheer number of indicators became cumbersome to monitor, and early on the Bank team focused on a limited number of intermediate indicators. Over time, as the original project neared the original closing date, all indicators came to be monitored. The scale-up of the StW and the introduction of the graduation strategy did call for additional monitoring indicators, but in the absence of a specific StW MIS, such monitoring was difficult. The elimination of impact evaluations of PATH and StW was compensated for by drawing on administrative data and the JSLC. c. M&E Utilization Data monitoring was critical for successful implementation of the PATH. Here, the beneficiary MIS, including registration, compliance, and payment data, were regularly monitored to assess effectiveness. Adherence to service standards served as an indicator of the quality of services. Page 15 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JM Social Protection (P105024) M&E Quality Rating Substantial 10. Other Issues a. Safeguards The project was classified as environmental category C. No safeguards were triggered. b. Fiduciary Compliance According to the ICR, the financial management Implementation Status Reports rated fiduciary compliance between moderately unsatisfactory and moderately satisfactory during much of the project period. This reflected MLSS’ delays and errors in financial reporting that were rectified only late in the project as additional financial management personnel were hired and the Bank team provided training. In December 2011, the discovery of fraudulent use of funds by a parish office in the amount of US$8,500 led to quick action by the Government and MLSS to strengthen audit capacity, and the Bank through its Integrity Vice Presidency decided not to pursue further action. The misused loan funds – some US$2,000 - were returned to the Bank. Procurement. Procurement processes were at times slow, delaying implementation of key activities. This reflected cumbersome national procurement processes and a lack of knowledge on procuring specific technical services. c. Unintended impacts (Positive or Negative) None reported. d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Page 16 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JM Social Protection (P105024) The IEG rating reflects Outcome Satisfactory Moderately Satisfactory "modest" ratings for two objectives, and for efficiency. The IEG rating reflects Bank Performance Satisfactory Moderately Satisfactory concerns especially about multi-sector coordination. Quality of M&E Substantial Substantial --- Quality of ICR Substantial --- 12. Lessons The following lessons are drawn from the ICR: Political support and ownership on a broad scale are essential to successful social protection reforms. Project success reflected strong overall Government support for objectives, a central forum for social protection in the form of the PIOJ, MLSS’ role as manager of implementation, and the commitment of the Ministry of Finance to pension reform. This created the right environment to implement the project and secure longer-term sustainability. Human capital development requires demand and supply-side interventions. For conditional cash transfers to be effective, they require appropriate working arrangements between stakeholders, notably the support of education and health sectors at national and sub-national levels to provide timely and quality services, and on the demand side the relaxing of financial and information barriers that keep the poor from utilizing services. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR discussed in a clear fashion the background and rationale for the project, and it provided sufficient evidence-based information and analysis to adequately inform the ICRR. Lessons built on matters raised in the text, and with broader applicability. The ICR provided a reasonable balance between results-orientation and narrative; it was internally consistent; the main text, while informative, could have been more succinct. The ICR was written according to the guidelines. Page 17 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JM Social Protection (P105024) a. Quality of ICR Rating Substantial Page 18 of 18