ICRR 14296 Report Number : ICRR14296 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 12/05/2013 Country : Chile Project ID : P082037 Appraisal Actual Project Name : Social Protection US$M ): Project Costs (US$M): 15.0 not available Technical Assistance L/C Number : L7211 Loan /Credit (US$M): Loan/ US$M ): 10.7 12.0 Sector Board : Social Protection US$M): Cofinancing (US$M ): Cofinanciers : Board Approval Date : 12/18/2003 Closing Date : 09/30/2007 01/31/2013 Sector (s): Other social services (100%) Theme (s): Social risk mitigation (29% - P); Social analysis and monitoring (29% - P); Indigenous peoples (14% - S); Poverty strategy; analysis and monitoring (14% - S); Administrative and civil service reform (14% - S) Prepared by : Reviewed by : ICR Review Group : Coordinator : Hjalte S. A. Sederlof Judyth L. Twigg Christopher D. Gerrard IEGPS2 2. Project Objectives and Components: a. Objectives: The Project Development Objective (PDO) as set out in Schedule 2 of the Loan Agreement was “to support the Borrower’s efforts to implement Chile Solidario and lay the base for a national system of social protection .� The same PDO was included in the PAD. [Note. Chile Solidario is Chile’s version of a conditional cash transfer program . It is targeted at extremely poor households and focuses on integrating participating families into local support networks that include psycho-social support, family coaching, and preferential access to local social services, transfers and benefits .] The PDO was revised in October 2009 when additional financing was introduced . The new PDO, as recorded in Schedule 1 of the new Loan Agreement kept the original PDO, and added the following : "...now including a subsystem for young children ." The project included the following key outcome indicators (they were not changed with the revision of the PDO, nor were new indicators introduced ): • Number of families invited to participate in Chile Solidario • Number of families participating in Chile Solidario • Number of families receiving a monetary transfer (that is, have agreed to work with a family coach ) • Percentage of coverage of social programs and money transfers included in Chile Solidario among the neediest families In addition, the monitoring and evaluation system that was put in place during the project included arrangements for results monitoring that allowed the evaluation of the project impact on beneficiaries . b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives /key associated outcome targets? Yes Date of Board Approval: 10/29/2009 c. Components: Note: Note Neither total project costs nor actual component costs were available for this Review . The listed component costs therefore include estimated project costs at appraisal and actual Bank spending . As noted in the paragraph on project costs and financing, counterpart spending was included in the overall spending of the Government on the Chile Solidario. Component 1: National information system for social protection (estimated project cost at appraisal US$ 4.0 million; actual Bank spending US$ 2.8 million). This component included the design and implementation of a national information system for social protection, including gathering, handling, and management of data on the social and economic characteristics of individuals. In the first phase, the system was to cover all families participating in Chile Solidario. Subsequently, it was to expand to include families participating in all social programs, and finally, cover the total population . Component 2: Training (estimated project cost at appraisal US$ 1.7 million; actual Bank spending US$ 1.6 million). There were two training sub-components. The first covered training of government officials at central and local levels, including from the Ministry of Planning and Cooperation (MIDEPLAN). Training was to include activities to help MIDEPLAN progress in the design of a system of social protection, based on the initial experience with Chile Solidario, and was to support the implementation of the national information system for social protection . The second sub-component was to strengthen the capacity of social workers and others who work with the families of Chile Solidario. Component 3: System for analysis, evaluation and monitoring of social programs and policies (estimated project cost at appraisal US$ 2.8 million; actual Bank spending US$ 2.3 million). This component was to strengthen capacity in the public sector to measure the impact of Chile Solidario and its associated programs. It also included monitoring that would permit the adjustment of actions as implementation proceeded. The component was to include: (a) the design of an integrated system to monitor the management, processes, and partial results of Chile Solidario; (b) the evaluation of the impact of selected programs of Chile Solidario in addressing the needs of beneficiaries of Chile Solidario; and (c) the carrying out of an impact evaluation of Chile Solidario in its entirety. The design and implementation of a pilot project of citizen accountability was to be supported under the component . The pilot would involve design and implementation of a process of consultations with community organizations in selected areas to assess the implementation of Chile Solidario. The information from this consultative process was to provide evaluation inputs and help to detect errors and weaknesses. Component 4: Institutional Strengthening of MIDEPLAN and project administration (estimated project cost at appraisal US$ 6.4 million; actual Bank spending US$ 5.2 million). This component was to strengthen MIDEPLAN ’s role as coordinator of social protection policy and programs in Chile. In order for MIDEPLAN to be able to carry out its designated role, the component would support institutional modernization and strengthening of the ministry . This was to include two types of activities : the first was the installation of several operating units in MIDEPLAN to support the operation of Chile Solidario. These units would be responsible for : (i) managing analytical work which would address issues and questions as they arose during implementation as more families were covered, (ii) organizing training; (iii) supporting the executive secretariat for Chile Solidario; (iv) coordinating the integration of information sources; (v) monitoring the progress of the program; and (vi) handling the day-to-day administration of the technical assistance loan . Several studies on issues related to indigenous families as well as other aspects of Chile Solidario were planned. With the introduction of additional financing, activities were introduced to cover the extension of the social protection system to cover young children, Chile Crece Contigo, CCC , and provide further support to the development of Chile Solidario. Activities included staff training to implement CCC, the development of supporting materials, and training to carry out modifications in the Chile Solidario. These were introduced in components 2, 3 and 4. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project cost and financing. Total project cost at appraisal was US$ 15 million, to be financed by a Bank loan of US$ 10.7 million and a Borrower contribution of US$ 4.3 million. The Bank approved US$ 3 million in additional financing in October 2009. US$ 1.4 million of the additional financing was subsequently cancelled at the request of the Government, and US$ 0.3 million remained unspent at the end of the project and was also cancelled . Ultimately, actual Bank financing totaled US$ 12 million. The two cancellations were made to avoid an extension of the project period, as implementation had been delayed due to an earthquake . Instead, necessary expenditures were covered by the Government budget as part of its overall US$ 27.3 million investment in Chile Solidario during the project period . There is no information available on total project cost, or on the actual costs of project components. Dates. The project was approved on December 18, 2003. The original Closing Date, September 30, 2007, was extended on September 29, 2009, to December 31, 2009, to allow time for the approval of additional financing . Additional financing was approved on October 29, 2009, with a Closing Date of August 31, 2011. This Closing Date was extended twice, first to December 31, 2012, and then to January 31, 2013. The initial extension was due to the emergencies relating to an earthquake, and the second to allow completion of an assessment . 3. Relevance of Objectives & Design: a. Relevance of Objectives: Relevance of objectives is rated high. high While Chile had experienced a gradual decrease in poverty over the decade preceding the project, extreme poverty had stagnated at a level of 6 percent since the mid-1990s. In May, 2002, the Government announced the Chile Solidario initiative, a set of legislative and institutional reforms to bring Chile ’s poorest households out of indigence poverty. It had already secured Bank financing for the reform process through a Social Protection Adjustment Loan and was now also requesting a technical assistance loan to ensure adequate advice on specific reforms. Additional financing was subsequently requested (and approved by the Bank ) that aimed at refining Chile Solidario interventions to better target young children . The technical assistance loan, including Additional Financing, is also in line with the continuing focus in the current Country Partnership Strategy (CPS, 2011-2014) to help eradicate extreme poverty by 2014. Notably, the current CPS seeks to bring to closure initiatives launched under the project under review . b. Relevance of Design: Relevance of design is rated substantial . Design was directly relevant to the PDO . Project activities aimed at operationalizing the Chile Solidario and laying the basis for a coordinated national system of social protection . It was expected that this would raise the standards of living of beneficiary households and also increase coverage of the extremely poor by other social programs. Other groups were also expected to benefit from improvements in social protection under the project . To monitor and evaluate such changes, the design included developing an integrated social information system for more effective targeting; a project monitoring and evaluation system; and institutional capacity building . The results framework provided a straightforward causal chain from spending to institutional improvements, including output indicators, as well as indicators to measure changes in coverage of very poor households by Chile Solidario. Surprisingly, the results framework did not draw on elaborate monitoring and evaluation arrangements for Chile Solidario and other social protection programs to measure effects of the project on the well -being of beneficiary households. 4. Achievement of Objectives (Efficacy): Objective : To support the Borrower ’s efforts to implement Chile Solidario and lay the basis for a national system of social protection, now including a subsystem for young children . Efficacy is rated substantial . Outputs An integrated social information system was developed, drawing together 60 databases from 46 institutions, and covering 80 percent of the Chilean population . Central and regional MIDEPLAN officials involved in coordinating Chile Solidario were trained to respond to evolving poverty alleviation policies . MIDEPLAN was reorganized at the central level to strengthen management and administration of Chile Solidario; and regional offices of MIDEPLAN were given the tools to support field level implementation of the program. A mechanism for coordinating Chile Solidario and other social programs was set up in MIDEPLAN . Case workers functioning as family coaches were trained in effective counseling skills and knowledge about available social services and programs . 11 studies were carried out to evaluate processes and procedures in Chile Solidario programs, and 3 panel surveys and a series of studies were undertaken for impact evaluation of those programs . Outcomes Through strengthening core business processes in Chile Solidario -- management, information and service delivery tools ; an integrated social information system, and arrangements for better coordination of social protection programs -- the project helped move the safety net from a set of isolated interventions towards a harmonized national system of social protection . Accompanying these institutional results, initial impact evaluations for the period 2003-2006 (drawing on findings from the M&E arrangements) show the following results: Of 650,000 eligible families invited to participate in Chile Solidario, 585,000 chose to participate and receive cash transfers, compared to a baseline of 91,500 and an original target value of 220,000 households; Improved participation in education and health programs : (i) higher pre-school and school enrollments – 4 to 6 percentage points and 7-9 percentage points, respectively; and higher take -up of adult literacy programs (4-5 percentage points); and (ii) increased enrollment in public health services (2-3 percentage points) and increased preventive health visits for children under 6 (4-6 percentage points) and for women (6-7 percentage points). Suggestive evidence that psycho -social support had increased awareness of social services; There is little information available about results such as decline in extreme poverty or changes in the quality of life of indigent families, which were listed as ultimate objectives of Chile Solidario (PAD, page 28). However, such outcomes mainly are of a longer term nature – a function of the use of cash transfers to build assets and raise consumption. 5. Efficiency: No estimates of rate of return were made at appraisal or in the ICR . Still, the project contributed to raising the efficiency and effectiveness of Bank engagement in social protection in Chile, as it provided timely technical assistance for implementing commitments made by the Government in the parallel Social Protection Adjustment Loan. All aspects of project design are likely to have contributed to increased efficiency in the social protection system, by focusing resources on key features that could strengthen the functioning of the system : better targeting and reduced program overlap, better program analysis, and better monitoring and evaluation mechanisms. Implementation took a long time, ten years, in part reflecting inefficiencies in start -up arrangements, but also adjustments to the project that are likely to have strengthened its impact (focus on children), and the effects of a major earthquake in 2010. Overall, money was spent on the right things . Efficiency is rated substantial . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The relevance of the PDO is rated high : it addressed an issue that was a government priority and remains so today. The relevance of design is rated substantial : the design responded to the PDO, and, while not well reflected in the results framework, at the same time it addressed underlying welfare objectives through the arrangements for monitoring and evaluation . Efficacy is rated substantial, drawing on still incomplete evaluation results of behavior changes and outcomes on the target population, some of which will show up only in the longer term. Efficiency is rated substantial, reflecting the twinning of the project with a parallel DPL, raising the efficiency of both projects, and the project ’s spending on key elements for a more efficient social protection system. a. Outcome Rating : Satisfactory 7. Rationale for Risk to Development Outcome Rating: The institutions put in place under the project continue to be central to government social policy . MIDEPLAN has been transformed into a Ministry of Social Development and at the same time, the emphasis has shifted towards execution rather than coordination, as was initially the approach adopted under the project . The integrated information system and the increased family counseling emphasis of the Chile Solidario, which were some of the main achievements under the project, remain central features of social protection . a. Risk to Development Outcome Rating : Negligible to Low 8. Assessment of Bank Performance: a. Quality at entry: The project supported initiatives that were highly relevant to the Government and the Bank : reducing extreme poverty and strengthening social protection . Design built on extensive sector analysis, close collaboration with government, and extensive consultation with local authorities and civil society . While focusing on strengthening the government ’s social assistance scheme, Chile Solidario, design also included measures to harmonize social protection programs provided by other sectors and agencies with Chile Solidario. Design took advantage of the favorable country environment to successfully integrate procurement and financial management functions into MIDEPLAN . Safeguard needs for indigenous peoples were identified. Risks were correctly assessed as minimal . The results framework was internally consistent, but it lacked focus on outcomes for the beneficiary population, failing to draw on information that was built into the design of monitoring and evaluation arrangements . at -Entry Rating : Quality -at- Moderately Satisfactory b. Quality of supervision: No significant supervision problems arose during project implementation . Extension of the project implementation period was mainly caused by the introduction of additional financing and a major earthquake in 2010. The Bank team focused on providing technical support as needed and on monitoring implementation. Technical support was successful in most instances, but appears to have fallen somewhat short in information technology . The Bank team was able to avoid potential disruptions in implementation due to numerous changes in key counterparts and institutional arrangements as the new structures for social protection were developed. Fiduciary staff visited regularly, and there was adequate follow -up on the safeguard policy triggered for indigenous people . Performance reporting through implementation status reports was thorough. Quality of Supervision Rating : Satisfactory Overall Bank Performance Rating : Moderately Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: The government was strongly committed to achieving the development objectives, and continues to be so . Several studies over the period preceding the introduction of Chile Solidario pointed to the need for a more pro-active system to avoid social exclusion of the very poor . Appropriate laws were put into place in 2004 and 2009 to support Chile Solidario and an integrated system of social protection . Budget resources were allocated to the programs, tied to achievement of specific coverage targets, and a certain minimum amount of financing was guaranteed . Government Performance Rating Satisfactory b. Implementing Agency Performance: MIDEPLAN's commitment to the development objective was initially determined by the central role it was given in social protection, and after that as it became the Ministry of Social Development . Some problems of coordination between MIDEPLAN and the agency in charge of family coaching under the Chile Solidario were overcome. Implementation arrangements and staffing were generally satisfactory, albeit with some changes in personnel over the implementation period . There were no major implementation issues . Financial management performance was adequate, procurement less so, but without significant problems . Monitoring and evaluation were best practice . A new administration came into place in 2010, adjusting some of the priorities for the use of additional financing to focus on high priority areas . This led to the cancellation of part of the additional financing, Implementing Agency Performance Rating : Satisfactory Overall Borrower Performance Rating : Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: Arrangements for results monitoring included : monitoring of all activities supported by the project; an impact evaluation of the Chile Solidario; and arrangements for an integrated social information system . For project activities, measurable output and intermediate outcome indicators were identified, reflecting the design of the results framework. They did not, however, include outcome indicators addressing the impact on beneficiaries of project interventions. These were to be addressed through the Chile Solidario -related evaluation that would draw on a survey instrument that allowed sample families and a control group to be compared . The design would also include complementary studies of the interventions on participant families . Finally, a set of studies was to monitor changes in the management of public services at the local level after the introduction of Chile Solidario . In addition to project-related monitoring and evaluation, M&E design included the development of the integrated social information system. b. M&E Implementation: Monitoring of project activities was implemented as planned, and three surveys on the impact of Chile Solidario were carried out. The integrated social information system was put into place . c. M&E Utilization: The introduction of the integrated social information system has encouraged ministries and agencies to use information in the system to justify budgets and facilitate assessment of the use of public resources . It has led to participating agencies sharing annual goals through formal inter -institutional agreements using the information system as a joint targeting instrument . The information from the system is also used to monitor performance of municipalities and family coaches, and progress of families in the system . M&E Quality Rating : Substantial 11. Other Issues a. Safeguards: There was satisfactory compliance with safeguard policies . The project triggered the safeguard policy on Indigenous Peoples. An Indigenous Peoples Development Plan was prepared with input from experts in the field. Main activities included incorporation of questions in the household survey and in targeting instruments to identify indigenous families and beneficiaries of Chile Solidario, leading to adjustments in family level interventions to make them culturally appropriate . b. Fiduciary Compliance: Financial management. Financial management was integrated with the MIDEPLAN financial management system, and project related procedures were defined in the financial section of the project operating manual . The financial management assessment preceding the project rated the financial management risk as low . Project audits were issued on time and were unqualified . Procurement. There were minor weaknesses in procurement . This was initially due to unfamiliarity with Bank procedures, and subsequently reflected frequent changes in staffing . c. Unintended Impacts (positive or negative): None reported. d. Other: 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Satisfactory Satisfactory Risk to Development Negligible to Low Negligible to Low Outcome : Bank Performance : Satisfactory Moderately The IEG rating of moderately Satisfactory satisfactory was influenced by quality at entry. While many elements were satisfactory at QAE, the lower rating was influenced by the results framework, which lacked focus on outcomes of the project for the beneficiary population. Borrower Performance : Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES NOTES: - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: Lessons drawn from the ICR: The experience with Chile Solidario illustrates that a single initiative or program can serve as a practical and concrete base for the installation of broader social protection systems that can be extended to cover other populations (ICR, pages 15-16, 38). When considering different policy interventions to reduce poverty and help households manage risks to their income, a comprehensive, integrated information system is critical in order to ensure that social programs reach those who need them the most . Without such a system, social policies and interventions cannot be effectively targeted, monitored, or evaluated (ICR, pages 15-16, 24-26). High-quality administrative data bases can be an important source of information not only for the monitoring of social programs, but also their impact evaluation, particularly if they can be linked to survey data (ICR, pages 35-37). 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The quality of the evidence and the analysis of designs and results is strong, but unnecessarily extensive, going beyond the direct issues to be addressed . This is especially the case in the analysis of outputs (Annex 2). Lessons are succinct and draw on the experience under the project . The ICR is concerned with outcomes and makes many references to impact analyses, but does not provide much evidence of impact on beneficiaries . The document is internally consistent and consistent with OPCS guidelines with the following exception that it does not provide an adequate picture of project costs and financing . It is not possible to determine total project costs or actual spending on components with the information provided . At 21 pages, the ICR is reasonably concise, although the actual analysis of results is confined to a 16-page annex (Annex 2). a.Quality of ICR Rating : Satisfactory