Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CL Inclusion for Shared Prosperity DPL(P154213) Report Number : ICRR0020888 1. Project Data Operation ID Operation Name P154213 CL Inclusion for Shared Prosperity DPL Country Practice Area(Lead) Chile Poverty and Equity L/C/TF Number(s) Closing Date (Original) Total Financing (USD) IBRD-85510 30-Dec-2016 100,000,000.00 Bank Approval Date Closing Date (Actual) 16-Nov-2015 30-Dec-2016 IBRD/IDA (USD) Co-financing (USD) Original Commitment 100,000,000.00 0.00 Revised Commitment 100,000,000.00 0.00 Actual 100,000,000.00 0.00 Prepared by Reviewed by ICR Review Coordinator Group Judyth L. Twigg Clay Wescott Malathi S. Jayawickrama IEGEC (Unit 1) 2. Project Objectives and Policy Areas a. Objectives According to the Letter of Development Policy and the Program Document (p. 12), the objectives were "to support: (a) promoting equal opportunities in education; (b) improving the conditions to enhance the quality of tertiary education; and (c) strengthening institutions for poverty measurement and enhancing targeting mechanisms of social programs." b. Were the program objectives/key associated outcome targets revised during implementation Page 1 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CL Inclusion for Shared Prosperity DPL(P154213) of the series? --- c. Pillars/Policy Areas The Program supported three Pillars (Program Document, pp. 14-23): 1. Promoting equal opportunities in education, by: (1) supporting Chile's comprehensive 2014 Education Reform designed to improve equitable access to high-quality primary and secondary education, including policy initiatives reflected in Inclusion Law No. 20845 to regulate the admission process, eliminate co- payments, eliminate profits in establishments that receive State funding, and improve quality by increasing public resources for education; (2) supporting the Government's Program for Effective Access to Higher Education (PACE), created by Ministerial Resolution 680 and designed to increase the number of highly vulnerable students making the transition from secondary education to tertiary education; and (3) supporting Government efforts to improve equitable access to tertiary education by expanding scholarships for students first in the five bottom deciles of income distribution (budgeted in the 2015 Public Sector Budget Law), to be expanded in 2018 to the seven lowest income deciles. 2. Improving the conditions to enhance the quality of tertiary education, through the 2015 Convenio Marco de Universidades Estatales, launched in 2015 as a network of all 16 public universities to harmonize and strengthen quality across regions through increased funding for institutional strengthening, research and innovation, teacher development, and outreach activities. A new accreditation system was to be developed, with a compulsory minimum entrance level based on quality standards for programs and institutions. 3. Strengthening institutions for poverty measurement and enhancing targeting mechanisms for social programs, by: (1) supporting Government efforts to improve the measurement and monitoring of poverty (Oficio No. 888, March 2015) by releasing and implementing a revised official methodology for measuring monetary poverty, and launching an official Multidimensional Poverty Index (MPI); and (2) supporting implementation of a new allocation model for benefits and social services (Oficio No. 1244, March 2015) that would move from largely self-reported targeting to targeting based on objective characteristics using administrative data. d. Comments on Program Cost, Financing, and Dates The Program was proposed as a single-tranche, US$ 100 million Social Inclusion for Shared Prosperity (SISP) Development Policy Financing (DPF). The full US$ 100 million was released on August 18, 2016, four months later than the planned April 2016; the ICR does not provide an explanation for the delay. Closing was as scheduled on December 31, 2016. 3. Relevance of Objectives & Design a. Relevance of Objectives Page 2 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CL Inclusion for Shared Prosperity DPL(P154213) At appraisal, despite a long period of strong and sustained economic growth, Chile lagged the Organization for Economic Cooperation and Development (OECD) country average in key education outcomes. Its public expenditure on tertiary education was among the lowest in the OECD countries. The resulting large share of tertiary education funding from tuition and fees produced a heavy financial burden for families. This limited access to quality education translated into inequality in the labor market, and firms faced difficulty hiring workers with appropriate skills. Chile also faced challenges stemming from a poverty measurement that was based solely on income, not addressing other deprivations in access to electricity, water, education, and labor markets. As monetary poverty reduction was achieved, the multidimensional aspects of deprivation became more important for the Government, as did movement away from a benefits targeting system that relied mostly on self-reporting and contained numerous inclusion and exclusion errors. Responding to this country context, the Government introduced a 2014-2018 development program, "Chile de Todos," prioritizing three key reforms: education, tax reform, and a new Constitution, all centrally aimed at strengthening citizens' rights. This operation's first two objectives directly supported education reform, and the third aligned more broadly with citizens' rights by improving poverty measurement and enhancing social protection systems. The Bank's FY 2011-2016 Country Partnership Strategy (effective at project closure) similarly focused on three key areas: public sector modernization; job creation and equity improvement; and promoting sustainable investment. Quality, equity, and financing of tertiary education were among its main priorities. The objectives of this development policy operation are therefore responsive to these Government and Bank priorities. Rating High b. Relevance of Design The single-tranche operation had a clear statement of objectives. It was appropriately chosen to give a push for reforms initiated by the Government, give time for Government to implement the reforms, provide flexibility for design of future operations, and ensure complementarity with other instruments under implementation. The activities to be implemented under the first pillar -- the Inclusion Law (which aims to eliminate segregation of students by socioeconomic level and provide an integrated education system), PACE, and full scholarships for students from poor backgrounds -- were logically and plausibly connected to expected outcomes related to promotion of equal opportunities in education. Increasing budget allocations to state universities under Convenio Marco, under the second pillar, was a plausible first step toward enhancing the quality of tertiary education, though the full impact of these initial investments would not reasonably be observed for several years. However, it is important to note that improving the conditions for enhancing the quality of education is not the same as actual learning; acknowledging that it takes time for investments to produce improvements in learning outcomes, student learning is noticeably absent from the statement of objectives. Under the third pillar, the adoption of the new multi-dimensional poverty methodology, together with the new enhanced targeting mechanisms, were strongly and logically linked to achievement of strengthening institutions for poverty measurement and enhancing targeting mechanisms. Page 3 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CL Inclusion for Shared Prosperity DPL(P154213) Rating Substantial 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective Promoting equal opportunities in education Rationale Efficacy is assessed against three outcome indicators: (i) number of students who receive the Preferential School Subsidy (SEP); (ii) number of students in secondary school incorporated in the PACE program; and (iii) number of full scholarship recipients that belong to the five lowest income deciles enrolled in their first year of tertiary education. Although Chile had universal primary education coverage, and over 90 percent coverage in secondary education, enrollment in pre-school and tertiary education were lower. The lowest income groups also had a higher concentration of incomplete schooling (PAD, p. 10). The SEP, the PACE program and scholarships were expected to more equal access to education, especially for the lowest income groups. As prior actions, the government enacted the Inclusion Law (prior action 1), created the program for PACE (prior action 2); and expanded scholarships for students that belong to the seven lowest income deciles to promote equitable access for tertiary education. The following built on the prior actions: The allocated nominal budget for the Ministry of Education increased by 6% in 2014, 13% in 2015, and 16% in 2016. A SEP program was implemented in all regions of Chile, establishing a new targeted voucher that gives more resources to schools for each eligible student enrolled. Eligible students were defined by several mechanisms that drew roughly from the poorest 40% of the income distribution. Effectively, the SEP eliminated tuition fees for vulnerable students in most private-subsidized schools. Under the Inclusion Law, SEP funds were tied to elimination of co-payments and for-profit motives in these schools, with 887 schools applying to be transferred from profit to non-profit status. The number of students receiving the Preferential School Subsidy increased from 1.25 million in 2015 to 1.9 million in 2016, exceeding the target of 1.63 million. Out of total enrollment of 3.26 million, over 58% of students now receive the subsidy. The PACE program was rolled out, with per capita annual costs estimated at 150,000 pesos for preparation during secondary school and one million pesos for support at the tertiary level. The number of secondary school students enrolled in the PACE program increased from 7,614 in 2015 to 74,562 in 2016, exceeding the target of 57,000. The program now runs in all 15 regions of the country. Questionnaires and focus groups administered during the program's first-year evaluation suggest that it was well targeted, and that it opened pathways to tertiary education for students from poor backgrounds who would not otherwise have had these opportunities. The results also revealed challenges related to information flows concerning the program, clarification of roles and responsibilities of different parties, and accrual of benefits to best- performing students with less attention to those lagging behind; according to the ICR (p. 24), some aspects of these critiques have been incorporated into program design revisions. Page 4 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CL Inclusion for Shared Prosperity DPL(P154213) The government scholarship program for students in lower income deciles was implemented. The number of full scholarship recipients enrolled in their first year of tertiary education who are from the lowest five income deciles increased from 2,479 in 2015 to 48,555 in 2016, exceeding the target of 30,000. Students with full scholarships for all years of study constituted 15.6% of the student body in 2016. Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 2 Objective Improving the conditions to enhance the quality of tertiary education Rationale Efficacy is assessed against one indicator: the number of students enrolled in state universities included in the Convenio Marco. As a prior action, the government provided resources to state universities for several areas to improve the conditions to enhance quality tertiary education. The Government provided funds for quality improvements through the Convenio Marco. 16 public universities applied for these funds and initiated actions to use them appropriately, reporting periodically to the Ministry of Education. As a result, the number of students enrolled in state universities included in the Convenio Marco increased from zero in 2015 to 178,000 in 2016, almost reaching the target of 180,000. The gap in reaching the target was due to a new 2.6% cap on annual enrollment increases at universities that receive full scholarship students; this cap was not foreseen during project design. Although universities received additional budget allocations as a result of this increased enrollment, and these funds supported teacher training (facilitated by passage of a new 2016 Law on teacher professional development; ICR, p. 27), it is not yet clear precisely how these funds are translating tangibly into improved conditions for enhanced quality of tertiary education. The ICR does not report on development of a planned new accreditation system. The Program Document (pp. 18-19) sets out two key aspects of improving conditions for quality enhancement. Achievement of the first, increasing public investments, has been demonstrated. Achievement of the second – deep reform around compliance with quality standards, institutional arrangements to guarantee effectiveness of the system, and permanent supervision of the system, all embedded in a new accreditation mechanism – has not. This is indicative of Modest achievement of this objective. Rating Modest PHREVDELTBL Page 5 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CL Inclusion for Shared Prosperity DPL(P154213) PHEFFICACYTBL Objective 3 Objective Strengthening institutions for poverty measurement and enhancing targeting mechanisms of social programs Rationale Efficacy is assessed against two outcome indicators: (I) Release of monetary and multidimensional poverty indicators based on the new methodology and its corresponding microdata with the release of the 2013 and 2015 household survey (CASEN); and (ii) number of social programs and subsidies whose eligible beneficiaries are determined by the new targeting system. There were two prior actions related to enhancing the measurement and monitoring of poverty by implementing the release of several laws and methodologies; and designing a new allocation model for benefits and social services to improve targeting for social programs. The Ministry of Social Development released new monetary and multidimensional poverty indicators, and has strengthened the multidimensional poverty indicators by incorporating additional dimensions (environmental pollution, travel time and distance to public services, social participation, unfair treatment/discrimination, security), based on microdata from 2013 and 2015 household surveys. The microdata are released for public use. As a result, 51 social programs and subsidies now determine eligible beneficiaries using a new targeting system based on these indicators, exceeding the target of 10. According to the ICR (p. 28), social targeting mechanisms now benefit from increased transparency, improved information quality, and larger population coverage through the use of administrative data. The average number of days to process a claim was reduced from 25 days in January 2016 to 6 days in December 2016. A Household Social Registry was established, enabling a shift away from the use of self-reported data toward a combination of social protection records, social records, and administrative data. A communication campaign through television, radio, press, and social networks was undertaken to support the new Registry. The new records system covers 12.7 million individuals and 4.7 million households representing 72% of the Chilean population, compared to 46% coverage under the old system. Exclusion error of the bottom 40% of the income distribution is less than 5%. A draft Law on the poverty line was being debated in Parliament at the time the ICR was written. The project team confirmed that this draft Law remains under discussion. Rating High PHREVDELTBL PHREVISEDTBL 5. Outcome Page 6 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CL Inclusion for Shared Prosperity DPL(P154213) The objectives were highly relevant to country conditions, Government strategy, and Bank strategy. The operation's pillars were substantially aligned with its objectives, though improvements in the conditions for education quality enhancement are observable only in the longer term. The objective to strengthen institutions for poverty measurement and enhancing targeting of social programs was highly achieved, with important gains from the new household registry and multidimensional poverty index. The objective to promote equal opportunities in education was substantially achieved through the successful implementation and scale-up of the SEP, PACE, and government scholarship programs, all effectively targeted at lower-income students. The objective to improve conditions to enhance quality of tertiary education, however, was only modestly achieved, as information is not presented on the ways in which increased financing of the Convenio Marco program resulted in improved learning conditions. Taken together, these ratings are indicative of minor shortcomings in the operation's preparation and implementation, and therefore the Outcome rating is Satisfactory. a. Outcome Rating Satisfactory 6. Rationale for Risk to Development Outcome Rating The Government remains committed to the reform agenda. The operation's contributions to institutional development -- putting in place stronger and more equitable institutions to facilitate access to quality education, building a network of state universities, and strengthening the transparency and multidimensionality of poverty measurement -- are likely to be sustained. However, moderate risks stem from fiscal sustainability (the success of ongoing fiscal reforms, coupled with fiscal pressures that may result from demographic change), as well as political risk emerging from presidential elections in November 2017. The project team stated that there is broad public support for the reforms, and agreement across the political spectrum that inequality of opportunity is the key challenge, but difference on the means toward addressing that challenge. Unexpectedly high demand for full university scholarships has created pressure on both the fiscal and supply sides, only partially alleviated through the 2.6% cap on annual enrollment increases at universities receiving government financing. In addition, improvements in access and learning conditions may not translate into improved learning outcomes in the absence of sustained attention to measuring outcomes, with targets. a. Risk to Development Outcome Rating Modest 7. Assessment of Bank Performance a. Quality-at-Entry A Joint Studies Program, launched in 2008 and operated under different governments across the political spectrum, has been the key instrument of cooperation between Chile and the Bank in undertaking relevant, Page 7 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CL Inclusion for Shared Prosperity DPL(P154213) targeted, and timely analytic work to ground various initiatives. That program's analytical work on tertiary education, tax reform, and social protection served as the underpinning for reforms supported by this operation. Policies also built on existing Bank Reimbursable Advisory Service operations (Tertiary Education Quality Assurance and Social Information System Redesign) as well as the Third Tertiary Education Finance for Results Project (US$ 40 million, 2012-2016). There was active collaboration between different Global Practices to ensure that the operation's equity perspective was well informed across sectors (ICR, p. 36). The Bank also worked closely with the Inter-American Development Bank (IDB) to ensure focus on areas of comparative advantage (the IDB in early childhood education, and the Bank in tertiary education). The main identified risk was low take-up of reforms by the public, mitigated through a wide national consultation with an array of stakeholders (academics, teacher unions, the private sector, student groups). There could have been more effort to spell out the timetable and causal links between the measures taken in the operation and eventual improvement in learning outcomes. Quality-at-Entry Rating Satisfactory b. Quality of supervision Only one supervision mission was conducted during the project period, appropriate for a single-tranche operation. The Bank team maintained a productive working relationship with the Ministries of Education and Social Development, as well as an active dialogue with the Government on issues of poverty and equity. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory 8. Assessment of Borrower Performance a. Government Performance According to the ICR (p. 37), the reforms covered by this operation remain a "top priority" for the Government. The Government has continued to support gradual implementation of reforms with larger budgetary allocations, even as economic growth in Chile has recently slowed. The Government made effective use of the operation's monitoring and evaluation system, fine-tuning implementation where necessary. Government Performance Rating Satisfactory b. Implementing Agency Performance Two ministries, Education and Social Development, were responsible for implementation. The ICR (p. 37) Page 8 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CL Inclusion for Shared Prosperity DPL(P154213) states that each ministry coordinated implementation effectively in its specific areas, placing high priority on effective deployment and use of monitoring and evaluation systems. Implementing Agency Performance Rating Satisfactory Overall Borrower Performance Rating Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design Six key results indicators, one corresponding to each policy action, were identified to monitor progress. These indicators, to be collected using administrative data of the Ministries of Education and Social Development, were also used internally by the Government. The formal indicators did not capture the outcomes of the policy actions on student learning, as improvements in such outcomes may be difficult to observe over a short time period. Several studies were also planned to assess the poverty and social impacts of the operation. b. M&E Implementation Data were routinely collected and reported as part of the internal monitoring and evaluation of the Ministries of Education and Social Development. Most of the initiatives supported by the operation were piloted first in small geographic areas and a select number of schools, with analysis and lessons from the pilots incorporated into the larger reform roll-out. M&E data were of acceptable quality and reliability, and M&E results and analysis contributed to streamlining processes and building institutional capacity. c. M&E Utilization M&E data and analysis were used for progress monitoring and decision making by the Ministries of Education and Social Development on a regular basis. Gradual roll-out of reforms was informed by ongoing assessment. For example, the 2.6% annual increase cap on new tertiary enrollment for students with full scholarships was put in place based on high revealed demand, to avoid overstretching the tertiary education system. M&E Quality Rating Substantial Page 9 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CL Inclusion for Shared Prosperity DPL(P154213) 10. Other Issues a. Environmental and Social Effects No safeguard policies were triggered. According to the Program Document (p. 25), "none of the elements under the proposed operation are likely to have adverse effects on the environment, forests and other natural resources. The operation does not support actions linked to the construction or rehabilitation of schools, or other civil works that would likely impact the environment." The ICR (pp. 31-34) undertakes detailed analysis demonstrating short-run and long-term positive impact on income inequality from increased resources to public education; a 9% increase in the probability of a student aged 15-19 years completing secondary education due to the elimination of co-payments; and reduction in stratification across Chilean schools due to the regulation of admissions processes in schools receiving public funding. Data from the first year of implementation of the scholarship program indicate that women were more likely to benefit than men, with women being the majority (55%) of recipients of full scholarships. b. Fiduciary Compliance The ICR does not provide information on fiduciary compliance. No financial issues with the operation or audits are cited. The project team confirmed that the project relied on country financial management systems. Audit was performed by the Supreme Audit Institution of Chile. c. Unintended impacts (Positive or Negative) None reported. d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory --- The ICR gives contradictory ratings for Risk to Risk to Development Modest Modest Development Outcome: Outcome Moderate (p. 36) and Low (p. 5). The project team confirms Page 10 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CL Inclusion for Shared Prosperity DPL(P154213) that the intended rating is Modest. Bank Performance Satisfactory Satisfactory --- Borrower Performance Satisfactory Satisfactory --- Quality of ICR Substantial --- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 12. Lessons The ICR (pp. 37-38) provides several useful and insightful lessons, including: Deliberate consensus-building can be key, even for seemingly less controversial reforms. In this case, a careful process of bringing stakeholders together in dialogue across the ideological spectrum built support for the full scholarship program, and also facilitated agreement on implementation details. The Bank can add important value in high-income countries. In this case, sustained analytic work brought the Bank's vast expertise on poverty, vulnerability, and equity measurement to bear on a long-term reform agenda. IEG adds the following lesson: Although improvement in learning outcomes cannot reasonably be expected in the time frame of a development policy operation, improved learning can still be acknowledged as a longer-term outcome, with a plan for addressing it with targets. In this case, the results chain linking outputs to increases in quality of education to the implicit goal of improved learning outcomes was not clear. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR is clear and concise, following established guidelines. It presents evidence beyond the project's formal results framework to assess achievement of the operation's objectives. It conducts extensive and useful analysis on the operation's likely long-term gender and poverty impact. It is lengthy, at 38 pages, and occasionally repetitive. Page 11 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CL Inclusion for Shared Prosperity DPL(P154213) a. Quality of ICR Rating Substantial Page 12 of 12