Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Strength. Serv. Delivery Ceará PforR (P127463) Report Number : ICRR0021875 1. Project Data Project ID Project Name P127463 BR Strength. Serv. Delivery Ceará PforR Country Practice Area(Lead) Brazil Finance, Competitiveness and Innovation L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-83020 31-Jan-2018 346,513,418.75 Bank Approval Date Closing Date (Actual) 21-Nov-2013 31-Jan-2019 IBRD/IDA (USD) Grants (USD) Original Commitment 350,000,000.00 0.00 Revised Commitment 346,513,418.75 0.00 Actual 346,513,418.75 0.00 Prepared by Reviewed by ICR Review Coordinator Group Ranga Rajan Christopher David Christopher David Nelson IEGFP (Unit 3) Krishnamani Nelson 2. Project Objectives and Components a. Objectives This operation was financed through two Bank instruments: The Program-for-Results (PforR) and the Investment Project Financing (IPF) instruments. The Project Development Objective (PDO) as stated in the Loan Agreement (Schedule 1, page 7) and the Project Appraisal Document (PAD, page vii): "To support the Government to improve public service delivery particularly in the areas of skills development, family assistance and water quality". Page 1 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Strength. Serv. Delivery Ceará PforR (P127463) This ICR Review will assess the achievement of the operation's goal of improving service delivery, in terms of three specific objectives. • Skills development. • Family assistance. • Water quality. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 10-Mar-2017 c. Components The PforR instrument supported three subprograms of Ceara State's Multi-Year Plan and the IPF instrument supported a technical assistance component (PAD, pages 8 and 9). 1. Skills development. This subprogram supported the program on vocational training, through expanding professional training schools, supporting research and providing scholarships to selected students. 2. Family Assistance. This subprogram supported the expansion of the Integrated Family Assistance Program, administered by the Social Assistance Research Centers in selected municipalities. 3. Water Quality. This subprogram supported the water management program aimed at improving water quality, through a water security plan for the watersheds, expanding household connections to existing sewerage networks in the metropolitan region of Fortaleza and in the Acarau and Salgado basins, implementing solid waste management plans and strengthening the hydro-environmental prediction and monitoring systems. 4. Technical Assistance (TA). This component provided technical assistance for: (i) cross-cutting public sector management functions (results-based management and capacity building of the Ceara Economic Research Institute); and (ii) sector-specific activities in the areas of skills development, family assistance and water services. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates Program cost. The estimated cost of the operation at appraisal was US$350.00 million. The actual cost was US$346.51 million. Page 2 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Strength. Serv. Delivery Ceará PforR (P127463) Program financing. The operation was financed by an IBRD loan of US$350.00 million (US$315.00 million through the PforR instrument and US$35.00 million through the IPF instrument). The total amount disbursed US$346.51 million. Borrower contribution. None was planned at appraisal. There was no borrower contribution during implementation. Dates. The operation, approved on November 21, 2013, became effective on December 20, 2013, and was originally scheduled to close on January 31, 2018. The program closed a year later on January 31, 2019. Other changes. There were two Level 2 restructurings. These changes were made through the first restructuring on March 10, 2017, after the Mid-Term Review on April 11, 2016. • The targets for Disbursement-Linked Indicators (DLIs), associated with the family assistance program and the public investment program, were reduced: The target for the families receiving family assistance was reduced by half from 15% to 7% in year three, and from 25% to 10% in year four, due to reduced federal transfers to municipalities, in the wake of the economic crisis of 2015 and 2016. The target for the percentage of public investments prepared using a results-based methodology was reduced from 25% to 15% in year 3.5, and from 40% to 20% in year four. • Funds (US$10.00 million out of US$35.00 million), were diverted from the IPF component to the PforR component, due to the cancellation of 14 planned TA activities. These changes were made by the second restructuring in October 2017. The closing date was extended by a year for using the remaining TA funds (due to the depreciation of the Brazilian Real relative to the US$ during implementation). Split rating. Given the substantial reduction in targets for the family assistance subprogram, this review is based on a split rating of objectives, when 70% of the loan (US$242.90 million) was disbursed before restructuring and 30% (US$103.61 million) disbursed after restructuring. 3. Relevance of Objectives Rationale Country Context. Despite improvements in economic and social indicators at the national level before appraisal, there were stark disparities between regions in Brazil. Ceara in the North East is one of the poorest states in Brazil, with a per capita income of US$5,236 in 2010, less than half the national average. An acute shortage of human capital has undermined the state's efforts at attracting private sector investment. The state experienced chronic water scarcity with long and severe droughts, and much of the state lacked access to basic water and sanitation facilities (with only 75% and 35% of its population connected to the public water and sewerage systems, as compared to the national averages of 82% and 61% respectively). These aspects underscored the need for improving service delivery through the better management of the public sector. Page 3 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Strength. Serv. Delivery Ceará PforR (P127463) Country Strategy. The program objectives were relevant to the state governments Multi-Year Plan (Plano Pluriannual), for 2012-2015. The plan underscored the need for: (i) promoting equitable social development, through improving service delivery in education and health; (ii) promoting sustainable economic development through public investments in water supply and sanitation services; and (iii) improving public sector management through results-based management. Bank Strategy. The PDOs were well-aligned with the objectives of the Country Partnership Strategy for 2012-2015; (i) increasing productivity of public and private investments; (ii) improving service delivery for low-income households: (iii) promoting regional economic development; and (iv) improving natural resource management. The PDOs are relevant to the Bank's current Country Partnership Framework (CPF) for 2018-2023. The CPF highlighted the need for improving government effectiveness. The focus area 1 of the CPF, highlighted the need for increasing effectiveness of the social protection system (Objective 1.2), improving service delivery of education (Objective 1.3), and addressing the risk of water scarcity in the Northeastern part of Brazil (Objective 3.3). Relevance of PforR instrument for the Bank strategy. The Bank has a history of sub-national level engagement in Brazil. The IPF projects (Rural Sustainable Development and the Ceara Regional Economic Development), supported rural development and urban integration in Ceara. Previous Sector Wide Approaches (SWAp) in Ceara, supported fiscal consolidation, introduced results-based management and expanded coverage of programs in health, education, water, and private sector development. The SWAp operations demonstrated that top-down public expenditure programs are inadequate, in cases where multiple secretariats are involved (for instance, the underperformance of the secondary water quality indicator in a SWAp operation, was due to the presence of pollution in watersheds that fed urban water supplies). This justified an approach that aimed at communicating strategic objectives across multiple secretariats (such as the different state agencies responsible for improving water quality), the agencies of education and health for improving family assistance, and collaboration between public training institutions and private sector for skills development), and allowed for periodic evaluation and self-correction. This operation, using the PforR instrument, evolved from the Bank's experience in Ceara, Minas Gerais, and Pernambuco, aimed at continuing the broad approach used in the SWAps, combining support for sector reforms with capacity building activities. The Bank approach through this instrument was to link disbursements to specific project results. The rationale for using the instrument was strengthened by the state's familiarity with the modalities of disbursements against results-based indicators. Given this alignment with the WB country strategy and the preparedness of the provincial government for performance driven reform, the relevance of the project is Highly rated. Rating High 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Page 4 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Strength. Serv. Delivery Ceará PforR (P127463) Objective 1 Objective To improve skills development Rationale Theory of Change. Developing vocational training programs with the private sector and monitoring such programs, was likely to increase the supply of graduates with vocational skills. The intended outcome of the increase in the number of graduates from these schools absorbed into the productive sector was problematic, given that the demand for graduates was likely to depend on factors, outside the scope of the interventions. Outputs (ICR, page 15 and pages 35-36). • The skills development strategy and related action plans were completed in the municipalities. The plans were by the Secretariat of Education and the Secretariat of Economic Development and Labor, as targeted. • Technical and vocational training facilities were established in 74 of Ceara's 184 municipalities. Over 38,000 students enrolled in these schools since the program began. • The Secretariat of Education and Labor in the municipalities developed a system for monitoring the Vocational Education Training Programs. No reports had been produced by the system to date, as the system became operational only in 2019. • 18 agreements were signed with private companies for developing the curriculum design and guarantee internship opportunities to graduates. Outcomes. • 31,600 technicians from the vocational schools were expected to be absorbed into the productive sector when the operation closed. Only 21,457 technicians were absorbed when the operation closed. The number of graduates employed in the productive sector was not only under the target, it was about 5% lower than the number of graduates (24,543), employed in the productive sector at the baseline in 2013. Some of this was due to the economic downturn, but the premise of linking training to inevitable increases in opportunities was ill conceived in the given context. Thus, achievement is rated as modest. Rating Modest PHREVDELTBL PHINNERREVISEDTBL Objective 1 Revision 1 Revised Objective Page 5 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Strength. Serv. Delivery Ceará PforR (P127463) The targets were not revised during the lifetime of the operation. Revised Rationale No changes to the detail outlined above. Revised Rating Modest PHEFFICACYTBL Objective 2 Objective To improve family assistance. Rationale Theory of Change. Better targeting of family assistance programs and training of relevant staff in the State Assistance and Research Centers, would improve the coverage of family assistance program in the municipalities. Outputs (page 16-18). • The Secretariat of Planning and Management set up a multi-sector advisory committee in 2014, to integrate the activities relating to social assistance, education, health and cross-sector management. • All members of the technical teams in the Social Assistance Reference Centers were trained on issues relating to infant and child education, using manuals developed through the technical assistance component of the project. • An Impact Evaluation was carried out to measure the impact of the pilot educational development program on infants and children in rural areas. • A framework was developed for assessing the impact of family assistance programs. This framework was applied to 96% of family assistance projects. • An independent monitoring system was not implemented as envisioned. However, an internal monitoring system was developed, and this system was used by the 36 participating municipalities. Outcomes. • 11% of families in the targeted municipalities received support from the Social Assistance Reference Centers. This marginally exceeded the revised target of 10%, but was well short of the original target of 25%. As discussed in section 2e, targets for this indicator were reduced, in the face of reduced fiscal transfers to the municipalities. Rating Page 6 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Strength. Serv. Delivery Ceará PforR (P127463) Modest PHREVDELTBL PHINNERREVISEDTBL Objective 2 Revision 1 Revised Objective The objectives were not revised, but the targets were revised downwards. Revised Rationale The revised targets were marginally exceeded. Revised Rating Substantial PHEFFICACYTBL Objective 3 Objective To improve water quality. Rationale Theory of Change. Better coordination of the different state agencies and the private sector involved in undertaking complementary activities in the water sector, together with establishing an environmental quality index, was likely to contribute to improving the quality of water in selected areas. Outputs. (pages 18-19). • An Inter-agency water security committee was established as intended to integrate the complementary actions between the different agencies (discussed in section 3b). • The water security plans for strategic watersheds were completed after the closing date, due to procurement delays. The watershed protection decree was passed by the project closing date. • The solid waste management law was submitted as intended. • An Environmental Quality Index (IQA) was created to monitor water quality. (According to the team, this index widely adopted in Brazil, provides a standardized method for monitoring water quality. The index is based on a scale of 0 (worst) to 100 (best). • The percentage of households with water connections exceeded the final year target of 85.9% in year two, implying that the goals were modest in terms of the level of ambition. It is unclear whether this was a major development limitation or was directed at filling a development gap. • A modeling tool to estimate the quality of water basins was developed to integrate climate forecasting and pollution factors into water resources management, under the technical assistance component of the project. • An integrated management plan was developed in 64 municipalities for activities associated with trash collection and waste site cleanup. Page 7 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Strength. Serv. Delivery Ceará PforR (P127463) Outcomes. • According to the IQA, the quality of raw water improved to 82%, as compared to the target of 67.50%. It is unclear how important this target was in terms of the additional outputs and requirements for the intervention. This appears largely to be a connection issue rather than a coordination challenge and thus limits the performance story on what determines water quality issues. Having said that, the objective is still rated Substantial given achievement against the target. Rating Substantial PHREVDELTBL PHINNERREVISEDTBL Objective 3 Revision 1 Revised Objective The targets for this area were not revised. Revised Rationale Details outlined above apply equally to the revised project. Revised Rating Substantial PHEFFICACYTBL Objective 4 Objective To improve public sector management through technical assistance activities. Rationale Theory of change. Capacity building activities aimed at results-based management would improve public sector management in the areas of skills development, family assistance and water quality. Outputs (pages 19-21). • As noted in section 2e, 14 technical assistance activities were cancelled, and funds allocated to these activities, were diverted for the PforR operation. • As part of technical assistance component, these activities were completed: (i) a new methodology for understanding the economic impact of policies for the Ceara Economic Research Institute (IPECE); (ii) A statistical analysis platform for integrating data from all the secretariats for IPECE; (iii) A new software platform was installed for automatic payroll auditing: (iv). An automated customized system was developed for the Municipal Court of Accounts; and (v) A user friendly platform was developed for engaging citizens in preparing Multi-Year Plans. Page 8 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Strength. Serv. Delivery Ceará PforR (P127463) •Six state agencies adopted the results-based management model when the project closed as targeted. The ICR notes that one year after the project closed, the number of agencies adopting results-based management model increased to 18. Outcomes. • 32% of the public investments were prepared using the results-based methodology when the project closed. This exceeded the revised target of 20% but was short of the original target of 40%. As discussed in section 2e, the targets for this indicator were revised downwards. Rating Modest PHREVDELTBL PHINNERREVISEDTBL Objective 4 Revision 1 Revised Objective The targets for the indicator were revised downwards with the project restructuring. Revised Rationale The revised targets were exceeded. Revised Rating Substantial PHOVRLEFFRATTBL Rationale Overall Efficacy Rating --- 5. Efficiency --- Efficiency Rating --- Page 9 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Strength. Serv. Delivery Ceará PforR (P127463) a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Point value (%) *Coverage/Scope (%) Available? 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Overall relevance is rated as Substantial. Efficacy of the objective, of improving water quality was substantial, as the intended targets were realized. Efficacy of the objective of improving skills development was modest. Efficacy of the objectives of improving family assistance and public sector management was modest before restructuring and substantial after restructuring, as the revised targets were exceeded. Outcome before restructuring was rated as moderately unsatisfactory and moderately satisfactory after restructuring. Taking these ratings discussed above and weighting them by the shares of Bank disbursement before and after restructuring,(0.70*3 = 0.30*4 = 3.30), the overall outcome is rated as moderately unsatisfactory. a. Outcome Rating Moderately Unsatisfactory 7. Risk to Development Outcome Macroeconomic risk. There is macroeconomic risk to areas associated with family assistance and skills development. In the case of the former, family assistance to municipalities depend on fiscal transfers, and demand for employment in the productive sector depends on the resiliency of the labor market. Environmental risk. Environmental changes increase the likelihood of drought and this could undermine the accomplishments in the area associated with improving water quality. Institutional risk. The government changes and high staff turnover could present risks to the commitment to the results-based approach at the state level. 8. Assessment of Bank Performance Page 10 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Strength. Serv. Delivery Ceará PforR (P127463) a. Quality-at-Entry This operation was prepared based on the experience from prior Bank-financed multi-sector operations at the sub-national level in Brazil (Minas Gerais and Pernambuco) and previous SWAP operations in Ceara (Second SWAP project and the Inclusive Growth project), which introduced disbursements against results- based indicators. Lessons incorporated at design, included having fewer executing agencies for better coordination, permitting re-negotiations on DLIs only after the Mid-Term Review and to the extent possible, relying on local procurement systems. The operation appropriately combined sector interventions with technical assistance activities. Several risks were identified at appraisal, including substantial technical risks associated with coordination of several state agencies. Mitigation measures incorporated at design, included the capacity building of state agencies (PAD, page 23). The implementation arrangements - with the Ceara Economic Research Institute (IPECE) in charge of project coordination - was appropriate, given that IPECE had executed prior Bank-financed operations. The arrangements at appraisal for monitoring and evaluation (discussed in section 8a) and safeguards and fiduciary compliance were appropriate (discussed in section 9). The multi sector operation, with two bank instruments, was complex and ambitious. Procurement delays contributed to cancellation of several technical assistance activities. There were moderate shortcomings in M&E design, with no indicators for monitoring the technical assistance activities financed through the IPF instrument. This is discussed further in section 3b. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision Eight Implementation Status and Results Reports were filed between 2014-2018. The number of supervision missions were more than the norm (48 missions during the project lifetime), in view of the complexity of the project (multi-sector operation, with two Bank instruments). Although the continuity of leadership was undermined through three changes to the task team leadership during the project lifetime, this was offset by the inclusion of team members from several Global Practices (GPs), including governance, social protection, and water specialists. The team aided in implementing M&E (discussed in section 9b) and the support provided by the team aided in safeguards compliance. The drawbacks in M&E design regarding the lack of indicators for the IPF component of the project, were not rectified during implementation. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Page 11 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Strength. Serv. Delivery Ceará PforR (P127463) Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The M&E indicators for the PDOs pertaining to family assistance and water quality were measurable and appropriate, though the institutional aspects of water coordination were not well handled. The indicators pertaining to the skills development area were problematic, as demand for graduates could depend on factors which were outside the project scope. As indicated in section 9a, there were no indicators for monitoring the technical assistance activities financed through the IPF instrument. This undermined a proper evaluation of the project's accomplishments. The Secretariat of Planning and Management through the Ceara Economic Research Center was responsible for M&E, through the State Systems. b. M&E Implementation The Bank supervision team provided support for facilitating data collection, through aligning the results framework and its indicators with the Government's own M&E systems. The lack of indicators for monitoring the IPF framework was not rectified during implementation. c. M&E Utilization The ICR does not provide details but notes that (paragraph 68) the data-driven frameworks were being used after the project closed. However, there was no comprehensive monitoring of continuous employment of graduates of technical and vocational training schools. M&E Quality Rating Modest 10. Other Issues a. Safeguards The PforR Program. The PforR activities comprised civil works, capacity building and policy reforms. The environmental impact of PforR program interventions (such as construction of new skills development Page 12 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Strength. Serv. Delivery Ceará PforR (P127463) facilities, maintenance of existing social inclusion centers and extension of water connections), were expected to be minor (PAD, page 93). The PAD (page 96) notes that the laws and guidelines for environmental assessment and resettlement were deemed to be adequate and an Action Plan was prepared at appraisal to mitigate the possible adverse environmental and social impacts of the operation. The IPF component. The project was classified as a Category B project for World Bank Safeguard policies. Two safeguard policies were triggered at appraisal (PAD, page viii): Environmental Assessment (OP/BP 4.01): and Indigenous Peoples (OP/BP 4.10). The ICR (page 26) does not report of any adverse environmental or involuntary resettlement issues during implementation. b. Fiduciary Compliance Fiduciary management. A fiduciary (financial management and procurement) assessment conducted at appraisal, concluded that the arrangements were satisfactory (PAD, page 20). The ICR does not report any fiduciary compliance issues during implementation. c. Unintended impacts (Positive or Negative) --- d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Overall relevance was rated as Substantial. Efficacy of the objectives was moderately Moderately Moderately unsatisfactory before Outcome Satisfactory Unsatisfactory restructuing when 70% was disbursed and moderately satisfactory when the balance was disbursed. Moderately There were shortcomings at Bank Performance Satisfactory Satisfactory Quality-at-Entry. Quality of M&E Modest Modest --- Quality of ICR Substantial --- Page 13 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Strength. Serv. Delivery Ceará PforR (P127463) 12. Lessons The ICR draws the following three lessons from the experience of implementing this project, with some adaptation of language. (1) The PforR instrument can be useful for supporting improvements in governance, with appropriate DLIs but need to be carefully and discretely targeted. The experience of this operation showed that the while the DLIs were appropriate for the areas associated with family assistance and improving water quality, they were of limited use for the area associated with improving skills development. The level of ambition regarding how the P4R would drive reform in all sectors was limiting given the scale and reach of where the reform agenda was directed. (2) While complementing the PforR instrument with a technical assistance component financed through an IPF instrument, can help in producing improvements across several result areas, it is necessary to include indicators for monitoring activities associated with the IPF instrument. Lack of indicators for monitoring the activities financed through the IPF instrument, besides creating a a bias towards a greater weight given to the PforR actions, undermine a proper evaluation of the project's accomplishments and do not allow clearer insight in how each instrument informs the other. (3) Better formulating of target values is required for PforR operations. In this operation, some indicators reached their target levels early in the project, implying that the targets were modest in terms of the level of ambition. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR is well-written and candid. It clearly discusses the issues associated with the PDO on skills development. It provides a clear narrative of the deep economic crisis of 2015 and 2016, which necessitated a reduction in targets for the family assistance area. The ICR is consistent with the guidelines and appropriately conducts a split rating of objectives. The ICR draws reasonably good lessons from the experience of implementing this operation. The ICR would have been enhanced by a clearer narrative on how the P4R instrument drove the reform approach of the state and explained how the breadth of engagement ultimately led to improved results for citizens. Given the long and established background of the Bank's work in the region, a clearer explanation around the differences in performance over time would have been useful. a. Quality of ICR Rating Page 14 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Strength. Serv. Delivery Ceará PforR (P127463) Substantial Page 15 of 15