PANAMA Improving Basic Health, Equity, Opportunities among Indigenous and Poor Rural Communities Report No. 110617 JUNE 12, 2017 © 2017 International Bank for Reconstruction This work is a product of the staff of The World RIGHTS AND PERMISSIONS and Development / The World Bank Bank with external contributions. The findings, The material in this work is subject to copyright. 1818 H Street NW interpretations, and conclusions expressed in Because The World Bank encourages Washington DC 20433 this work do not necessarily reflect the views of dissemination of its knowledge, this work may be Telephone: 202-473-1000 The World Bank, its Board of Executive reproduced, in whole or in part, for Internet: www.worldbank.org Directors, or the governments they represent. noncommercial purposes as long as full attribution to this work is given. Attribution—Please cite the work as follows: The World Bank does not guarantee the World Bank. 2017. Project Performance accuracy of the data included in this work. The Any queries on rights and licenses, including Assessment Report: Panama— Improving Basic boundaries, colors, denominations, and other subsidiary rights, should be addressed to Health, Equity, Opportunities among information shown on any map in this work do World Bank Publications, The World Bank Indigenous and Poor Rural Communities. not imply any judgment on the part of The Group, 1818 H Street NW, Washington, DC Washington, DC: World Bank. World Bank concerning the legal status of any 20433, USA; fax: 202-522-2625; e-mail: territory or the endorsement or acceptance of pubrights@worldbank.org. such boundaries. F Report No.: 110617 PROJECT PERFORMANCE ASSESSMENT REPORT PANAMA SOCIAL PROTECTION PROJECT (IBRD LOAN NO 74790) HEALTH EQUITY AND PERFORMANCE IMPROVEMENT PROJECT (IBRD LOAN NO 75870) June 12, 2017 Human Development and Economic Management Independent Evaluation Group ii Currency Equivalents (annual averages) Currency Unit = Panamanian balboas (B) 2007 US$1.00 B 1.00 2008 US$1.00 B 1.00 2009 US$1.00 B 1.00 2010 US$1.00 B 1.00 2011 US$1.00 B 1.00 2012 US$1.00 B 1.00 2013 US$1.00 B 1.00 2014 US$1.00 B 1.00 2015 US$1.00 B 1.00 All dollar amounts are U.S. dollars unless otherwise indicated. Abbreviations and Acronyms AIN-C Atención Integral de la Niñez en la Comunidad MIDES Ministry of Social Development (Comprehensive Community-Based Health Care MINSA Ministry of Health for Children) OE Organizaciones Externas (external CCT conditional cash transfer (program) organizations) CODIPRO Consejo Directivo de Proyectos (Project OI Organizaciones Internas (internal organizations) Executive Council) PAD project appraisal document DHS demographic and health survey PAISS Paquete Básico de Atención Integral de DRS Dirección Regional de Salud (regional health Servicios de Salud (Comprehensive Basic department) Package of Health Services) EEC Estrategia de Extensión de Cobertura (Coverage PAISS+N Paquete de Atención Integral de Salud + Extension Strategy) Nutrición (Basic Health and Nutrition Services GDP gross domestic product Coverage Expansion Package) HEPI Health Equity and Performance Improvement PBF performance-based financing (project) PDO project development objective IBRD International Bank for Reconstruction and PHC primary health care Development PPAR Project Performance Assessment Report ICR Implementation Completion and Results Report PSPV Protección en Salud para Poblaciones IDB Inter-American Development Bank Vulnerables (Health Protection for Vulnerable IEG Independent Evaluation Group Populations) IPP Indigenous People’s Plan RBF results-based financing IT information technology RO Red de Oportunidades M&E monitoring and evaluation UGSAF Unidad de Gestión de Salud, Administrativa y MEDUCA Ministry of Education Financiera (Ministry of Health’s Administrative MIS management information system and Financial Management Unit) Fiscal Year Government: January 1–December 31 Director-General, Independent Evaluation: Ms. Caroline Heider Director, Human Development and Economic Management: Mr. Auguste Tano Kouame Acting Manager, Corporate and Human Development: Ms. Soniya Carvalho Task Manager: Mr. Antonio Giuffrida iii Contents Principal Ratings ..............................................................................................................................v Key Staff Responsible......................................................................................................................v Preface........................................................................................................................................... vii Summary ...................................................................................................................................... viii 1. Background and Context..............................................................................................................1 2. Social Protection—Support to the “Red de Oportunidades” Project ...........................................6 Objectives, Design, and Relevance ..............................................................................................6 Project Development Objectives (PDOs) ................................................................................6 Relevance Of Objectives..........................................................................................................8 Relevance of Design ................................................................................................................8 Implementation ............................................................................................................................9 Achievement of Objectives ........................................................................................................13 Objective 1. Improve the management and operation of the RO ..........................................13 Objective 2. Increase beneficiaries’ participation in the RO, promoting therewith demand for education and maternal and infant health services ...........................................................15 Objective 3. Strengthen the supply of growth promotion interventions in the areas targeted by the RO ...............................................................................................................................16 Objective 4. Enhance the borrower’s capacity to design, implement and monitor coherent and efficient social sector policies interventions ...................................................................18 Efficiency ...................................................................................................................................18 Ratings .......................................................................................................................................19 Project’s Outcome ..................................................................................................................19 Risk to Development Outcome ..............................................................................................19 Bank Performance ..................................................................................................................19 Borrower Performance ...........................................................................................................20 Monitoring and Evaluation ....................................................................................................21 3. Health Equity and Performance Improvement Project ..............................................................22 Objectives, Design, and Relevance ............................................................................................22 Project Development Objectives............................................................................................22 Relevance Of Objectives........................................................................................................24 Relevance of Design ..............................................................................................................25 Implementation ..........................................................................................................................25 Safeguards and Fiduciary Compliance ..................................................................................30 Achievement of Objectives ........................................................................................................31 This report was prepared by Antonio Giuffrida (task manager) and Vibecke Dixon and Alex González (consultants) who assessed the project in June 2016. The report was peer reviewed by Mercedes Vellez and panel reviewed by Judyth L. Twigg. Aline Dukuze provided administrative support. iv Objective 1. Increase access of populations in selected underserved rural communities of its territory to quality basic health services known to improve mother and child health ...........31 Objective 2. Develop strategic planning, regulatory, and monitoring mechanisms known to improve health system performance ......................................................................................35 Efficiency ...................................................................................................................................36 Ratings .......................................................................................................................................36 Project’s Outcome ..................................................................................................................36 Risk to Development Outcome ..............................................................................................37 Bank Performance ..................................................................................................................37 Borrower Performance ...........................................................................................................38 Monitoring and Evaluation ....................................................................................................38 4. Lessons .......................................................................................................................................39 References ......................................................................................................................................40 Figures Figure 1.1. GDP Growth and Poverty Headcount in Panama and LAC, 1991–2015 ..................... 1 Figure 1.2. Social Spending as a Percent of GDP, 2007–13........................................................... 2 Figure 1.3. Maternal and Under-Five Mortality Rates in Panama and LAC, 1990–2015 .............. 2 Figure 1.4. Reasons to Consult a Health Professional, by Area and Indigenous Population, 2008 3 Figure 1.5. Reasons for No Consultation, by Area, 2008 ............................................................... 4 Figure 3.1. Registered Population as a Percentage of Census Estimate, Rondas, 2013–15 ......... 34 Map Map 1.1. Coverage of the EEC in Panama ..................................................................................... 6 Tables Table 1.1. Key Characteristics of World Bank and IDB Projects Supporting the EEC in Panama 5 Table 2.1. Social Protection Project Financing by Component ...................................................... 9 Table 2.2. Evolution of Chronic Malnutrition and Exclusive Breastfeeding Indicators in the Indigenous Comarcas Covered by the Program during Implementation ...................................... 17 Table 3.1. Health Services Comprising the Comprehensive Basic Package—PAISS ................. 23 Table 3.2 HEPI Project Financing by Component........................................................................ 26 Table 3.3. Component 1 Results-Based Financing Performance Indicators, 2013–14 ................ 28 Table 3.4. Component 1 Results-Based Financing Coverage Indicators, 2013–14...................... 33 Appendixes Appendix A. Basic Data Sheet.......................................................................................................45 Appendix B. Other Data ................................................................................................................49 Appendix C. List of Persons Met ...................................................................................................72 v Principal Ratings Social Protection—Support to the Red de Oportunidades Project ICR* ICR Review* PPAR Outcome Moderately satisfactory Moderately satisfactory Moderately satisfactory Risk to development Moderate Moderate Moderate outcome Bank performance Moderately satisfactory Moderately satisfactory Moderately satisfactory Borrower Moderately satisfactory Moderately satisfactory Moderately satisfactory performance * The Implementation Completion and Results Report (ICR) is a self-evaluation by the responsible global practice. The ICR Review is an intermediate product of the Independent Evaluation Group that seeks to independently validate the findings of the ICR. Panama—Health Equity and Performance Improvement Project ICR ICR Review PPAR Outcome Moderately satisfactory Moderately satisfactory Moderately satisfactory Risk to development Low to negligible Negligible to low Significant outcome Bank performance Moderately satisfactory Moderately satisfactory Moderate satisfactory Borrower Moderately satisfactory Moderately satisfactory Moderately satisfactory performance Key Staff Responsible Social Protection—Support to the Red de Oportunidades Project Project Task manager or leader Sector director Country director Appraisal Manuel Salazar Helena G. Ribe Jane Armitage Completion Edmundo Murrugarra Margaret Ellen Grosh J. Humberto Lopez Panama—Health Equity and Performance Improvement Project Project Task manager or leader Sector director Country director Appraisal Fernando M. Torres Keith E. Hansen Laura Frigenti Completion Carmen Carpio Daniel Dulitzky J. Humberto Lopez vi IEG Mission: Improving World Bank Group development results through excellence in independent evaluation. About this Report The Independent Evaluation Group (IEG) assesses the programs and activities of the World Bank for two purposes: first, to ensure the integrity of the World Bank’s self-evaluation process and to verify that the World Bank’s work is producing the expected results, and second, to help develop improved directions, policies, and procedures through the dissemination of lessons drawn from experience. As part of this work, IEG annually assesses 20-25 percent of the World Bank’s lending operations through field work. In selecting operations for assessment, preference is given to those that are innovative, large, or complex; those that are relevant to upcoming studies or country evaluations; those for which Executive Directors or World Bank management have requested assessments; and those that are likely to generate important lessons. To prepare a Project Performance Assessment Report (PPAR), IEG staff examine project files and other documents, visit the borrowing country to discuss the operation with the government, and other in-country stakeholders, interview World Bank staff and other donor agency staff both at headquarters and in local offices as appropriate, and apply other evaluative methods as needed. Each PPAR is subject to technical peer review, internal IEG Panel review, and management approval. Once cleared internally, the PPAR is commented on by the responsible World Bank country management unit. The PPAR is also sent to the borrower for review. IEG incorporates both World Bank and borrower comments as appropriate, and the borrowers' comments are attached to the document that is sent to the World Bank's Board of Executive Directors. After an assessment report has been sent to the Board, it is disclosed to the public. About the IEG Rating System for Public Sector Evaluations IEG’s use of multiple evaluation methods offers both rigor and a necessary level of flexibility to adapt to lending instrument, project design, or sectoral approach. IEG evaluators all apply the same basic method to arrive at their project ratings. Following is the definition and rating scale used for each evaluation criterion (additional information is available on the IEG website: http://ieg.worldbankgroup.org). Outcome: The extent to which the operation’s major relevant objectives were achieved, or are expected to be achieved, efficiently. The rating has three dimensions: relevance, efficacy, and efficiency. Relevance includes relevance of objectives and relevance of design. Relevance of objectives is the extent to which the project’s objectives are consistent with the country’s current development priorities and with current World Bank country and sectoral assistance strategies and corporate goals (expressed in Poverty Reduction Strategy Papers, country assistance strategies, sector strategy papers, and operational policies). Relevance of design is the extent to which the project’s design is consistent with the stated objectives. Efficacy is the extent to which the project’s objectives were achieved, or are expected to be achieved, taking into account their relative importance. Efficiency is the extent to which the project achieved, or is expected to achieve, a return higher than the opportunity cost of capital and benefits at least cost compared to alternatives. The efficiency dimension is not applied to development policy operations, which provide general budget support. Possible ratings for outcome: highly satisfactory, satisfactory, moderately satisfactory, moderately unsatisfactory, unsatisfactory, highly unsatisfactory. Risk to Development Outcome: The risk, at the time of evaluation, that development outcomes (or expected outcomes) will not be maintained (or realized). Possible ratings for risk to development outcome: high, significant, moderate, negligible to low, not evaluable. World Bank Performance: The extent to which services provided by the World Bank ensured quality at entry of the operation and supported effective implementation through appropriate supervision (including ensuring adequate transition arrangements for regular operation of supported activities after loan/credit closing, toward the achievement of development outcomes. The rating has two dimensions: quality at entry and quality of supervision. Possible ratings for World Bank performance: highly satisfactory, satisfactory, moderately satisfactory, moderately unsatisfactory, unsatisfactory, highly unsatisfactory. Borrower Performance: The extent to which the borrower (including the government and implementing agency or agencies) ensured quality of preparation and implementation, and complied with covenants and agreements, toward the achievement of development outcomes. The rating has two dimensions: government performance and implementing agency(ies) performance. Possible ratings for borrower performance: highly satisfactory, satisfactory, moderately satisfactory, moderately unsatisfactory, unsatisfactory, highly unsatisfactory. vii Preface This is the Project Performance Assessment Report (PPAR) for the Social Protection—Support to the Red de Oportunidades (Social Protection) project and the Health Equity and Performance Improvement (HEPI) project. The Social Protection project was approved in July 2007 and closed in September 2014. It had a total cost of $46.1 million and received an International Bank for Reconstruction and Development (IBRD) loan of $23.2 million. The HEPI project was approved in August 2008 and closed in December 2014. It had a total cost of $56.3 million and was supported by a $40 million IBRD loan. The Independent Evaluation Group (IEG) Implementation Completion and Results Report Review flagged the HEPI project for a PPAR, highlighting the value of better understanding the performance-based financing (PBF) mechanisms used in the project. The Social Protection project is included in the PPAR because it, too, featured PBF, but with different verification and payment modalities. In addition, the joint review of the two projects allows an assessment of how different strategies (for example, the demand-side incentives provided by the conditional cash transfer—CCT—program Red de Oportunidades, and the supply-side incentives provided by PBF) contribute to improved access and utilization of health services. The assessment is based on a review of all relevant project documentation (that is, project appraisal documents [PADs], Implementation Completion and Results Reports [ICRs], IEG’s ICR Reviews, country strategies, and relevant sector strategies), interviews of World Bank staff who participated in the design and implementation of the two projects, the findings of the IEG mission that visited Panama June 6–18, 2016, and semistructured reviews of the literature, including data from the World Development Indicators database and population health surveys conducted in Panama during the relevant period. Appendix C lists the persons interviewed during the mission and in Washington, DC. It includes government officials, public officials who participated in the implementation of the two projects, staff of the World Bank and Inter- American Development Bank resident missions in Panama, and the projects’ stakeholders, such as representatives of municipalities, service providers, and beneficiaries. The report presents a detailed assessment of the two operations using the standard IEG PPAR methodology. Lessons learned from the assessment of the two projects will be used as inputs to the forthcoming IEG evaluation of World Bank Group support to health services. Following standard IEG procedures, a copy of the draft report was sent to the relevant government officials and agencies for their review and feedback. No comments were received.’ The cooperation and assistance of all persons who contributed to the preparation of the report is gratefully acknowledged. Summary Background This report assesses the performance of two projects: (i) the Social Protection—Support to the Red de Oportunidades (Social Protection) (P098328) project and (ii) the Health Equity and Performance Improvement (HEPI) (P106445) project. The Social Protection and HEPI projects were approved by the World Bank Board of Directors in July 2007 and in August 2008, respectively. The two projects were prepared during the administration of President Martín Torrijos (in office September 1, 2004–July 1, 2009) and implemented during the administration of President Ricardo Martinelli (in office July 1, 2009–July 1, 2014). The Social Protection project was completed in September 2014 and the HEPI project a few months later, in December. The two projects presented a clear vision of supporting human capital and improving maternal and child health outcomes through a joint strategy of incentivizing the use of basic health services through the conditional cash transfer (CCT) program, Red de Oportunidades (RO), and of extending the coverage to indigenous and rural communities (Estrategia de Extensión de Cobertura—EEC) through mobile health teams. The two projects also shared similar interventions and activities: (i) both projects financed the EEC, (ii) both projects envisaged the implementation of impact evaluations, and (iii) both projects aimed at improving the government’s capacity to design, implement, and monitor coherent and efficient sector policies. The RO program was launched in 2006 as a national strategy to alleviate extreme poverty and to foster the use of health and educational services. The program provides cash transfers to extremely poor families living in rural, indigenous, and urban marginal communities on the condition they maintain school attendance of school-age children and ensure pregnant mothers and children under five years old visit health providers according to the country’s health protocol. The World Bank and Inter-American Development Bank (IDB) have supported the RO since its conceptualization, providing technical assistance for the elaboration of poverty maps to identify the poorest communities (corregimentos) and for the development of the proxy means test used to assess the poverty level of RO beneficiaries. The EEC strategy has been implemented in Panama since 2000 to provide a comprehensive basic package of health services (Paquete Básico de Atención Integral de Servicios de Salud—PAISS) to isolated communities through mobile health teams (Redes Itinerantes). The basic package is delivered by private providers (Organizaciones Externas/Extra-institucionales—OEs) remunerated using capitation payment based on the estimated covered population with final payment related to performance. In some cases, the package of basic health services was delivered by health professionals hired by the regional health departments (Organizaciones Internas/Institucional—OIs). Social Protection Project This project had four development objectives: ix (i) “Improve the management and operation of the CCT program Red de Oportunidades (RO) that was recently established by the government of Panama.” The main beneficiaries of the CCT program were 75,000 poor and extremely poor households, primarily located in rural and indigenous areas (comarcas). (ii) “Increase beneficiaries’ participation in the CCT program and promote demand for education and maternal and infant health services through the contracting, training, and deployment of a network of direct family support social workers (promotores).” These promotores were also responsible for supporting the household committees (Comites de Familia) where beneficiaries could discuss and resolve issues related to the program. (iii) “Strengthen the provision of health and nutrition services through mobile health care teams in the indigenous comarcas targeted by the RO.” The EEC strategy through mobile health teams has been implemented in Panama since 2000 and was already supported by the World Bank and the IDB. (iv) “Enhance the government’s capacity to design, implement, and monitor coherent and efficient social sector policies and interventions.” To achieve this objective, the component provided technical assistance to the Technical Secretariat of the Social Cabinet, responsible for policy coordination in the social sector. The project’s outcome rating is Moderately Satisfactory. Its objectives are substantially relevant to current country conditions, national strategies and priorities, and the World Bank’s country partnership strategy (CPS). The design was substantially relevant, with a clear and logical results chain supporting the majority of objectives. The objective to improve the management and operation of the RO was substantially achieved. The capacity of the program to reach the poor was improved, and RO is one of the best-targeted CCTs in the Region and one of the best-targeted social assistance programs in Panama. The objective to increase beneficiaries’ participation in the CCT program and promoting therewith demand for education and maternal and infant health services was modestly achieved. As a result of the culturally appropriate direct support and targeted communication provided by the program about 76 percent of the beneficiaries knew their right to access services and their co-responsibilities, but data are not provided on the extent to which this knowledge was translated into demand for services. The objective to strengthen the provision of health and nutrition services in the indigenous comarcas targeted by the RO was substantially achieved. About 90 percent of children registered were receiving the package of basic health care services. The percentage of pregnant women receiving no fewer than three prenatal controls did not improve in comarcas, but the prevalence of exclusive breastfeeding for at least six months improved markedly in indigenous comarcas compared with the rest of the population. Finally, the achievement of the objective to enhance the government’s capacity to design, implement, and monitor coherent and efficient social sector policies and interventions is rated modest. The capacity of the Technical Secretariat of the Social Cabinet to coordinate social policies across ministries remained weak throughout the life of the project. Efficiency is rated modest considering that some activities did not achieve the desired results (for example, the impact evaluation was not completed), the per capita cost of the package of basic health services increased, and various extensions were granted for a total of 27 months. x Risk to development outcome is rated Moderate. The RO continues to be the flagship social assistance program in Panama, but moderate risks, mainly related to verification of beneficiary co-responsibilities, affect its day-to-day operation. The EEC continues to be successfully implemented in indigenous comarcas by the Ministry of Social Development (MIDES) and is supported both in the communities and nationally. The risk related to cultural appropriateness is rated low because of adequately developed culturally relevant implementation approaches and education materials. Bank performance is rated Moderately Satisfactory. Quality at entry is rated moderately satisfactory. The project design drew on extensive and high quality work that the World Bank and the IDB had developed from the implementation of various CCT programs in Latin America, in particular, in Brazil, Chile, and Mexico. However, the design for the verification of co- responsibility was overly complex. Quality of Supervision is rated moderately unsatisfactory. The World Bank team conducted 14 supervision missions during the seven years of implementation, revised project activities when necessary, and provided quality technical assistance to MIDES. However, World Bank supervision could have been more proactive in supporting effective synergies between the CCT program and the EEC strategies, problems with the verification of co-responsibility could have been quickly addressed, and shortcomings in the implementation of the Indigenous People’s Plan fixed. Borrower performance is rated Moderately Satisfactory. Government performance is rated moderately unsatisfactory. Although MIDES participated in project preparation and implementation, there were delays in both phases of the project because of factors associated with the government’s commitment and involvement. The delays also negatively affected the impact evaluation of the program. Implementing agency performance is rated moderately satisfactory. MIDES, the Ministry of Health (MINSA), and the Ministry of Education implemented the program relatively successfully, although various challenges and delays affected inter-ministerial collaboration and communication. HEPI Project The project had two development objectives: (i) to increase access in selected underserved rural communities to quality basic health services known to improve mother and child health; and (ii) to develop strategic planning, regulatory, and monitoring mechanisms known to improve health system performance. To achieve the first objective, the project financed the EEC to cover about 180,000 people living in rural (nonindigenous) communities and activities to strengthen the primary health services networks in these rural areas. To achieve the second objective, the project provided technical assistance to finance various studies, including a national health strategy and a new demographic and health survey (DHS). The project’s outcome rating is Moderately Satisfactory. Its objectives are highly relevant to current country conditions, national strategies and priorities, the World Bank’s country partnership strategy (CPS), and its Health, Nutrition, and Population Strategy. The design was substantially relevant, with a clear and logical results chain supporting the two objectives. The objective to increase access in selected underserved rural communities to quality basic health services known to improve mother and child health was substantially achieved. The mobile health teams achieved virtually all targets related to population coverage and to the provision of xi health services: at least three prenatal controls (one in each trimester) to pregnant women; children under one-year old with full vaccination scheme; and women delivering children with the assistance of trained personnel from MINSA. However, improvement in the capacity of the supporting primary health care (PHC) networks was more modest. The objective to develop strategic planning, regulatory, and monitoring mechanisms known to improve health system performance is rated modest. Few of the studies and surveys envisaged were completed, and even if the availability of essential drugs at primary health centers improved, the overall capacity of MINSA to develop strategic planning, regulatory, and monitoring mechanisms did not improve. Efficiency is rated modest considering the increase in the per capita cost of the package of basic health services, the cumbersome and expensive third-party verification system, and the 18-month extension. Risk to Development Outcome is rated Significant. The EEC continues to be financed by external resources (that is, an IDB-financed project). Strategic planning, financial management, and implementation capacity at MINSA remain weak. The level of coordination with MIDES is also limited. World Bank performance is rated Moderately Satisfactory. Quality at entry is rated moderately unsatisfactory. The World Bank team did not adequately consider the practicality of the verification system in the project’s design. The project’s operational manual included a complicated verification process that never became fully operational during project implementation. This slowed down implementation, led to payment delays, and created additional administrative burdens. Quality of Supervision is rated moderately satisfactory. The World Bank team conducted 13 supervision missions during the seven years of project operation. The project team was reactive in adjusting the capitation amount in response to concerns about insufficient cost recovery for the ambulatory teams working in the least accessible areas, and processed two restructurings to improve project execution. However, the World Bank team did not manage to address the problems resulting from the complex verification and payment system for the mobile health teams. Borrower performance is rated Moderately Satisfactory. Government performance is rated moderately unsatisfactory. The borrower contribution was less than expected, and the provision of counterpart funding experienced delays and hindrances because of to internal control procedures and legal compliances that negatively affected payments to providers and some activities such as the impact evaluation of the program. The role of the regional health departments in the verification and payments to private providers created administrative burdens and delays in disbursements. Implementing agency performance is rated moderately satisfactory. The Health, Administrative, and Financial Management Unit (UGSAF) showed a satisfactory capacity in implementing the project and in maintaining an adequate monitoring and evaluation system and managed to maintain strong collaboration and relationships with MINSA at both central and regional levels. However, the Project Executive Council (Consejo Directivo de Proyectos—CODIPRO) that oversaw implementation and reported to MINSA was active only during the first phase of project implementation. Lessons Five important lessons are drawn from the project. xii  Attention to possible synergies across interventions within a World Bank–financed project and across two or more World Bank–financed projects when they are being implemented in the same place at the same time can be beneficial. In this case, achievement of such synergies was plausible, for example, through the integration of the management information systems (MISs) of the RO/Social Protection and HEPI projects. The RO MIS makes available a registry of program beneficiaries that may also serve as the basis for an integrated registry of beneficiaries for social programs across the country. Examples of highly integrated MIS are Bolsa Familia in Brazil and Chile Solidario in Chile (Villalobos, Blanco, and Bassett 2010). The integration of the RO and EEC MIS could have allowed for updating of the RO beneficiary roster using information on household composition collected by the mobile health teams, and the prioritization of RO expansion in communities where utilization of health services is suboptimal among the poor. However, these synergies did not materialize in Panama, indicating that opportunities for improvements in efficacy and efficiency may have been missed.  To ensure adequate provision of basic health services, the supporting PHC network, including public clinics and hospitals, needs to be strengthened. Mobile health teams cannot provide the whole range of health services that are critical to reducing maternal and child mortality (Kuruvilla et al. 2014). Thus, as recognized in the original design of the HEPI project, it is necessary to strengthen the entire PHC network, which includes the public clinics and hospitals that serve these vulnerable populations, but this was not accomplished in Panama.  The use of short-term contracts by private health providers can have negative impacts on turnover. The contractual terms offered to private providers of the basic health package (PAISS and PAISS+N) were for one year (renewable to up to four years). In turn, private providers offered short-term contracts (usually of six months’ duration) to the doctors, nurses, technicians, and drivers comprising the mobile health teams. These short contracts led to very high staff turnover, which in turn became an extra burden for the regional governments as they had to provide all the training for newly appointed staff.  Verification systems need to be lean and efficient and set up before results-based financing (RBF) payments are made. Processes, roles, and responsibilities, especially in complex RBF payment and reporting systems, need to be clearly identified upfront and kept at a minimum. Modern management information systems and technological solutions should be identified beforehand and used to reduce administration costs and avoid potential delays. Unfortunately, the specific operational aspects of the performance-based financing (PBF) scheme in the HEPI project, including its payment and verification mechanisms, were not fully defined before starting implementation. In particular, the role of the regional health departments (DRSs) in verification and authorizing payments to mobile health teams created bottlenecks and delays that affected the provision of health services.  Successful impact evaluations require both technical capacity and commitment from the government side and technical support from the World Bank. The impact evaluations envisaged in the Social Protection and the HEPI programs were complex. They required the collection of primary data using household surveys at baseline and at xiii endpoint, and maintaining a clear distinction between those that received the interventions and those that did not. However, the initial delays in contracting the firms to conduct the impact evaluations and the change in government following the presidential election reduced the buy-in from the counterpart in charge of decisions regarding the expansion of the programs, thus reducing the feasibility and quality of the evaluation. Auguste Tano Kouame Director Human Development and Economic Management Independent Evaluation Group 1 1. Background and Context 1.1 Panama has registered high economic growth since 1991, which contributed to significant poverty reduction. Gross domestic product (GDP) annual growth has consistently outpaced the average for the Latin America and the Caribbean Region over the 1990–2015 period, with the exception of the years 1995–96 and 2009–10. Moderate poverty (those living with less than $3.1 per day, 2011 purchasing power parity—PPP) declined from 25.1 percent in 2002 to 8 percent in 2013, and extreme poverty (those living with less than $1.9 per day, 2011 PPP) declined from 16.4 percent to 2.9 percent over the same period (see figure 1.1). Figure 1.1. GDP Growth and Poverty Headcount in Panama and LAC, 1991–2015 35 Panama - Annual % GDP growth Latin America & the Caribbean (IDA & IBRD countries) - Annual % GDP growth 30 Panama - Poverty headcount ratio at $1.90 a day (2011 PPP) - % of population Panama - Poverty headcount ratio at $3.10 a day (2011 PPP) - % of population 25 25.1 22.8 20 16.4 15 15.0 10 8.0 5 2.9 0 -5 Source: World Development Indicators (database), October 2016. Note: GDP = gross domestic product; IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; PPP = purchasing power parity. 1.2 In the last few years, Panama’s social spending increased significantly in real terms (see figure 1.1), but decreased slightly as a percentage of GDP (see figure 1.2). During this period, social security consistently had the largest share of overall social spending, followed by health, education, and then social assistance and labor (see figure 1.2). During the past decade, Panama has expanded its social assistance programs through a number of cash transfers: the conditional cash transfer (CCT), Red de Oportunidades (RO), introduced in the year 2006 that currently covers 72,563 households;1 the scholarship program, Beca Universal, that covers more than 600,000 students at the national level; the noncontributory pension program, “120 a los 65,” that covers 100,000 elderly beneficiaries; a school supplies program, Uniformes y Utiles Escolares; and a social assistance transfer for people with disabilities, Angel Guardian, with 10,000 beneficiaries. 2 Figure 1.2. Social Spending as a Percent of GDP, 2007–13 Source: World Bank Central America Social Sector Expenditure and Institutional Review (World Bank 2015a, 3). 1.3 Compared with the Latin America and the Caribbean Region, Panama has underperformed in terms of maternal and child mortality. Under-five mortality rate decreased by 45 percent over the 1990–2015 period in Panama, but decreased by 67 percent in the entire Region. Over the same period, the maternal mortality ratio (model estimate) went down in Panama by only 3 percent, while the Region as a whole managed to halve it (see figure 1.3). Figure 1.3. Maternal and Under-Five Mortality Rates in Panama and LAC, 1990–2015 160 LAC - Mortality rate, under-5 (per 1,000) Panama - Mortality rate, under-5 (per 1,000) 140 LAC - Maternal mortality ratio (modeled estimate, per 100,000 live births) Panama - Maternal mortality ratio (modeled estimate, per 100,000 live births) 120 100 80 60 40 20 0 1990 1995 2000 2005 2010 2015 Source: World Development Indicators (database), October 2016. Note: LAC = Latin America and the Caribbean Region. 3 1.4 Accessibility to health services and poor health outcomes are key challenges in rural and indigenous areas (comarcas). For example, the national level maternal mortality rate in 2007 was estimated at 59.4 deaths per 100,000 live births, but in the indigenous comarcas of Ngöbe Buglé and Kuna Yala and the rural province of Darién the rates were much higher: 376.4, 584.8, and 292.7, respectively. In 2009, infant mortality rate in the indigenous comarcas was 20.3 deaths per 1,000 live births, while the national average was 12.2 deaths per 1,000 live births.2 Ancillary data from Encuesta de Niveles de Vida (1997, 2003, and 2008) show an increasing trend in chronic malnutrition for children under five years old in comarcas, increasing from 54 percent in 1997, to 59.6 percent in 2003, and reaching 62 percent in 2008. The disparities in health outcomes are matched by inequality in the accessibility and utilization of health services. For example, in 2008 only 41 percent of indigenous individuals consulted with a doctor, compared to 74 percent of urban dwellers and 68 percent of rural dwellers. Almost a third of indigenous peoples (31 percent) were not able to consult any health professional when ill, compared to 20 percent of those living in urban areas and 26 percent of those living in rural areas (see figure 1.4). In effect, distance is the second major factor cited by indigenous peoples for not seeking care. In addition, financial reasons (expensive or no money) were the second most important factor for rural populations not seeking care (see figure 1.5). Figure 1.4. Reasons to Consult a Health Professional, by Area and Indigenous Population, 2008 Source: World Bank Central America Social Sector Expenditure and Institutional Review (World Bank 2015a, 65). 4 Figure 1.5. Reasons for No Consultation, by Area, 2008 Source: World Bank SSEIR team’s analysis of household surveys, calculations using ENV 2008 (World Bank 2015a, 66). 1.5 The World Bank’s first social sector operation in Panama was the Rural Health Project (P007846), approved in 1995. The project comprised a Basic Health and Nutrition component to support government efforts to develop a targeted and nutritionally superior program of food supplementation reaching the most vulnerable members of the household. The project was restructured in 2000 to introduce the Coverage Extension Strategy (EEC), based on contracting out to nongovernmental organizations (also known as external organizations or, in Spanish, as Organizaciones Externas or Extrainstitucionales—OE) the provision of a Comprehensive Basic Package of Health Services (Paquete Básico de Atención Integral de Servicios de Salud— PAISS). The new strategy benefited approximately 90,000 residents in 120 rural communities (five regions) located in districts where the incidence of poverty was more than 70 percent (see World Bank 2004, 5). An impact evaluation of the Basic Health and Nutrition Component was carried out in the second half of 2003. The impact evaluation supported “the perception that, because of the project (although maybe not exclusively), the general health and nutrition status of the target population has improved in recent years over the last two years” and that “the introduction of the PAISS had a positive impact on the use of preventive health services among the poor rural population” (World Bank 2004, 8–9). 1.6 The EEC continued to be supported by the Multiphase Program for the Institutional Transformation of the Health Sector project (PN-0076) of the Inter-American Development Bank (IDB) over the 2002–08 period.3 Under the IDB-financed project, the Ministry of Health (MINSA) continued to use capitated payments (census-estimated population) and introduced the use of performance payments for each of the quarterly payments to OEs to deliver the PAISS and the use of social audit. The program also created the fund for integral medical tours (Fondo para Giras Integrales de Salud—FOGI) to enable the regional health departments (Dirección Regional de Salud—DRS) to respond on equal footing in terms of availability of inputs. 1.7 In 2008, the government of Panama launched the Health Protection for Vulnerable Populations (Protección en Salud para Poblaciones Vulnerables—PSPV) program. The PSPV program continued to support the EEC with financial resources from HEPI in rural 5 nonindigenous communities and with resources from the Social Protection project in indigenous communities (comarcas). However, some important innovations were introduced:  The introduction of the PAISS strengthened with Comprehensive Community-based Health Care for Children (Atención Integral de la Niñez en la Comunidad—AIN-C) in the indigenous comarcas.4 The PAISS strengthened with AIN-C is also identified by the acronym PAISS+N.  The possibility of providing the PAISS through health professionals contracted by MINSA’s regional health departments—DRS (Organización Internas—OI),5 which replaced the FOGI.  The role of DRS to authorize payment to mobile health teams in the HEPI project with the intention of strengthening its role as steward of the primary health care (PHC) networks.  The use of capitation payments based on the registered population. Until this point in time, health providers were remunerated through capitation based on the estimated population. 1.8 From its inception in 1995, the Panama EEC received financial and technical support from both the World Bank and the IDB. The key characteristics of the World Bank and IDB projects that supported the EEC in Panama are summarized in table 1.1 and in appendix B (table B.1). Map 1.1 presents the specific EEC coverage in each of the 14 provinces in Panama. Table 1.1. Key Characteristics of World Bank and IDB Projects Supporting the EEC in Panama Multiphase Program Health Equity Rural Health for the Institutional Social Protection Performance Features of the Project Transformation of the Project Improvement project (P007846) Health Sector Project (P098328) (HEPI) Project 1995–2002 (PN-0076) 2008–14 (P106445) 2003–08 2007–14 World Bank and IDB/World Bank and World Bank and IDB and government of Financial source government of government of government of Panama Panama Panama Panama PBSIN PAISS PAISS+N (Paquete Básico de (Paquete de Atención (Paquete de Atención PAISS Salud Integral y Integral de Servicios Integral de Servicios (Paquete de Health package Nutrición) de Salud) de Salud con Atención Integral Comprehensive Comprehensive Basic Atención Integral de de Servicios de Basic Package of Package of la Niñez en la Salud) Nutrition and Health Health Services Comunidad–AIN-C) Beneficiaries 90,000 people 250,000 people 456,000 people 5 Indigenous 5 regions: comarcas: Rural not Bocas del Toro, Emberá Wounaan, indigenous Chiriquí, Geographical area The entire country Guna Yala, communities Darién, Madugandí, in all 14 regions Ngóbe Buglé, Ngöbe Buglé, of the country and Veraguas and Wargandí 6 Map 1.1. Coverage of the EEC in Panama 2. Social Protection—Support to the “Red de Oportunidades” Project Objectives, Design, and Relevance PROJECT DEVELOPMENT OBJECTIVES 2.1 The Social Protection project (P098328) comprises four project development objectives (PDOs). As stated in the loan agreement, the PDOs were to assist the borrower to “(i) improve the management and operation of the Red de Oportunidades (RO); (ii) increase beneficiaries’ participation in the RO and promote demand for education and maternal and infant health services; (iii) strengthen the supply of growth promotion interventions in the areas targeted by the RO; and (iv) enhance the borrower’s capacity to design, implement, and monitor coherent and efficient social sector policies interventions.” The objectives remained unchanged throughout the project; however, the formulation of the objectives in the project appraisal document (PAD) was slightly different.6 This assessment is based on the formulation of the PDO in the loan agreement. The project comprises four components:  Component 1: Improve the management and monitoring and evaluation of the Red de Oportunidades (RO) program. This component was to finance technical assistance to improve the effectiveness and efficiency of the RO by (i) developing and 7 implementing a management information system (MIS), (ii) developing a methodology to update and correct the targeting and payment mechanisms in accordance with changes in extreme poverty and performance, and (iii) implementing a quality control mechanism in the field. Activities to measure the impact and evaluate the process of the RO included an impact evaluation. Also, the management of the RO central and provincial/comarcas staff was to be strengthened through capacity building and training. The governance of the RO was to be strengthened through technical assistance in areas such as defining roles and responsibilities and developing a clear oversight and accountability system.  Component 2: Increase beneficiary families’ participation in the conditional cash transfer (CCT) program and therefore boost their demand for health and education services. This component was to finance direct support to beneficiary families for participating in the program (promotores), support to targeted communities to develop or strengthen local organizations (for example, comites de familia) to participate in the program’s social audit and oversight, and support to the Ministry of Social Development’s (MINDES) local and provincial operations. Also, this component was to finance an information, education, and communication strategy to provide necessary information to the beneficiary population, including the process for registering and issuing identity cards to undocumented—especially indigenous—people, who were identified in a vulnerability survey.  Component 3: Strengthen the supply of growth and development promotion interventions (Basic Health and Nutrition Services Coverage Expansion Package— PAISS+N) and secure access to health and services for RO program beneficiaries in indigenous areas (comarcas). This component was designed to: (i) strengthen the PAISS+N with the promotion of community-based growth and interventions to prevent malnutrition, (ii) expand the strengthened PAISS+N maternal and infant health services to indigenous areas, (iii) strengthen the capacity of the Ministry of Health (MINSA) to supervise the implementation of the PAISS+N, and (iv) develop and implement a monitoring and evaluation (M&E) system. This component was implemented by MINSA.  Component 4: Improve social sector performance; enhance government capacity to design, implement, and monitor coherent social policies and interventions. This component was to finance technical assistance to the government of Panama to build capacity in areas including program and policies planning, program design and implementation coordination, and program implementation and progress monitoring. 2.2 The Social Protection project was approved in July 2007 as a $46.9 million operation to be completed by June 30, 2012. IBRD was to finance a $24 million loan with the Inter-American Development Bank (IDB), and the government of Panama was to provide cofinancing of $22.8 million. The project became effective in February 2008 and was closed in September 2014, 27 months later than planned, with a total execution period from effectiveness of 79 months (6 years and 7 months). 8 RELEVANCE OF OBJECTIVES 2.3 The relevance of the objectives is rated Substantial.7 2.4 The review by the Independent Evaluation Group (IEG) confirmed that the project’s objectives were substantially relevant at both project appraisal and closure. The Interim Strategy Note for FY2006–07 (World Bank 2005) emphasized poverty reduction especially among the poor and indigenous population; the objectives of the Social Protection project to enhance the RO and strengthen the supply of basic health interventions in the indigenous population covered by the RO are well aligned with that strategy. The project’s objectives were also in line with the country partnership strategy (CPS) at project closure (World Bank 2010), which aimed to improve quality and access to basic health and nutrition services, especially among rural and indigenous populations; and to enhance targeting and monitoring and evaluation of social programs, under its “greater opportunities for all” pillar. 2.5 However, the PDO to “increase beneficiaries’ participation in the RO and promote therewith demand for education and maternal and infant health services” is defined at the level of outputs rather than at the level of outcomes. This made the objective slightly less relevant. In addition, the formulation at output level misses the opportunity of ensuring adequate coordination between the supply-side and the demand-side of services. RELEVANCE OF DESIGN 2.6 The relevance of the design is rated Substantial.8 2.7 The project and its results framework were structured around the four components, each of which was in turn designed to achieve one of the four PDOs. The components and activities of the project at appraisal were relevant, necessary, and sufficient to achieve the project objectives (see table B.4 in appendix B). The results chain for PDO 1 (improving management and enhancing effectiveness of the RO) is well conceived. Component 1 was designed to achieve PDO 1, and the activities are relatively clear and convincing. The technical assistance is centered on building capacity for improving the targeting, payment, and governance mechanisms of the RO, and the capacity to monitor and evaluate its results. The results chain for PDO 2 is equally well conceived. The main activity of the component was the establishment of the network of “promotores” to support poor families to access health and education services. Additional technical assistance was provided to improve accountability mechanisms at the local level and to develop culturally appropriate communication and information strategies. The results chain of PDO 3 (strengthen supply of nutrition and child growth promotion interventions) is clearly formulated. PDO 3 is defined at the input level. It is also worth noting that even if PDO 3 focused on child health services, the basic health package provided by component 3 covered a wider package of basic health care services. PDO 4 (enhance government capacity to develop social sector policies and interventions based on evidence) included both input and output elements (for example, “enhance capacity” to “develop policies and develop interventions based on evidence”). The activities envisaged in component 4 to achieve the PDO were defined in general terms without defining the underlying results chain. However, IEG confirmed that this aspect of the design was intentional to achieve adequate flexibility and potential buy-in from the next administration. 9 Implementation 2.8 The implementation of the Social Protection project was carried out following the existing institutional arrangements at MIDES and MINSA. Project components 1, 2, and 4 were carried out under the overall direction of the Social Cabinet at MIDES. Component 3 was implemented by the Ministry of Health’s Administrative and Financial Management Unit (Unidad de Gestión de Salud, Administrativa y Financiera—UGSAF) at MINSA that was implementing the overall EEC for both indigenous and rural communities. Therefore, no additional project coordination units were created to implement the project. However, the Water Center for the Humid Tropics of Latin America and the Caribbean (Centro del Agua del Trópico Húmedo para América Latina y El Caribe—CATHALAC) was engaged to support the administrative and fiduciary functions of the project. 2.9 Several factors affected the implementation of the project, including the limited technical capacity of the ministries responsible to implement project activities and the capacity of the Minister of Economy and Finance to provide the space in the budget to implement the project. Additional challenges included (i) the change in government shortly after effectiveness that delayed key decision-making processes, (ii) low government capacity to manage World Bank Group fiduciary processes, and (iii) high turnover of technical personnel in the implementing units. These challenges coupled with the complex project design and interagency setup led to a slow start-up of the project. Two years after approval the project had disbursed about 8 percent of financing against the 41 percent originally planned. Project financing (estimated and actual) by component is presented in table 2.1. The project went through four level-2 restructurings during its implementation:  April 13, 2009, to provide additional time for the selection of independent agencies to carry out the impact evaluation;  March 15, 2012, to (i) reallocate resources across components; (ii) extend the project’s closing date by 18 months to December 31, 2013; and (iii) revise the Results Framework and Monitoring Indicators;  December 9, 2013, to provide a 6-month extension to complete critical activities, including the strategy for the verification of co-responsibilities;  May 20, 2014, to grant a 4-month extension until September 30, 2014 to finalize the impact evaluation data collection. Table 2.1. Social Protection Project Financing by Component Appraisal Estimate Actual IBRD IDB and (actual as IBRD Government IBRD IBRD a percent IBRD (percent of Panama (US$, (percent of Component (US$, millions) of total) (US$, millions) millions) of total) appraisal) 1 4.0 17 6.0 4.56 20 114.0 2 5.5 23 5.1 2.23 10 40.5 3 10.0 41 10.7 15.70 68 157.0 4 4.6 19 1.0 0.75 3 16.3 Total 24.1 100 22.8 23.24 100 96.4 10 2.10 The complex design, combined with changes in the political administration in 2009, resulted in implementation delays. The key implementation challenges by component are summarized below:  Component 1. The MIS was never fully functional as intended. The challenges were due to information constraints outside the control of the RO and to rigidities in the MIS software itself. For example, the first version of the MIS did not have the predefined list of school names. Therefore, factual errors and misspelling caused difficulties in the verification of conditionality related to school attendance. These types of errors were reduced drastically in subsequent MIS versions. However, up-to-date attendance records remained a challenge throughout the project. Notwithstanding the sound design, various project assumptions proved to be overly optimistic about the counterpart’s institutional capacity to implement the required investments and institutional arrangements. For example, the project design envisaged that the fully functional MIS would allow the exchange of electronic information on enrollment and health service utilization between the Ministries of Health (MINSA), Education (MEDUCA), and Social Development (MIDES) (see World Bank 2015b, 5). The technology and institutional arrangements required to achieve this level of integration are seldom available even in more advanced middle- and high-income countries. Nevertheless, IEG found that even with the existing limitation on the verification of co-responsibilities between the three ministries, the MIS and the overall management and operation of the RO were satisfactory and comparable to other well-performing CCT programs in the Region (see Robles Rubio and Stampini 2016, 8, 9 and 24). MIDES also showed limited capacity in the complex procurement processes required to contract the impact evaluation of the RO. The selection and appointment of the firm to carry out the impact evaluation took longer than the 12 months from the effectiveness date envisaged in the Loan Agreement, thus a level-2 restructuring was necessary to grant additional time to complete the process. The delays in contracting the firm reduced the number of children under two years old in the roster of beneficiaries of the RO. Consequently, while the data collection was successfully completed, the small number of observations (that is, for children under two years old) affected the statistical analysis, and the impact of RO on children with chronic malnutrition could not be estimated.  Component 2. The terms of reference were developed for the social workers (the promotores) who provide direct support to beneficiary families based on best practices from CCT programs in the Latin America and the Caribbean Region, such as Chile Solidario. The network of promotores provides tailored support and information about the RO and other social assistance programs to beneficiary families to facilitate the integration of the various programs and enhance their results. Important delays affected the initial selection and contacting of the promotores. However, the delays were justified by the strict arrangements designed to avoid political interference in selecting and contracting promotores, as these processes were conducted during the 2009 presidential campaign. In addition, budget constraints in 2012–13 caused downsizing of this activity.  Component 3. Initial delays related to contracting private health providers also affected the ability to reach out to the target population with timely services (such as achieving three prenatal controls for pregnant females). However, once the procurement process 11 was completed, disbursements under the component rose quickly and reached 96 percent between August and December 2011. The high disbursement rate was mainly due to the increase in the capitation rate to compensate for increments in the cost of inputs, salaries, and additional requirements of mobile health team composition. Therefore, the March 2012 project restructuring provided additional resources for component 3 to complete four years of health service delivery through the mobile health teams. To provide AIN-C (Comprehensive Community-based Health Care for Children) as part of the PAISS, (that is, to deliver PAISS+N) a nutritionist would need to join the mobile health team; however, there were not enough trained nutritionists in the country to meet the needs of the project. Therefore, MINSA agreed to replace the nutritionists with additional nursing technicians on the mobile health teams to supervise the AIN-C activities.9 The indigenous chiefs selected the women, called monitoras, who performed the function of community health workers supporting the delivery of the AIN-C.  Component 4. This component was affected by the reduced political role and coordination capacity of the Technical Secretariat of the Social Cabinet during the implementation period. The new administration had a different vision for the sector that weakened the capacity of the Social Cabinet to lead social policy formulation at MIDES as well as to coordinate with other ministries such as MINSA and MEDUCA, and to deliver the specific products and results envisaged under the component.10 2.11 The IEG mission confirmed that the RO program was, in practice, implemented as a semiconditional cash transfer program (see World Bank 2015b, 8); compliance with co- responsibilities was monitored but not enforced. However, the IEG mission confirmed through spot checks that securing beneficiaries’ fulfillment of co-responsibilities seemed to be working well at the local level because of peer-monitoring.11 In the smaller municipalities the social workers know their clients well, as do the teachers and the health workers; in the case of noncompliance of the co-responsibilities, for example if a child has not attended school as expected, the social worker (and/or teacher and headmaster) would look into the case to find the reasons behind the absence and to see how the problem could be solved (for example if the child has been absent because of a parent’s illness). In addition, the IEG mission confirmed that the beneficiaries in general carry their verification cards and comply well with their co- responsibilities. All the beneficiaries interviewed by the IEG mission expressed satisfaction with the RO; the funds were received on time and were generally spent on the family’s basic needs (for example, food, clothing, and medicines). These results are consistent with broader findings that semiconditional cash transfer programs can be effective in improving health and educational outcomes (Baird et al.2013).12 2.12 IEG reviewed the Health Situation Analysis (Análisis de Situación de Salud—ASIS) performed in Panama in the year 2014 to assess how the social and health programs, were articulated at the regional levels.13 The review, summarized in table B.3, reveals that there were no explicit coordination mechanisms between RO and the health delivery networks. In particular, the DRS of Chiriqui and Los Santos highlighted poor integration and coordination between the social and health policies and programs. 2.13 The visit of the IEG mission to Darién found that beneficiaries received RO payments regularly and on time, but also experienced operational challenges. Payments are disbursed 12 bimonthly by a bank in a nearby village on specific dates, which means that on one specific day every two months, 180 beneficiaries need to travel from Yaviza to a town approximately an hour away by bus and queue up to receive their payments. This process is impractical (the expense and difficulty for all the beneficiaries to find transport at the same time) and has security issues when 180 (mostly) women return from the bank carrying cash to their village. The program was working toward introducing electronic debit cards for beneficiaries to use in an ATM. However, ATMs are not currently available in the village. 2.14 The IEG mission also confirmed that good ground work had been done in implementing the PAISS+N under component 3 to improve children’s nutrition and growth. In the comarca Kuna Yala, background and cultural studies had been undertaken initially to make sure that interventions were culturally adequate to the indigenous people; researchers spent time with people in their houses to observe traditional customs, such as cooking, sleeping, breastfeeding, and caring habits. Educational material and training contents were developed accordingly. The women (beneficiaries) interviewed by the IEG mission in Kuna Yala confirmed that they could identify with the content of the material, and that their clothes, houses, and customs were adequately presented. Malnutrition, especially among infants and young children, was and still is a challenge in Kuna Yala; as part of the PAISS+N interventions, selected women (monitoras) from the villages were trained to monitor infant and under-five growth.14 2.15 However, the IEG mission identified several challenges affecting the work of the community health workers (monitoras). Each monitora was responsible for 5–10 families; their tasks were to visit the families regularly, to weigh and measure babies, and to give nutritional advice to mothers. Traditionally, infants have been breastfed exclusively until approximately 12 months and have been chronically malnourished (stunted) as a consequence. The monitoras advised giving the infants additional traditional food from 6 months (traditional food in Kuna Yala is nutritious and adequate for infants and toddlers). The intervention looked very promising with the potential of producing good results; the mothers followed the advice, and the infants and toddlers in the program gained weight and generally became healthier. In addition, the monitoras took the initiative to raise money for some of the neediest children. Part of the program plan was to remunerate the monitoras and pay them $25 per month as “allowance”; they could not receive a salary because they were not formally hired by the project. The Controlaría stopped the disbursement to the monitoras because by law allowance cannot be given to non-staff. Despite repeated efforts from project staff and from the UGSAF to find alternatives to remunerate the monitoras, they could not find a solution acceptable to the Controlaría. Consequently, the monitoras stopped working and this part of the project came to a halt. The monitoras interviewed by the IEG mission had enjoyed their work and expressed that they would happily start up again if the payments would come through. However, they would have appreciated recognition from the project staff, which they felt was lacking.15 Safeguards and Fiduciary Compliance 2.16 The Social Protection project triggered the indigenous peoples safeguard (OP 4.10), as it covered indigenous comarcas. The indigenous peoples plan (IPP) prepared under the project consisted of a brief social assessment, based on information collected at the local level in the indigenous comarcas. The IPP identified activities to ensure that the RO program benefited indigenous beneficiaries, including their participation in the program and that RO activities were 13 culturally appropriate. The main activities carried out were the creation of family committees (comites de familia) in the comarcas, and the significant presence of bilingual program staff among promotores and staff visiting the area on payment days. However, the ICR noted that the implementation of the IPP had very limited supervision until 2013, when many of its actions were already obsolete. It states that this limited supervision resulted in the IPP meeting a very low proportion of its original objectives (see World Bank 2015b, 10). However, the ICR did not indicate whether there was compliance with the triggered safeguard. The safeguard issue merits follow-up by the World Bank. 2.17 The Social Protection project maintained a satisfactory rating for financial management (FM) in the Implementation Supervision Reports (ISRs). FM arrangements, in terms of accounting, budgeting, flow of funds, internal control, and financial reporting were reported to be adequate in the ICR (World Bank 2015b). The project interim financial reports (IFRs) and external audit reports were submitted regularly and deemed acceptable. Specifically, (i) loan proceeds were used for the intended purposes, (ii) the project’s FM-related arrangements allowed an appropriate level of transparency that facilitated oversight and control while supporting smooth implementation, and (iii) the project’s legal FM-related covenants were met. The main FM-related issues identified during supervision were related to the project’s insufficient budget allocation approved annually by the Congress, which required the project implementation unit to request additional funds from the Ministry of Economy and Finance to ensure that all project activities were completed by the closing date. In addition, the financial controls provided by the General Comptroller of the Republic (Contraloria General de la República—CGR) were duplicative of the project’s FM control, which occasionally slowed implementation. 2.18 Procurement activities were consistently slower than planned, particularly during the first two years. As reported in the ICR (World Bank 2015b, 10–11), delays affected various aspects of the procurement processes: (i) preparation of the terms of reference, (ii) management of the calls for expression of interests, (iii) selection of candidates, and (iv) retention of consultants once they were contracted. The procurement of consultancy services, in particular, suffered because of limited understanding of human resource management aspects, an area that should be strengthened in future operations. Because of procurement delays key activities such as the development of the MIS and the impact evaluation of health interventions were not completed. IEG consulted the World Bank’s Integrity Vice Presidency (INT) database, which showed that none of the complaints related to the Social Protection project was subsequently substantiated. Achievement of Objectives OBJECTIVE 1. IMPROVE THE MANAGEMENT AND OPERATION OF THE RO 2.19 Outputs. The project produced the majority of the planned outputs.16 All modules of the MIS were implemented (for example, beneficiary roster, targeting system, payment system) with the exception of the module to verify co-responsibilities in education and health and to apply sanctions. As a result, some remote areas had significant differences between the data from official sectoral ministries and the RO information. To address the deficiency of the MIS, the project developed a verification strategy that could rely on different data sources depending on the region of implementation. However, the challenge of verification of co-responsibilities has remained difficult in remote areas. The MIS limitations resulted in reduced capacity to manage 14 the program, such as the inability to keep track of compliance among beneficiaries and to issue reports of noncompliance; more importantly, it resulted in the inability to systematically identify the reasons for noncompliance and to adequately respond with family accompaniment advice. The planned impact evaluation was another key activity of the component that was only partially accomplished. Because the roster of beneficiaries went through a natural aging of family members (children), the contracting delays of the impact evaluation affected the evaluation design, which needed to have enough potency in estimating certain indicators (for example, child malnutrition). As a consequence, while the data collection was successfully completed, the impact evaluation did not produce results and properly identify RO impacts. 2.20 The intermediate indicators related to the first PDO were achieved:17  Mechanisms of complaints operational (percentage of complaints and claims resolved in a timely manner). The complaints module of the MIS was operational during project execution. Reports based on complaints forms showed that all complaints and claims were resolved in a timely manner, surpassing the target of 70 percent.  Specific sections of the Operational Manual cover program operation in indigenous areas. This indicator was achieved during implementation. However, while the ICR considered it a PDO-level indicator, the IEG review is considering it an intermediate product-level indicator.18 2.21 Outcomes. During the project implementation period, thanks to project support, the management and operation of the RO improved. Overall, the targeting of the RO program exceeded expectations. Data from the 2013 household survey shows that 73.1 percent of the RO beneficiaries belong to households in the first quintile of income distribution and 92.5 percent belong to the first two quintiles (World Bank 2015b, 17). According to the World Bank’s Atlas of Social Protection Indicators of Resilience and Equity,19 the RO is one of the best-targeted CCTs worldwide (World Bank 2015d, 50). Even if RO covers only about 34.5 percent of all poor in Panama, it manages to reach about 69.5 percent of the indigenous population. A number of outcome indicators related to PDO 2 were achieved during implementation:  Percentage of disbursed cash transfers received by households in quintiles 1 and 2. Data from the 2013 household survey show that 92.5 percent of the RO beneficiaries belong to households in the first two quintiles of income distribution, exceeding the target of 80 percent.  Percentage of households living in indigenous jurisdictions receiving transfers (as a share of total number of households in indigenous jurisdictions). The baseline coverage among the poor indigenous jurisdictions was 50 percent, reaching 69.5 percent, thus substantially reaching the target of 70 percent.  Percentage of RO beneficiaries’ children 4 to 17 years old who are enrolled and attend school in compliance with their education co-responsibility. The baseline was 50 percent, reaching 91.8 percent, thus surpassing the target of 90 percent.  Proportion of bimonthly payments based on a complete cycle of co-responsibilities verification and application of sanctions. The indicator was not achieved; the MIS supported by the program was not fully functional and could not be used to verify of co- responsibilities. 15  Number of jurisdictions (corregimientos) covered by RO: The coverage of the program expanded from 503 to the target of 621 corregimientos in 2009 and maintained the same coverage throughout the program.  Number of households benefiting from the RO. The RO expanded from 50,000 families in 2007 to reach the target value of 75,000 families in 2010. However, after reaching the target, the natural attrition in the roster of beneficiaries20 and the lack of new enrolment rounds contributed to a diminishing number of households covered by the program. The total number of households benefiting from RO was 74,481 in 2011, 72,485 in 2012, 72,534 in 2013, 72,895 in 2014, and 72,563 in November 2016. 2.22 Overall, there were issues with the verification aspects of the RO program. As noted earlier, MIS limitations resulted in reduced capacity to manage the program, such as the inability to keep track of compliance among beneficiaries and to issue reports of noncompliance; more importantly, these limitations resulted in the inability to systematically identify the reasons for noncompliance and to adequately respond with advice to the family. 2.23 The achievement of objective 1 is rated Substantial. OBJECTIVE 2. INCREASE BENEFICIARIES’ PARTICIPATION IN THE RO, PROMOTING THEREWITH DEMAND FOR EDUCATION AND MATERNAL AND INFANT HEALTH SERVICES 2.24 Outputs. The project implemented the planned strategies to help families participate in the program and achieve full access to its benefits. Direct support was provided to participating families, and targeted communication efforts were tailored to different social and cultural conditions. As a result of the justified delays in the selection and contracting of promotores and the subsequent budget, only about 389 out of 586 territorial units (corregimientos) were covered by October 2013. Promotores visited 58,749 out of 64,194 beneficiaries, providing tailored support and information about the RO program including their rights and responsibilities. A nationwide information campaign was launched to inform beneficiaries of their rights and responsibilities. The program also provided training to all education and health service providers. The project achieved two out of three intermediate indicators related to the second PDO:  Share of jurisdictions covered by the RO with technical support teams for the accompaniment of families. Promotores reached about 66 percent of the corregimientos instead of the 100 percent target.  Share of education service providers trained on RO (Ministry of Education personnel from central offices, provincial offices, and schools). The 100 percent target was reached.  Share of health service providers trained on RO (MINSA personnel from central offices, provincial offices, and health care providers). The 100 percent target was reached. 2.25 Outcomes. As a result of the culturally appropriate direct support and targeted communication provided by the program about 76 percent of the beneficiaries knew their right to access services and their co-responsibilities, against a target of 70 percent. However, no conclusive evidence was identified regarding either increased participation or regarding the demand for health services. 16 2.26 A semistructured review of the literature was performed to identify and extract evidence from the academic literature on the effects of those participating in the RO program to the demand for education and health services.21 The review identified evidence that even if RO was implemented as a semiconditional cash transfer program (see World Bank 2015b, 8), participation in the program has a statistically significant positive impact on enrollment22 comparable to the other CCT programs implemented around the world (Baird et al. 2013, 69; Robles, Rubio, and Stampini 2016, 8). For example, the review of the impact of CCT programs by Ranganathan and Lagarde (2012) makes reference to the study by Arraiz and Rozo (2010) that analyzes the 2008 Panama Living Standards Measurement Survey and finds “no impact on visits to health care providers in both rural and indigenous areas.” The review of the literature did not identify a more recent evaluation of the RO that assessed if the improvement in the management and operation of the program had, in turn, increased the demand for education and health services.23 2.27 Since increased knowledge about the beneficiaries’ rights and responsibilities is not necessarily evidence of increased beneficiaries’ participation. 2.28 The achievement of objective 2 is rated Modest. OBJECTIVE 3. STRENGTHEN THE SUPPLY OF GROWTH PROMOTION INTERVENTIONS IN THE AREAS TARGETED BY THE RO 2.29 Output. The basic health package strengthened with the promotion of community-based growth and interventions to prevent malnutrition (PAISS+N) was designed and delivered in the indigenous comarcas. 2.30 Outcomes. The PAISS+N strengthened the supply of growth promotion interventions in the indigenous areas. After some initial delays in contracting the OEs and OI in Kuna Yala, the program provided regular visits to the indigenous communities. In the last year of the program, 90 percent of registered children were receiving the basic health care package. However, while 74 percent of the pregnant women in the comarcas were receiving at least 3 prenatal controls in 2011, that trend declined to 69 percent at the end of the program in 2014. The key indicators for objective 3 are the following:  Reduction of chronic malnutrition prevalence among children under two years old registered in the PAISS+N in indigenous comarcas.24 The preliminary data of the impact evaluation showed results in the expected direction: a 3.7 percentage point reduction in chronic malnutrition versus a target of 4 percentage points.25 However, the results of the impact evaluation did not show statistical significance or sufficient precision to measure the level of malnutrition reduction among the target population because of limited number of children under 24 months old identified.  Percentage of children under two years old registered in the PAISS+N program benefiting from the strengthened health care package. Ninety percent of children registered in the PAISS+N received the strengthened health care package. UGSAF indicated that results at the national level reached 98 percent in the last quarter of 2014, and 95 percent in the first quarter of 2015, but it was not able to provide coverage data for the comarcas after program completion. 17  Additional coverage of pregnant women by PAISS+N receiving no less than three prenatal controls, among pregnant women in the comarcas. Coverage increased from 60 percent to 69 percent but did not reach the target of 85 percent. 2.31 Information on chronic malnutrition among children living in indigenous comarcas was sought to perform difference and double differences. Table 2.2 summarizes the relevant population health surveys conducted in Panama.26 Unfortunately, it is not possible to compare the prevalence of malnutrition in indigenous comarcas because of the inconsistency across studies in the age groups in the reference sample (for example, the project refers to chronic malnutrition among children under two years old, but other studies refer to children under five years old) and the limited number of studies that measured chronic malnutrition in the indigenous comarcas covered by the project. 2.32 Information on exclusive breastfeeding among children living in indigenous comarcas was sought considering the known association between breastfeeding practices and stunting (Jones et al. 2014). The 2009 National Survey of Sexual and Reproductive Health of Panama (Encuesta Nacional de Salud Sexual y Reproductiva—ENASSER 2009) provides baseline value for the comarcas Kuna Yala and Ngöbe Buglé, but not for Emberá Wounaan, which was not covered by the survey (ICGES 2011). The end line is provided by the multiple indicator cluster survey (MICS) conducted in Panama in 2013 (CGR 2014).27 Difference and double differences (DD) from the national average in exclusive breastfeeding are presented for Kuna Yala and Ngöbe Buglé in table 2.2. Even if the results summarized here are not always conclusive, they suggest an overall improvement of exclusive breastfeeding practices among indigenous women living in the indigenous comarcas that received support under the program. 2.33 The achievement of objective 3 is rated Substantial. Table 2.2. Evolution of Chronic Malnutrition and Exclusive Breastfeeding Indicators in the Indigenous Comarcas Covered by the Program during Implementation Exclusive breastfeeding Prevalence of chronic malnutrition among children