Report No: AUS8365 . Republic of the Philippines Completion Report for the Technical Assistance “Impact evaluation of the Philippines CCT Program� . November 11, 2016 . GSP02 EAST ASIA AND PACIFIC . . Document of the World Bank Standard Disclaimer: . This volume is a product of the staff of the International Bank for Reconstruction and Development/ The World Bank. The findings, interpretations, and conclusions expressed in this paper do not necessarily reflect the views of the Executive Directors of The World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. . Copyright Statement: . The material in this publication is copyrighted. Copying and/or transmitting portions or all of this work without permission may be a violation of applicable law. The International Bank for Reconstruction and Development/ The World Bank encourages dissemination of its work and will normally grant permission to reproduce portions of the work promptly. For permission to photocopy or reprint any part of this work, please send a request with complete information to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA, telephone 978-750-8400, fax 978-750-4470, http://www.copyright.com/. All other queries on rights and licenses, including subsidiary rights, should be addressed to the Office of the Publisher, The World Bank, 1818 H Street NW, Washington, DC 20433, USA, fax 202-522-2422, e-mail pubrights@worldbank.org. 2 PHILIPPINES SOCIAL PROTECTION AND LABOR SUBSTASK: IMPACT EVALUATION OF THE PHILIPPINES CCT PROGRAM (P150532) PROGRAMMATIC ANALYTICAL AND ADVISORY ASSISTANCE FY2014-FY2017 (P148753) COMPLETION SUMMARY1 The first Social Protection and Labor (SPL) Programmatic Analytical and Advisory Assistance (PAAA) in FY2012-14 successfully solidified the Bank’s role as the Government’s partner of choice in delivering high quality technical assistance and advice related to the strengthening of the Government’s flagship programs, such the Pantawid Pamilya Conditional Cash Transfer Program, as well as to the policy and institutional capacity development aimed at efficient and effective delivery of public programs to the population. The technical assistance provided by the Bank has contributed significantly to the transformation of social protection in the Philippines. Building on the achievements and the momentum of that first programmatic engagement, this second phase of assistance to the Department of Social Welfare and Development continued the support to (i) fine-tune the implementation of the Conditional Cash Transfer (CCT) program, the Pantawid Pamilyang Pilipino Program (Pantawid Pamilya), and Government’s household targeting system, the Listahanan, and (ii) broaden the knowledge base to inform advancements in the Philippine’s overall social protection system. The assistance was provided through the second SPL PAAA that was implemented over FY2014-17. The activities were significantly supported through a trust fund resources provided by the Australian DFAT. This technical assistance program has contributed to the refinement of several key implementation aspects of the Pantawid Pamilya CCT and the national household targeting system, Listahanan. The CCT coverage was expanded from 3.8 million beneficiary households in December 2013 to 4.4 million in December 2015, accompanied by an expansion to cover older children (15-18 years old) beyond the original eligibility of 0-14 years old. By the end of 2015, the CCT coverage reached 100% of households with children identified as poor by Listahanan. The Impact Evaluation conducted undet this TA, as well as a Benefit Incidence Analysis using household survey data showed that the CCT program is performing very well in delivering on its objectives. The heightened attention received by the CCT program over these years from program beneficiaries, partners, legislators, and observers has not only led to enhancements in the program’s grievance redress mechanism but also to more and better articulation of evidence of the program’s impacts. 2015 marks the first update of the Listahanan since it was first done in the Philippines over 2007-2010. Overall, this program has contributed strongly to the client’s successful achievement of remarkable results in implementing the CCT program and in establishing and operating one of the most comprehensive national household targeting systems in the world. 1 This report was prepared by Aleksandra Posarac, Lead Economist, Pablo Acosta, Senior Economist, and Rashiel Velarde, Economist, from the Social Protection & Labor Global Practice. It reflects the outputs and results achieved by the programmatic analytical and advisory activities delivered by the Social Protection & Labor Team of the World Bank Manila Office, which comprised of WB Staff and Consultants listed in Appendix 1. 3 I. BACKGROUND The Government of the Philippines has made significant progress in developing its social protection sector over the last 10 years, which has been one of its main strategies to address poverty and vulnerability. The most notable developments had been the introduction of a national household targeting system (more popularly known as Listahanan), a conditional cash transfer (CCT) program, and the universal health care (UHC) program that reaches out to the poorest Filipino families, i.e., those who belong to the bottom 40 percent as identified in the Listahanan. The flagship social protection program is the Pantawid Pamilyang Pilipino Program, a CCT program. It was introduced to support poor families’ investment in health and education of children 0-14 and to alleviate their poverty. It has rapidly grown from 360,000 beneficiary households at the time it was launched in 2008 to 3.8 million poor households in 2013. By then, the Pantawid has become the world’s third largest CCT program – covering nearly 20 million people or 75% of the poor population and with a budget that reached 0.4% of GDP. Free health insurance and access to primary health care has been extended to all the Pantawid beneficiaries, as well as to other families that belonged to the bottom 40 percent of the population, as part of the UHC program. Both the Pantawid and UHC programs benefited from the Listahanan that was established in 2009. The Listahanan contains information of 11 million households nationwide (about 60 percent of the population), of which 5.2 million households were classified as “poor� and prioritized for various Government assistance programs. The World Bank, jointly with Australian Aid (DFAT), first supported the Government’s ambitious social protection agenda through a series of discrete technical assistance grants from 2006 through 2010. This was followed by the first Social Protection Programmatic AAA (SP PAAA) that covered FY2010-FY2013. The Bank’s assistance throughout this foundational period for the Philippine SP system has been instrumental, particularly in the areas of poverty targeting, CCT implementation and monitoring, and rigorous analytical work to inform the broader policy dialogue on social safety nets and poverty reduction.2 The second SPL PAAA built on the results achieved in the first seven years of close collaboration with DSWD and DFAT. It sought to continue the technical support and advice to the Government in further strengthening its flagship SP programs while at the same time open new areas of engagement beyond social assistance and targeting. While the foundational management information systems and procedures had already been established and functioning in support of Listahanan and the world’s third largest CCT program, the Pantawid Pamilya, to deliver on their respective objectives, DSWD was keen to ensure that these systems continued to be strengthened and primed to meet evolving demands. The Government remained keen on learning more about how to make its CCT program more effective using rigorous evaluations and in-depth process assessments to inform necessary recalibration of the current program design and implementation. In 2 The achievements from the first SP PAAA (P118853) is summarized in the Completion Report in June 2015 (http://imagebank.worldbank.org/servlet/WDSContentServer/IW3P/IB/2015/07/29/090224b083032f5f/1_0/Render ed/PDF/Philippines0000nd0completion0report.pdf). 4 addition, attention to other components of the social protection system was just as important. Thus, the second phase of technical assistance was designed to continuously provide high quality technical support to DSWD in its efforts to enhance the design of the CCT program and Listahanan and strengthen their operation systems and business processes. Also, the TA was designed to open the dialog in insurance/pensions and to further Bank engagement in the labor, skills, and jobs agenda. Relevance of the Program This program was aligned well with the World Bank Group (WBG) twin goals of poverty reduction and shared prosperity. It directly contributed to the achievement of the objectives of the World Bank’s Country Partnership Strategy for the Philippines, specifically in Engagement Area 2: Empowering the Poor and Vulnerable. Besides contributing to the achievement of better health and education outcomes for poor children, it also informed and assisted DSWD and other policy making agencies in efforts to improve program’s performance by taking actions to continuously improve its implementation. More importantly, the PAAA was focused on strengthening the social protection as a whole - SP played an important role in the Philippine Government’s overall strategy to reduce poverty and boost shared prosperity. With its long and excellent tradition and track record in social protection, the WBG was uniquely positioned to conduct this PAAA. The program was a continuation of technical assistance which laid the foundations for the core safety net program, and continued to guide the dynamic evolution of the social protection system. II. OBJECTIVES AND INTENDED RESULTS The SP&L PAAA primarily benefited the Department and Social Welfare and Development (DSWD), the main implementing agency for the CCT program and targeting system. It also benefited other government agencies involved in SP, such as the State Social Security System (SSS), the Department of Labor and Employment (DOLE), NEDA, DBM and others Specifically, the Concept Note (CN) for the SP&L PAAA articulated the following objectives: (i) Assist DSWD sustain the role of the CCT program as a well-established and implemented backbone of the SSN in the Philippines. This will be achieved by further strengthening of the CCT operations and management, including identification of beneficiaries, eligibility and compliance verification, grievance and redress and payment delivery, MIS integration, monitoring and evaluation capacity and expansion to cover poor children in secondary education. (ii) Advance the implementation of the Government’s convergence agenda by broadening and deepening the knowledge base for overall social protection policy dialog to include social security and labor market policies. In doing so, the program committed to contribute to achieving the following outcomes: 5 Intermediate outcome Indicators Government policy/strategy informed, Public 1. Policy/strategy debate stimulated/initiated, Bank country strategy informed informed Design capacity strengthened, Implementation 2. Client capacity capacity strengthened, Monitoring & Evaluation increased capacity increased 3. Knowledge Facilitated exchange of best practice w/ clients deepened To deliver on this ambitious program, a combination of just-in-time technical assistance on the daily operation and troubleshooting of DSWD programs, discrete analytical products, and capacity building workshops was designed over a 3-year period (FY14-FY16) to benefit mainly DSWD management and technical staff. All activities in the programs have been discussed and agreed on with the counterparts in DSWD and in consultations with DFAT staff. III. SUBTASK: IMPACT EVALUATION OF THE PHILIPPINES CCT PROGRAM (P150532) Building upon the successful conduct of the first round of the Impact Evaluation (IE) in 2012 supported under the first phase SP PAAA, this pillar continued the technical support to DSWD in the design, supervision, and analysis of the second IE of the Conditional Cash Transfer Program and to document its impact on key outcomes related to health and education. This emanated from the strong demand from DSWD for continued rigorous impact evaluation of the biggest program of the Department to justify its continuation, but also the demand for continued knowledge transfer on how to conduct and analyze rigorous IEs. The objectives of this task were to: (i) undertake capacity building in DSWD and in a team of local researchers for the second round IE design, implementation and data analysis; and (ii) provide high-level technical assistance to teach and guide relevant DSWD staff and local researches to analyze this data set, with the assistance of global experts and local researchers. At the request of DSWD, an IE expert was hired to provide continuous hands-on technical advice, guidance and training, throughout the complete cycle of the second round CCT IE. Two large impact evaluation data sets from the first round IE comprising more than 7,500 households were thoroughly revised and cleaned by research assistants (consultants) under technical guidance and supervision of WB staff. This task was performed jointly with DSWD technical staff, as one of the main goals of this activity was to teach DSWD relevant 6 staff, as well as to document good and bad practices when collecting and encoding household survey data. Presentations on steps that were taken to clean the IE data and on good and bad practice found while reviewing and cleaning the data sets were given to DSWD technical staff. Documentation of good practice is expected to help DSWD in the conduct of the third round of the IE data collection and in other data collection efforts. The cleaned first round IE data sets are expected to be published online by DSWD for further research. DSWD’s demand for technical assistance in all steps of conducting a quantitative impact evaluation was met by providing adequate, hands on and on-time guidance throughout the IE process, from survey design, conduct of sampling and data collection supervision to technical review and assistance in publication of data and dissemination of results. Below are some of the activities supported by the TA: WB and DSWD technical staff carried out the sampling for the Regression Discontinuity and cohort analysis studies of the second round IE. The joint exercise allowed DSWD to participate in and provide inputs to the sampling of households and at the same time learn how to conduct complex sampling exercises. In addition, a presentation of all processes was done for other DSWD staff, IE technical Working Group and local IE research team conducting the second round analysis.  WB and DSWD technical staff were heavily involved in the collection, encoding and cleaning of second round IE data which surveyed 9000 households in 30 provinces of the Philippines. The vast data collection work performed by the survey firm was complemented by the supervision of WB and DSWD technical staff. Other activities that were undertaken focused on TA to improve the encoding software, encoding methods and cleaning techniques used by the IE survey firm, resulting in good quality data on time for analysis.  Technical meetings and workshops with DSWD technical staff and IE local research team were conducted in order to plan and perform the analysis of the second round CCT impact evaluations. Direct transfer of knowledge on the technical aspects of IE and close guidance in interpreting and policy implication of results were provided to DSWD staff. In addition, a technical review focusing in the technicalities of the two methodologies used for analysis was held with DSWD technical staff and the local research team. The technical review, which benefited from the participation of a WBG senior international IE expert from DEC, served three main purposes: 1) to assure sound technical procedures in the conduct of the IE analysis, 2) to capacitate further the DSWD technical staff and the local IE team (particularly important given that same methodologies will be used for the third round IE), and 3) to brainstorm about possible design and research questions for the third round IE.  The nationally representative second round IE report, Keeping Children Healthy and in school: Evaluating the Pantawid Pamilya Using Regression Discontinuity Design was reviewed by senior WBG IE and SPL experts who discussed technical and policy relevant issues related to the implementation of the CCT program. The review 7 also conveyed international good practice experience to DSWD and local research team.  The challenging task of transforming technical results into laymen terms was undertaken by WB communication staff, a communication STC and an economic editor. WB consultants together with the DSWD communication staff were able to effectively present and disseminate the findings and recommendations of the IE report in legislative, academic and public fora. The findings from the IE analysis and related implementation and policy recommendation are now being considered by policy makers and implementers.  WB technical staff has provided technical guidance and advice on the data analysis methods for the second IE report, Impact of Time-Critical Household Investments in Pantawid Pamilyang Pilipino Program, which uses a cohort analysis approach to examine the impact time-critical inputs of Pantawid Pamilya on select health and education outcome variables. Additionally, four quantitative reports complementing the first round RCT IE report were prepared.  A spillover effect analysis has helped quantify and understand the magnitude of potential spillover effect of the CCT program. An important complement to the IE report, the findings of negative externalities need to be taken into consideration in order to fine tune program implementation.  A CCT regional/provincial heterogeneity report. This spatial analysis informed program implementers about the variation in effectiveness of the program as noted in the first RCT report.  A nutrition report has estimated the effects on anthropometric indicators, health service utilization (for 6-36 month olds) and dietary intake (for 6-60 month olds). Pantawid Pamilya was associated with a significant reduction in severe stunting as well as marginally significant increase in HAZ scores.  A child labor report contributes to understanding the relationship between schooling and work-for-pay and to an assessment of the cost effectiveness of transfer schemes. The study finds that the increase in school participation was accompanied by an increase in children’s work-for-pay. The program appears to have encouraged children who would otherwise be idle to attend school and to start working. In addition to the impact evaluation support provided to DSWD, this task also enabled the SP team to continue providing hands-on technical assistance in monitoring the implementation of the CCT through spot checks with the objective to inform implementation arrangements and build technical capacity within DSWD for M&E. Spot checks provide the Pantawid Pamilya National Program Management Office with immediate and short term insight into program performance, offer timely monitoring assessments of the quality, effectiveness and efficiency of program business processes, and identify bottle necks in service delivery. Specifically, this TA required World Bank staff to help design the spot checks sampling, tools, methods, and process data collection using Stata (statistical software). World Bank support on spot checks worked concurrently to build capacity in 8 DSWD in conducting its own subsequent internal spot checks. Beyond the conduct of spot checks, TA and hands-on transfer of knowledge were provided to ensure the effective use of the spot check findings to inform policy and program policies. A follow up task also performed under this pillar is a video-documentary with the history and main results of Pantawid Pamilya CCT, narrating the story of three beneficiary groups. The documentary was produced by a well-known director in the Philippines, Francisco “Paco� Guerrero, who was also responsible for the photography and exhibition at the Gala premier at the Ayala Museum in Manila in June 2016. Several high-level Government authorities and development partners attended the Gala premier. Four spin-off short storyline videos were also produced for wider dissemination in public channels on the impact and achievements of the Pantawid CCT program. IV. RESULTS ACHIEVED The positive results from the second impact evaluation and spin-off papers (on nutrition, spill-over effects, regional heterogeneity, and child labor) solidified support for the continuation of the CCT program as an effective investment to break the cycle of poverty. DSWD Secretary have referred to the findings in Cabinet, as well as other policy makers and academia in the media, to pledge for program expansion and continuation over the last administration. Among other things, the second IE highlighted that the CCT continues to deliver on its development objective of keeping children healthy and in school. This finding was consistent with the results of the first IE that was released in 2012. In two rounds of IEs, the CCT program proved to sustain its developmental impact on poor households since it was piloted in 2007 and its effectiveness as a long-term investment for the Government to combat poverty. It has also helped dispel critics’ claim of the CCT as a mere “dole-out� program that does not deliver any lasting positive impacts on the poor. On the contrary, results from this rigorous evaluation, coupled with direct beneficiary feedback, helped DSWD convert former critics of the program into staunch supporters. Support from other government agencies, academics, opinion-makers, and legislators manifested in the CCT program’s continued expansion and budget until 2016. The program has also gained the support of the new administration, which started in July 2016, with no less than the President and Vice-president publicly promoting to continue the program. Sustained mentoring and technical assistance strengthened staff capacity within DSWD in monitoring and evaluating large programs. Unlike the first IE of the CCT program which principally used a randomized control trial (RCT) design, the second IE employed a new evaluation methodology (regression discontinuity design, RDD). This required consistent hand-holding and intensive mentoring to DSWD staff under the Project Monitoring and Evaluation Division (PMED) of the Pantawid PMO to ensure that they fully grasp the use and implications of using the RDD methodology for their own dialog with DSWD Management (primary audience of the IE report) and other stakeholders external to DSWD (secondary audience). TA was provided to DSWD on how to clean and prepare datasets for analysis, regular discussions with PMED were done to explain step-by-step procedures in undertaking the IE analysis, and in interpreting policy implications of results. 9 Beyond DSWD, this task also contributed to broadening technical expertise of the pool of local researchers in undertaking impact evaluations. While some expertise on impact evaluations exist in-country, the use of RDD methodology has not been widely used by local academics and researchers until this second round of IE for the CCT program. An international expert on impact evaluations from the Bank’s Development Economics Group (DECPI) and another international expert from academia provided guidance and technical support to the local research group hired by DSWD to prepare the second IE report. The technical guidance from the IE experts served three main purposes: (i) to assure sound technical procedures in the conduct of the second IE analysis based on international best practice experiences; (ii) to capacitate further the DSWD technical staff and the local research team, which was particularly important given that same methodologies will be used for the third round IE; and (iii) to brainstorm about possible design and research questions for the third round IE. Workshops were held with the local research group, which consisted of a 1 Senior Economist and 4 Economists and Data Analysts, to discuss methodological details of undertaking RDD analysis and peer review results. It is hoped that the third round of IE for the CCT will further intensify participation and involvement of local researchers and academics in all aspects of the study. This pillar also helped build capacity of DSWD to perform spot checks, including capacity to execute sampling for spot check surveys. The pillar provided technical oversight not just to ensure that the spot checks were carried out to a high standard, but that the findings are indeed understood and used to improve policy and program review. Finally, this pillar also supported packaging key messages of the IE report and CCT video for the broader audience, which include media and the general public. A communication team, consisting of an Editor and a Communication Specialist, was hired to assist in finalizing the report and packaging information materials to disseminate its key findings, and in disseminating results through audio-visual means. The communications team assisted in finalizing presentation materials with the aid of infographics to ensure that results are presented as accurate yet palatable to the general audience as possible. 10 APPENDIX: SUBTASK MAIN OUTPUTS (Saved in WBDOCs) 1. Second IE Report of the Pantawid Pamilya CCT http://wbdocs.worldbank.org:/wbdocs/drl/objectId/090224b0846ddfea 2. CCT and Child Labor IE Report http://wbdocs.worldbank.org:/wbdocs/drl/objectId/090224b0846ddf78 3. CCT and Heterogeneity Effects IE Report http://wbdocs.worldbank.org:/wbdocs/drl/objectId/090224b0846ddf79 4. CCT and Nutrition IE Report http://wbdocs.worldbank.org:/wbdocs/drl/objectId/090224b0846ddfe9 5. CCT and Spillover Effects IE Report http://wbdocs.worldbank.org:/wbdocs/drl/objectId/090224b0846ddf7a 6. CCT Video Documentary: http://www.worldbank.org/en/news/video/2016/08/01/conditional-cash-transfers-in-the-phili ppines-nurture-dreams 11