77730 MAY 2013 • Number 117 From Noise to Signal: The Successful Turnaround of Poverty Measurement in Colombia João Pedro Azevedo In the mid-2000s, poverty measurement in Colombia was at a standstill. A dated poverty measurement methodology was clashing with improvements in the national household survey system. As a result, official poverty rates showed volatile trends, and a weak communication strategy produced an unconvincing storyline, which further resulted in the rapid deterioration of indicator credibility. This happened during a period of high and sustained growth that also included a number of poverty reduction interventions, such as the flagship program Familias en Accion and the Unidos strategy. The public debate on poverty lost focus and moved from substantial policy discussions to technical measurement methods. This note summarizes the main lessons learned from govern- and the Bank to focus their attention on a more fruitful dis- ment efforts, supported by national and international ex- cussion of poverty trends and policy implications. perts, including the World Bank, to restore credibility to the In 2006, the Colombian National Department of Ad- official methodology for poverty measurement in Colombia. ministrative Statistics (DANE) sought to improve its primary These lessons are valuable for both low- and middle-income household survey (the Encuesta Continua de Hogares, ECH), countries that might need to face the difficult task of revising and introduced a series of changes. The revised survey was in- their official poverty methodology, or any other politically troduced as the Gran Encuesta Integrada de Hogares (GEIH).1 sensitive indicator (see box I for a summary of the lessons However, important methodological steps to ensure that learned). these changes did not affect the comparability of the data The World Bank’s engagement in this process had four with previous surveys were not taken. When such changes are primary objectives: (i) ensure that poverty figures are backed made to household surveys, it is highly recommended that by international practices; (ii) help build interagency consen- the older methodology be simultaneously implemented along sus and foster transparency; (iii) lend credibility as an active with the new methodology to understand how the new meth- member of a poverty committee; and (iv) share knowledge odology affects measurement. Without a counterfactual, gained through participation in similar regional experiences. there is no way to detangle changes in data results attributed Providing clarity and transparency on the technical challenges to the change in methodology versus those attributable to ac- of poverty measurement enables both country governments tual changes in the realities of the people surveyed. 1 POVERTY REDUCTION AND ECONOMIC MANAGEMENT (PREM) NETWORK    www.worldbank.org/economicpremise Box 1. Summary of Lessons Learned Box 2. Anonimized Microdata Going Public Persistence: The process was time intensive; the MESEP In November 2011, DANE announced the launch of its Committee met once or twice per month, from three to six anonimized microdata portal. hours per meeting, for 32 months. http://formularios.dane.gov.co/pad/ Scope: Focus on the poverty measurement methodology, DANE’s online microdata portal uses the International and not on the poverty reduction strategy. Household Survey Network (IHSN) developed National Data Media and public perception of the new statistics: Showcas- Archive (NADA). The metadata produced is compliant with ing a transparent consensus-building process signaled the Data Documentation Initiative (DDI) 2.n and the Dublin credibility to the public, which contributed to wide media Core XML metadata standards, and was produced using the interest and acceptance of poverty measures. Despite previ- IHSN Metadata Editor developed by Nesstar at the Norwe- ous contention, poverty estimates from 2002–8 were ac- gian Social Science Data Archive (NSD) and distributed as cepted by experts and residents throughout the country; the freeware. results of the 2nd phase of the MESEP were also widely ac- This initiative was made possible by the continuous support cepted. of the Accelerated Data Program (ADP) to DANE. The ADP Sustainability: The committee was legally formalized with an is a joint imitative of Paris 21 and the World Bank to promote inter-institutional agreement, January 27, 2009 (DANE-DNP and support international best practices in microdata docu- 2009a). mentation and worldwide dissemination. Source: Author’s compilation. Institutional commitment: This process was overseen by three different directors of DNP (Carolina Renteria, Esteban Piedrahíta, and Hernando José Gómez), two directors of dent experts. The MESEP’s objective was to facilitate a par- DANE (Héctor Maldonado and Jorge Bustamante), two dep- uty directors from DANE (Christian Jaramillo and Carlos ticipative and transparent process to generate reliable poverty Sepúlveda), and two presidents (Uribe and Santos adminis- estimates for Colombia. MESEP work was divided into two trations). This demonstrates that Colombia has the neces- phases/tasks: the first phase’s objective was to respond to the sary capacity and long-term commitment. immediate demand for reliable poverty estimates; the second Openness: Transparency was seen as a critical prerequisite phase generated the space for the experts to produce sound to build credibility for the new numbers. The MESEP Com- recommendations for a more in-depth revision of the poverty mittee recommended that both the old and new poverty se- methodology. ries should be computed and disseminated in parallel for a minimum period of two years. In addition, DANE decided to MESEP Phase 1: Data Reconciliation release the anonimized microdata used to compute the new poverty numbers to the general public through its Web site The MESEP’s phase 1 objective was to reconcile data pro- (see box 2 for more details). duced by the GEIH using the Encuesta Continua de Hogares Source: Author’s compilation. Paralela. The MESEP generated statistics for the period be- tween 2002 and 2008, making them comparable through time, adjusting for the changes in the survey methods insti- Given the complex and difficult-to-communicate im- tuted in 2006, but with no major changes in the poverty mea- pacts of the new methodologies on the poverty measures, the government of Colombia decided not to publish their official surement methodology. The MESEP Committee met 20 quarterly poverty statistics beginning in the second semester times between January 2009 and August 2009 to ensure the of 2006 through the end of 2007. During this period, Colom- completion of this phase, and published the statistics in Au- bia had experienced significant economic growth, yet given gust 2009. the new changes in the methodology, there were concerns The MESEP was able to draw several important conclu- that the new poverty measures did not fully reflect the im- sions from the first phase that contributed to its work going proved economic performance. Credibility in the poverty es- forward. Experts concluded that the new survey methodolo- timates declined, and the debate moved away from the ac- gy implemented in the GEIH did have a positive effect on data tions needed to reduce poverty to how to better measure it. quality: the changes represented an improvement in captur- In January 2009, an inter-institutional committee of ing the reality of people surveyed.2 Going forward, the MESEP poverty experts was formally launched to ensure a transpar- Committee decided to anchor the new variables in the GEIH ent process for data reconciliation. The Misión para el Em- (the newer methodology) and adjust the variables used in the palme de las Series de Empleo, Pobreza y Desigualdad (MESEP) ECH (the previous survey) from 2002–5, according to appro- included participants from the Department of National Plan- priate comparability scales derived from the Encuesta Conti- ning (DNP), DANE, United Nations Economic Commission nua de Hogares Paralela. The MESEP committee concluded for Latin America and the Caribbean (ECLAC), and the that reporting data/statistics for 2006 and 2007 would be World Bank as well as Colombian academics and indepen- imprudent given the methodological changes that took place 2 POVERTY REDUCTION AND ECONOMIC MANAGEMENT (PREM) NETWORK    www.worldbank.org/economicpremise in those years. In August 2009, the MESEP published data ed following one of the main recommendations of the MESEP, from 2002, 2003, 2004, 2005, and 2008. namely, the electronic publication of the anonimized micro- data (see box 2 for more details on the Colombia Microdata MESEP Phase 2: Revision of the Colombian dissemination) and algorithms required replicate the revised Poverty Measurement Methodology poverty methodology (DANE 2011). The poverty measurement methodology being used by Co- In addition, the National Council for Social and Eco- lombia had not been updated since 1998, and at that time nomic Policy (CONPES), the highest level economic and so- was using data from the 1994/1995 Household Budget Sur- cial policy coordination body in Colombia, published specific vey to construct its poverty line. Since then, Colombia has guidelines on how monetary poverty in Colombia should be conducted a newer Household Budget Survey (2005/2006) measured going forward, and more importantly, what the and moved to a continuous and integrated system of house- specific roles of the National Office of Statistics, Ministry of hold surveys, namely the GEIH. Given the dated poverty Planning, and Ministry of Social Protection should be in this methodology and the availability of a newer Household Bud- process (CONPES 2012). To strengthen the credibility of the get Survey, the work of the MESEP’s second phase focused on new series, the experts also recommended parallel publica- the full revision of the poverty measurement methodology. tion of both the old and new numbers for at least two years to The Colombian poverty line had historically been set mitigate any speculation regarding the political convenience higher than the regional average. An examination of the pur- of the methodological revision. chasing power parity (PPP) equivalent of the national poverty Going Forward lines in Colombia reveals that they are roughly twice the aver- age of those used in the rest of Latin America. The Colombian In October 2012, DANE launched a new poverty committee poverty line has been an outlier since it was first produced. and started to work within the revised framework, which However, this choice had never resulted in a significantly high- clearly attributed DANE as the producer of the indicators; er poverty rate because the government of Colombia had also DNP as the validating institution responsible for replicating followed the practice of adjusting the income data to the na- the calculations; and the Ministry of Social Protection (DPS) tional accounts, which resulted in reduced poverty statistics— as a critical stakeholder responsible for designing Colombia’s reduced from the high levels that would have been generated poverty reduction strategy. with the abnormally high poverty line. The effect of making As the monetary poverty measurement methodology in the adjustment is not trivial. Comparing adjusted and unad- Colombia has gained credibility, the country has been able to justed data showed that in some time periods, the trends have measure its progress and focus on new challenges, such as its opposite signs, given important differences between the sys- high inequality and how to promote prosperity for all. Co- tem of household survey and the national accounts (Ravallion lombia also used the institutional arrangement developed un- 2003; Korinek, Mistiaen, and Ravallion 2006). More impor- der the MESEP to regulate the production and dissemination tantly, such adjustment, based on a multiplication of a con- of other critical indicators, such as its multidimensional pov- stant factor to selected components of the welfare aggregate, erty index, also regulated by CONPES (2012). such as labor, capital and pensions, implicitly assumed the en- tire discrepancy between a particular component and the na- Acknowledgment tional account aggregate is distributional neutral, which is The author would like to thank all members and collabora- very hard to justify (Leyva-Parra 2005; MESEP 2012). tors of the MESEP, in particular Jorge Iván González, Manuel Given this context, the objective of the second phase of Ramírez, Carlos Eduardo Vélez, Mauricio Santa María, Juan the MESEP Committee was to recommend a new official Carlos Feres, Jairo Nunes, Alejandro Gaviria, Christian Jara- methodology to measure poverty, which would: (i) revise millo, Carlos Sepúlveda, Juan Mauricio Ramírez, Roberto both the poverty line and the income aggregate used and (ii) Angulo, José Fernando Arias, Guillermo Rivas, Alejandro propose an institutional arrangement going forward that Mateus, Sandra �lvarez, Francisco Espinosa, Diana Carolina would help shield the poverty and inequality measures of Co- Nova, Rocío Bermúdez, Claudia Gordillo, Sandra Guayazán, lombia. The second phase of the MESEP concluded after 25 Carlos Arturo Mora, Andrés Castañeda, Bernardo Atuesta, working sessions and 35 technical briefs, which were jointly Diana Esperanza Sánchez, Silvia Esperanza Botello, Daniel prepared by the DNP and DANE technical secretariat. Valderrama, Juan Carlos Ramírez, Olga Lucía Acosta, Javier The final numbers were announced on August 24, 2011, Herrera, Xavier Mancero, Darwin Cortés, Luis Carlos Gó- at a large international conference chaired by President San- mez, Eduardo Freire, Irma Parra, Carolina Renteria, Hernan- tos, which was followed by a subsequent series of technical do José Gómez, Esteban Piedrahíta, Jorge Bustamante, and workshops, publications, and legislation to solidify the re- Héctor Maldonado. A number of World Bank colleagues also vised methodology. More specifically, this process also includ- provided valuable inputs throughout this process, with a spe- 3 POVERTY REDUCTION AND ECONOMIC MANAGEMENT (PREM) NETWORK    www.worldbank.org/economicpremise cial thanks to John L. Newman, Jaime Saavedra, Luis-Felipe the number of subnational stratum in the sample increased Lopez-Calva, Carolina Diaz-Bonilla, Renos Vakis, Olivier Du- from 13 to 24. priez, Johan Mistien, Viviane Sanfelice, Jordan Schwartz, References Eduardo Somensatto, Lars Moller, Nobuo Yoshida, Gloria M. Grandolini, Sabine Hader, Harold L. Bedoya, and Louise J. CONPES (National Council for Social and Economic Policy). Cord. This paper benefitted from comments and suggestions 2012. “Metodologías Oficiales y Arreglos Institucionales para from Roberto Angulo, Louise J. Cord, and Samuel Freije. la Medición de la Pobreza en Colombia.� CONPES Social 150, DNP, Bogotá. About the Author DANE (National Department of Administrative Statistics). 2011. “Resolución 1503.� November 16, Bogotá. Joao Pedro Azevedo is a Senior Economist in the Poverty Reduc- DANE-DNP (Department of National Planning). 2009a. “Con- tion and Economic Management Network of the World Bank. venio Interadministrativo.� No. DNP 002-09, Celebrado entre He currently works for the European and Central Asia Region el DNP and DANE, January 27, Bogotá. (ECA) on Tajikistan and Turkey, and leads the ECA Stats ———. 2009b. “Resultados Fase 1: Empalme de las Series de Mercado Team. Prior to joining the Bank, he was the Superintendent of Laboral, Pobreza y Desigualdad.� Presentation, Bogotá. Monitoring and Evaluation at the Secretary of Finance from the Korinek, Anton, Johan Mistiaen, and Martin Ravallion. 2006. State of Rio de Janeiro, and was a Research Fellow at the Brazil- “Survey Nonresponse and the Distribution of Income.� Journal of Economic Inequality 4(1): 33–55. ian Institute of Applied Economic Research (IPEA). He is a for- Leyva-Parra, Gerardo. 2005. “El Ajuste del Ingreso de la ENIGH mer Chairman of the LACEA/IADB/WB/UNDP Research Net- con la Contabilidad Nacional y la Medicion de la Pobreza en work on Inequality and Poverty (NIP). Mexico.� In Números que Meven al Mundo: La Medición de Pobreza en México, Cord. Miguel Székely. Mexico. Notes MESEP (Misión para el Empalme de las Series de Empleo, Pobreza 1. Changes made include: the size of the sample (number of y Desigualdad). 2009. “Misión para el Empalme de las Series de Empleo, Pobreza y Desigualdad (MESEP) entrega Series households); the method for implementing the survey—a Actualizadas al Gobierno Nacional.� Presentation, DNP-DANE, change from paper and pen surveys to the use of digital hand- Bogotá. held devices; and the length and order of the questions in the ———. 2010. “MESEP/Misión para el Empalme de las Series de Em- survey. Moreover, in the previous ECH, one adult head of pleo, Pobreza y Desigualdad: Avance Construcción Canasta.� household would answer the survey questions, whereas in the Presentation, DNP-DANE, Bogotá. new GEIH, different sections of the survey correspond to dif- ———. 2012. “Pobreza Monetaria en Colombia: Nueva Metodología ferent respondents—some some sections are completed by the y cifras 2002–2010; Resultados 2ª Fase de la MESEP.� DNP/ head of the household, other parts are asked directly to house- DANE. Bogota, Colombia hold members that are 10 years of age and older (DANE-DNP Muñoz, C. Manuel, and M. Guillermo Rivas. 2006. Construcción de las Canastas Normativas de Alimentos para Trece Ciudades, Resto 2009b). Urbano, y Zona Rural. Misión para el Diseño de Una Estrategia 2. Improvements in data quality include: (i) fewer missing en- para la Reducción de la Pobreza y la Desigualdad, DNP, Bogotá. tries (nonrespondents) on the income variable; (ii) less under- Ravallion, Martin. 2003. “Measuring Aggregate Welfare in Develop- reporting on the income variable; (iii) the labor market, pov- ing Countries: How Well Do National Accounts and Surveys erty, and inequality estimators became more precise; and (iv) Agree?� The Review of Economics and Statistics 85(3): 645–52. The Economic Premise note series is intended to summarize good practices and key policy findings on topics related to economic policy. They are produced by the Poverty Reduction and Economic Management (PREM) Network Vice-Presidency of the World Bank. The views expressed here are those of the authors and do not necessarily reflect those of the World Bank. The notes are available at: www.worldbank.org/economicpremise. 4 POVERTY REDUCTION AND ECONOMIC MANAGEMENT (PREM) NETWORK    www.worldbank.org/economicpremise