February 2011 Number 166 www.worldbank.org/enbreve A regular series of notes highlighting recent lessons emerging from the operational and analytical program of the World Bank`s Latin America and Caribbean Region (LAC). Cutting costs, boosting quality and collecting data real-time ­ Lessons from a Cell Phone-Based Beneficiary Survey to Strengthen Guatemala's Conditional Cash Transfer Program by Christian Schuster and Carlos Perez Brito 60356 A 2010 Country Governance and Anti-Corruption (CGAC)- Due to this rapid growth, mobile-based applications have funded pilot in Guatemala employed entry-level mobile penetrated areas as diverse as public administration, phones in conjunction with EpiSurveyor, a free, web-based health, education, banking, agriculture, and business software for data collection, to drastically reduce cost, development--even in the remotest areas of the globe, facilitate accuracy and accelerate implementation of a reaching poor and isolated communities. For example, nationally-representative beneficiary survey of Guatemala`s mobile-based applications can now be undertaken conditional cash transfer program. As such, it illustrates in a multitude of specific development interventions: the potential of mobile phone-based data collection financial services to store and send money, alerts to to strengthen program monitoring, evaluation and refugees about food aid deliveries, distance education implementation, in particular in remote and marginalized courses, citizen polling station surveillance, remote areas highly populated by indigenous peoples. health care consultation and diagnosis, and timely market information to fishermen to name just a few in a fast-growing list. 1. Seizing the Development Potential of While research to evaluate these mobile interventions The Mobile Phone Revolution has been growing, there are relatively few studies of the use of mobile technology itself as a research instrument The growth of mobile phones in the last decade has been in developing countries. Mobile data collection projects exceptional. By the end of 2010, global mobile phone to capture outcomes are abundant, but there are far subscriptions are expected to surpass the 5 billion mark. fewer large-scale and complex surveys using mobile In particular, mobile phone users in developing countries, phones. The potential mobile phones hold in this regard including the poor, have benefited from this trend; frequently is striking; compared to a traditional process using paper- skipping fixed-line infrastructure and leapfrogging directly and-pencil forms with later transcription to a computer into mobile technology they now account for two-thirds of system, mobile devices offer immediate digitization and all mobile phone subscriptions. This trend is reflected in Latin transmission of collected data at the point of survey, America, where mobile phones have evolved from a luxury followed by automated data aggregation. As such, item to an essential good. Almost 90 percent of the region's mobile phones promise faster, more cost-effective and population now owns a mobile phone. In Guatemala, the more accurate surveys. number of active mobile phone subscriptions--17.3 million-- even exceeds the country's population of 13.3 million and In view of these advantages and to expand the evidence network coverage extends to most of the country's territory, base on the use of mobile phones as a research including numerous historically marginalized rural areas instrument, a national survey of beneficiaries of Mi highly populated by indigenous peoples. Familia Progresa (Mifapro), Guatemala's conditional cash 1 transfer (CCT) program, was undertaken with low-cost In 2010, this survey exercise was repeated with the same mobile phones and free software for data collection institution as part of the CGAC-funded activities and as part of the 2010 Country Governance and Anti- strategy. This time, however, the methodology for data Corruption (CGAC)-funded activities. collection shifted from paper-based to mobile-phone- based. As such, a comparative analysis of the cost, accuracy 2. Project Context and Objectives and speed of the two methodologies was similarly possible. In March 2008, the Guatemalan government developed 3. Methodological Approach the CCT program in the face of historically high levels of exclusion--with 51 percent of the population living While paper-based data collection methods have been below the poverty line and nearly half of all children the standard method for decades, they tend to suffer suffering from chronic malnutrition. By the end of 2010, from frequent errors, storage burdens and high costs more than 900,000 women were benefiting from the of double data entry. Handheld devices--primarily program. While Mifapro has the potential to significantly personal digital assistants (PDAs)--are increasingly reduce extreme poverty, its main challenges lie in the replacing paper-based methods, but face shortcomings effective management of its monitoring and evaluation themselves: the data need to be downloaded to laptops system, including monitoring of conditionalities in in frequent intervals, are not available in real-time, and health and education. A critical step to overcome this may be corrupted or even lost if PDAs are damaged, challenge is gaining access to quick and accurate first- misplaced, or stolen. hand information on activities at rural clinics and schools in isolated parts of the country. This puts a premium on a Mobile-phone-based data collection systems have cost-effective mechanism to collect field data. the potential to overcome these limitations. They can be used with several types of data collection clients, As part of the Bank's CGAC sector strategy in the the most common of which include fixed format SMS, country, a paper-based survey of 200 beneficiaries in voice-based data collection, web-based forms and Java- five of the poorest municipalities was financed with based applications. Short Message Service (SMS) and 2009 CGAC funds to provide demand-side information voice-based surveys are particularly useful for short, to strengthen Mifapro's implementation. The Instituto self-administered data collection. Voice-based surveys de Investigaciones Económicas y Sociales (IDIES), are especially preferable in regions with low levels of of the Universidad Rafael Landívar1 carried out the literacy. For more complex surveys, however, web-based activity. Quantitative information about key aspects of forms and Java-based applications offer advantages in the program--including the targeting effectiveness terms of cost and user-friendliness. First, they facilitate of the program, compliance with and awareness of adherence to context-dependent questionnaires, as they conditionalities, costs of compliance for beneficiary can determine which questions should be answered or households and behavioral changes induced by the skipped. Second, they allow for uploading of completed program--supplied the government's monitoring and surveys to host servers using low-cost general packet evaluation system with essential input. radio service (GPRS) without limiting the size of the survey form--as, for instance, the case with SMS. While web-based forms require mobile phones with web browser capabilities, Java-based applications can be downloaded onto less expensive, entry-level mobile phones. The 2010 CCT survey therefore chose to rely on the Java-based application, EpiSurveyor, a mobile data collection tool funded by the United Nations Foundation, Vodafone Foundation, and a World Bank Development Marketplace Grant. EpiSurveyor works on a "freemium" model, offering a free basic version (which more than 99 percent of subscribers use) and a paid version with premium features. Within less than a year after launch, it Source: Universidad Rafael Landivar (2010) had over 2,000 users. 1 http://www.url.edu.gt/Portalurl/ 2 EpiSurveyor is the first cloud-computing application had been interviewed in five municipalities in the 2009 for data collection in international development. survey. The sample included beneficiaries living in some Users with basic computer skills can log in online at of the poorest and most isolated municipalities in the www.episurveyor.org to create survey forms and country, demonstrating the potential to monitor and questions, including multiple-choice, free text, numeric, improve program implementation in rural areas. data, and other types. The survey forms can then be downloaded to any Java- and GPRS-enabled mobile phones. The phones can thus be used to collect and 4. Outcomes upload data in real time to an EpiSurveyor account Compared to its 2009 paper-based counterpart, the 2010 through a secure server or to a laptop, as needed. The mobile-phone-based survey proved highly superior in EpiSurveyor website allows for real-time visualization of terms of cost and showed notable improvements in quality the survey responses and analysis of the aggregated control and the implementation speed of the survey. and disaggregated data through graphs, charts, and maps as well as data exporting to common third-party Efficiency Gains: data analysis programs such as Microsoft Excel or Access Cell-Phone-Based vs. Paper-Based Data Collection (Figure 1). In addition, the mobile phones themselves can Average Interview Cost -71% be used to view and analyze aggregated data. All survey data are thereby encrypted to maintain confidentiality of Average Interview Length -3.6% responses. Access to the data is restricted by password, Data Collection Quality Improved with the ability to provide different access privileges for % of Interviewers Preferring Cell-phones different types of users. 88.9% for Data Collection Figure 1: Visualization of EpiSurveyor Most notably, the cost savings resulting from immediate Survey Question: ¿Qué compromisos u obligaciones digitization and transmission of collected data with adquirió usted con MIFAPRO para tener derecho al subsequent automated data aggregation offset by far beneficio en efectivo? the purchasing and data transmission costs of mobile phones. Through the mobile phones, the average cost Option Checked Unchecked Total per interview data collection and digitization was cut by (%) (%) (%) 71 percent, which permitted an increase in the survey's A Llevar regularmente a los sample size from 200 to 700 and the achievement of 376 (75) 124 (25) 500 (100) national representativeness while maintaining a flat niños a la escuela B Llevar a los niños a control budget for the survey. 360 (72) 140 (28) 500 (100) en el centro de salud In addition, real-time data collection facilitated quality control by the supervisors of the field work. The system 600 124 140 380 437 487 457 458 495 registered date and time of each interview, not the time 500 of submission; when no mobile 400 376 360 phone coverage was available, 300 surveys were stored in the phone 200 120 100 63 43 42 until a signal was found. The web 13 0 5 interface allowed the supervisor A B C D E F G H to monitor the work rate, when work began and ended, and to Unchecked Checked detect any data inconsistencies. Supervisors could then respond Source: EpiSurveyor, WB CCT Survey, 2010 immediately by calling the data collectors. In addition, easier data entry improved accuracy; The Bank's 2010 CCT survey was the first nationally most interviewers noted that representative study using EpiSurveyor in the country. the mobile phones reduced the A team of nine data collectors interviewed 500 Mifapro likelihood of errors in entering beneficiaries (mainly indigenous women) in 25 data. In addition, automated municipalities, in addition to the 200 beneficiaries which uploading of the data omitted 3 the need for digitization, which tended to enhance errors improvements over traditional methods, the study by between 1 and 2 percent in the previous survey. Finally, has already had an important demonstration effect the survey never experienced any data loss, as is frequently on the adoption of the technology in the country and the case with paper- or PDA-based approaches. within the World Bank. To illustrate, the government has announced that it will adopt the approach not only Beyond lower cost and facilitated accuracy, mobile for the monitoring and evaluation system of the CCT phones also helped speed up the time required for program, but also in a nationwide survey on nutrition interviews. According to measurements undertaken in for the Ministry of Health. To further illustrate, a proposal the initial training workshop, mobile phones brought (Use of Cell-Phone Data to Collect and Process Data to down the time required for surveys by 3.6 percent. Strengthen Monitoring, Control and Accountability of After the survey, nearly 90 percent of the data collectors CCT Programs in Latin America) will be funded by the concluded that interviews can be undertaken faster with highly competitive Bank Innovation Fund in 2011. mobile phones than with their paper-based counterparts. While these findings add important evidence to the These positive outcomes are reflected in the favorable viability of mobile phones as a data collection tool in perceptions of interviewers about the use of mobile developing countries, the number of studies providing phones for data collection. All of the interviewers similar evidence is still limited.2 Not less important, reported that it was either `easy' or `very easy' to learn mobile-phone-based solutions should not be considered how to use mobile phones for data collection­not the as a panacea for all data collection problems. To illustrate, least thanks to their familiarity with mobile phones in while they facilitate quality control, they cannot detect their private lives. Nearly 90 percent preferred to collect all types of data fabrication; random and timed answers, data using mobile phones in future surveys. for example, would still go undetected. The survey, however, also pointed to a potential pitfall As this study and the growing adoption of EpiSurveyor in the use of mobile phones. Slightly over 20 percent suggest, mobile phones are increasingly powerful tools of interviewers observed negative reactions among for data collection. In the future, ongoing technological their interviewees to the use of mobile phones for advances and increased access will further improve the data collection. While the increasingly ubiquitous access potential of mobile phones for accurate data collection. to mobile phones is likely to lower resistance, their Mobile phones with Global Positioning System (GPS) use as a survey instrument should thus be based on capabilities, for example, could detect the aforementioned an assessment of their acceptability among the target data fabrication. Recognizing and harnessing this potential population, particularly poor indigenous populations. can provide high returns for future survey exercises and, as such, strengthen program governance in a quick, 5. Caveats and Conclusions accurate and cost-effective manner -- in particular in rural and marginalized areas. For the 2010 CCT study, the use of low-cost mobile phones in conjunction with the free EpiSurveyor software 2 See, for instance: Tomlinson, Mark et al. (2009) The use of mobile phones as a data collection tool: A report from a household survey in South Africa. BMC drastically cut costs while facilitating quality control Medical Informatics and Decision Making 2009, 9:51. Available for download and improving implementation speed. Thanks to these at http://www.biomedcentral.com/content/pdf/1472-6947-9-51.pdf About the Authors Christian Schuster is an Economist in the World Bank's Guatemalan Country Office and Carlos Perez Brito is a Human Development Specialist in the World Bank's Latin America Region. Disclaimer: The findings, interpretations, and conclusions expressed herein are those of the author(s) and do not necessarily reflect the views of the Executive Directors of the International Bank for Reconstruction and Development / The World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. About "en breve"... "en breve" is produced by the Knowledge and Learning Team of the Operations Services Department of the Latin America and the Caribbean Region of The World Bank ­ http://www.worldbank.org/lac. Visit the entire "en breve" 4 collection at: www.worldbank.org/enbreve.