POVERTY THE WORLD BANK REDUCTION AND ECONOMIC MANAGEMENT NETWORK (PREM) Economic Premise JANUARY J E 2 1 • Number 2013 Numb 102 18 Collecting High-Frequency Data Using Mobile Phones: 74668 Do Timely Data Lead to Accountability? Kevin Croke, Andrew Dabalen, Gabriel Demombynes, Marcelo Giugale, and Johannes Hoogeveen As mobile phone ownership rates have risen dramatically in Africa, there has been increased interest in using mobile telephones as a data collection platform. This note draws on two largely successful pilot projects in Tanzania and South Sudan that used mobile phones for high-frequency data collection. Data were collected on a wide range of topics and in a manner that was cost-effective, flexible, and rapid. Once households were included in the survey, they tended to stick with it: respondent fatigue has not been a major issue. While attrition and nonresponse have been challenges in the Tanzania survey, these were due to design flaws in that particular survey, challenges that can be avoided in future similar projects. Ensuring use of the data to demand better service delivery and policy decisions turned out to be as challenging as collecting the high-quality data. Experiences in Tanzania suggest that good data can be translated into public accountability, but also demonstrate that just putting data out in the public domain is not enough. This note discusses lessons learned and offers suggestions for future applications of mobile phone surveys in developing countries, such as those planned for the World Bank’s “Listening to Africa� initiative. Timely, high-quality information about socioeconomic out- mobile phones to respondents at the time of baseline, who comes related to well-being, service delivery, income, security, then are called regularly (weekly or every two weeks) with health, and many other topics is not readily available in Afri- follow-up questions. These questions can be comparable to ca. This is because such data are typically collected by nation- questions asked earlier, to allow for tracking of changes over ally representative, face-to-face household surveys. Such sur- time, or can address new issues that have arisen since the im- veys are expensive and time consuming and are, for these plementation of the baseline survey. This approach enables reasons, not implemented very frequently. There is strong high-frequency, low-cost collection of a wide range of data re- (latent) demand for more timely welfare and monitoring in- lated to household welfare. formation—decision makers need information that allows Two African countries, South Sudan and Tanzania, have them to monitor the situation in their country in “real time.� already piloted mobile phone surveys. The survey in Tanzania Program managers, researchers, and citizens could all benefit has been running longest (33 rounds to date), while the sur- from such information. vey in South Sudan operates under more difficult conditions. Because mobile phone networks now reach most areas of These are not the first examples of using a mobile phone plat- Africa, this note proposes a mobile phone survey model that form to collect high-quality panel data in Africa. Dillon combines a standard baseline survey with the distribution of (2012), for instance, used mobile phones to track how farm- 1 POVERTY REDUCTION AND ECONOMIC MANAGEMENT (PREM) NETWORK    www.worldbank.org/economicpremise ers’ expectations of the next harvest changed over time. This is also when phones are distributed, and in locations with note differs from Dillon’s work in two ways: it proposes the limited access to electricity, solar chargers. Alternatively, a creation of nationally representative (panel) surveys that ask village kiosk owner who provides phone-charging services questions via mobile phones every two weeks, and then pro- could be contracted to offer free phone charging to partici- poses the data be publicly released (in anonymity) within pating households. weeks of collection. This note also discusses how to translate Once the baseline has been completed, respondents re- the data into policy in the public domain. ceive calls from a call center at regular intervals during which additional questions are asked. In this way, a high-frequency Mobile Phone Surveys panel is created. Answers from respondents can be collected Conducting surveys by phone is standard practice in devel- in various ways because mobile phones offer a multitude of oped countries, but typically has not been considered feasible opportunities to obtain feedback, including through SMS, in poor countries, because phone ownership rates are too low WAP, IVR, and USSD. When this note mentions mobile sur- (especially in the premobile phone era). In Tanzania, for ex- veys, it refers to surveys carried out by call centers where op- ample, just 1 percent of households own a landline phone erators call respondents, interview them, and enter the re- (DHS 2010). However, the rapid rise of mobile phones in Af- sponses into a database using a CATI (computer-aided tele- rica has changed this. Again in Tanzania, mobile phone own- phone interview) system. ership increased from 9 percent of all households according The decision to rely on call centers for mobile phone data to the 2004–5 Demographic and Health Survey (DHS), to 28 collection was informed by experiences with WAP, IVR, and percent by 2007–8. By 2010, this number had almost dou- USSD in the early stages of the Tanzania mobile survey. The bled again, to 46 percent of households. Even higher rates of flexibility that call centers offer, particularly when asking phone ownership have been recorded in urban areas: phone complex questions requiring explanation, for accommodat- ownership was 28 percent for urban households in the ing illiterate respondents or low-end phones without Internet 2004–5 DHS, and more than doubled to 61 percent in connectivity makes voice the most feasible technology for 2007–8, and reached 78 percent by 2010. In the Tanzania sub-Saharan Africa. Call centers are able to deal with respon- baseline survey in Dar es Salaam, mobile ownership was dents who speak different native languages, good enumera- found to be 83 percent. In Kenya, the sub-Saharan country tors can build a rapport between respondent and phone op- that is leading in terms of mobile phone ownership, the Af- erator, and call centers offer the opportunity to ask in-depth robarometer survey of November 2011 shows that 80 per- (qualitative) questions. cent of Kenyan adults have their own mobile phone; 81 per- Results from the Tanzania and cent use their mobile phone to make a call at least once a day; South Sudan Mobile Surveys 61 percent send or receive a text message at least once a day; and a remarkable 23 percent send or receive money or pay a The mobile survey in South Sudan revisited 1,000 respon- bill via mobile phone at least once a day. dents in 2010 in 10 urban areas covered in South Sudan’s With such high rates of mobile phone ownership, repre- 2009 National Baseline Household Survey. During this revis- sentative household surveys using mobile phones start to be- it, enumerators selected respondents, handed out mobile come a feasible option. For example, only 80 percent of U.S. phones (half of them with integrated solar chargers), and then households own landlines, but political polling still typically called respondents on a monthly basis from a call center in uses landline samples only, corrected by reweighting (Blum- Nairobi, using interviewers capable of speaking South Su- berg and Luke 2009). This suggests that phone ownership in dan’s main languages. Respondents who successfully com- Kenya or in urban Tanzania is already high enough for reason- pleted an interview were rewarded with an amount varying able inference to be made from surveys that exclusively rely from $2 to $4. on mobile phones. The survey in Tanzania visited 550 households in Dar es Salaam in August 2010, administered a baseline survey, ran- Listening to Africa domly selected an adult respondent for the mobile survey, Mobile surveys, such as the proposed Listening to Africa and called respondents on a weekly basis (25 rounds), and project, will not rely exclusively on interviews over the later (8 rounds) every two weeks. The survey in Dar es Salaam phone, but will employ a face-to-face baseline survey where did not initially distribute phones, but recently, after round household and respondent characteristics will be collected, 33, some phones have been distributed to respondents who and during which the respondent for the mobile phone part had never participated in the survey before. Both surveys are of the survey will be selected. Listening to Africa aims to col- still running. lect population data, requiring that an (adult) respondent be The mobile survey interview format does not appear to randomly selected from the household. The baseline survey pose major limitations on what can be asked, except that 2 POVERTY REDUCTION AND ECONOMIC MANAGEMENT (PREM) NETWORK    www.worldbank.org/economicpremise the length of an interview should probably not be more The data can be used to report on a single issue, but become than 20–30 minutes. So an elaborate consumption mod- of greater interest when the same information is tracked ule, for example, or a detailed health module with birth over time (figure 1). histories, is less suited to this type of survey. The mobile Because questions can be changed every round of the sur- surveys in South Sudan and Tanzania collected informa- vey, it is possible to accommodate new data requests or to re- tion on a wide variety of issues including health, education, spond to emerging issues. For instance, relevant questions water, security, nutrition, travel times, prices, electricity, were included in the Dar es Salaam survey after major floods and governance. The surveys have been used to ask percep- hit the city in December 2011 to get a sense of the number of tion questions on topics varying from what respondents people that had been affected. Mobile surveys can also ask the considered the most pressing problems to be addressed by main respondent to pass the phone on to someone else in the the city government to people’s opinion about the draft household to get certain questions answered, if necessary. Fig- constitution. They have also been used to collect baseline ure 2, for instance, presents responses to questions asked to information for large-scale programs on food fortification. children attending primary school about the presence of their teacher and the use of books. Finally, though this has not been Figure 1. In the Last Month, How Often, If Ever, Have You or a tried yet, the mobile survey can be used to field screening Member of Your Household Gone Without Enough Food to Eat? questions to identify respondents who qualify for in-depth never once, twice interviews. many times or always or several times Nonresponse and Attrition 100 A key challenge for high-frequency mobile surveys is nonre- 80 sponse (when a respondent participates in some but not all rounds) and attrition (when a respondent drops out of the 60 survey completely). What does the Tanzania experience tell percent us about attrition and nonresponse in mobile phone sur- 40 veys? On the negative side, there was a large initial burst of attrition. This can largely be attributed to the fact that the 20 survey team did not hand out phones to those who did not already own one. When the team initially visited and ad- 0 ministered the baseline survey to 550 respondents, it was December January February March found that 418 owned their own phone. Once the mobile Source: Croke et al. 2012. survey began in earnest, an average of 304 respondents, or 66 percent, participated during the first 19 rounds of the Figure 2. Questions Asked to Primary School Children about Teacher Presence and survey. Later, once the survey was put Use of Books while in School under World Bank management and oversight was tightened (after a four- 90 83 month gap in interviews!), the number 80 of respondents increased to 343 re- 70 spondents on average (75 percent of the 60 54 sample). So after 33 rounds of mobile interviews, the overall response rate is percent 50 46 40 75 percent from the 458 households in the sample that had access to phones 30 (62 percent of the 550 households in 20 the baseline survey). The rate of attri- 8 9 10 tion, narrowly defined as those who did 0 not respond at all to the mobile survey taught taught for didn't teach did use did not use is much lower: only 4 percent of 18 out through all part of at all books books of the 458 households never responded class period class period yesterday to a mobile survey, while 66 percent re- sponded to at least two out of every teacher presence use of books three surveys. Source: Croke et al. 2012. 3 POVERTY REDUCTION AND ECONOMIC MANAGEMENT (PREM) NETWORK    www.worldbank.org/economicpremise Is the Tanzania Mobile Survey at the breakdown across the 458 respondents that were in- Representative? cluded in the mobile phone survey, one notes that poor households are underrepresented. The distribution be- Attrition and nonresponse are particularly problematic when comes more skewed toward wealthier households in round they occur in a nonrandom manner. The working paper from 26 (341 respondents). The final set shows what the final dis- which this note is drawn includes regression analysis of the tribution looks like once the mobile phone sample has been determinants of attrition. In these regressions, the dependent reweighted using information about the determinants of at- variable is the number of rounds (out of 25) in which the trition. It shows that the original distribution is essentially household participated. When attrition among all 550 ini- restored. tially visited households is analyzed, results show—perhaps unsurprisingly for a survey that did not distribute mobile Using Mobile Data to Enhance phones—that wealth is correlated with survey participation. Accountability When analysis restricts the regression to households that were identified as reachable in the first full round of the sur- Since management of the Tanzania survey was transferred to vey (that is, households that own phones and that gave a work- the World Bank, active efforts have been made to use the sur- ing phone number to the survey team), the impact of wealth vey for accountability purposes. Questions are carefully iden- is no longer significant, while residence in the rural part of tified for their potential use as accountability tools. Once data Dar es Salaam and using the premium mobile phone provider are collected, easy-to-understand, factual reports are prepared (Vodacom) remain significant variables. presenting the findings. These reports are disseminated The lesson learned from this regression is that with through a dedicated Web site from which all survey data, phone distribution to poor households and more careful col- baseline survey as well as data from the mobile survey rounds, lection of multiple, verified phone numbers from each re- can be downloaded (www.listeningtodar.org). The reports are spondent, nonresponse could have been significantly re- also shared by email using a distribution list that includes duced. The second lesson is that this survey, in order to journalists and other stakeholders. remain representative, would have to be reweighted ex post. So, does it work? While the Web site itself attracts rela- Figure 3 illustrates how reweighting is able to restore the sur- tively modest traffic, the project has gotten quite a bit of me- vey’s representativeness by showing how the changing com- dia traction, especially since May 2011, when it partnered position of the sample affects the percentage of households with a local television journalist, who now holds a press con- allocated to different wealth quintiles. The first column ference to introduce each new survey report to the local me- presents the survey baseline (550 respondents); the sample dia. Reports produced have been discussed on blogs and in is then divided equally among the five wealth quintiles: each academia, on television in Tanzania, and have led to various quintile has exactly 20 percent of the sample. When looking newspaper articles. It is harder to assess what happens once information is published, but there are indications that the information is “received� by those responsible for results. For Figure 3. Changing Wealth Composition of Sample instance, the managing director of Tanzania’s electricity com- 25 pany felt compelled to explain to the media why so many households connected to electricity are experiencing power 20 cuts and what his company is doing about them.1 Once pub- lished, information travels fast and far. A brief about food price increases was used in a front page article in Tanzania’s 15 Citizen newspaper, which was picked up by others including percent the Rwandan Times,2 and is cited in the World Bank’s 2012 Global Monitoring Report. Information also tends to go in un- 10 expected directions. A brief about the limited increase in wa- ter connections in Dar es Salaam, despite a large-scale invest- 5 ment program, got media attention because of the discrepancy between the data reported by the mobile survey and official government statistics.3 0 What lessons can be drawn thus far? One lesson is that full sample mobile survey round 26 round 26 reweighted providing citizens with relevant, timely, and accurate data poorest middle wealthiest about the actions of politicians, policy makers, and public ser- second fourth vice providers is not sufficient. For the data to have impact, Source: Croke et al. 2012. they need to be accessible and disseminated widely, and in 4 POVERTY REDUCTION AND ECONOMIC MANAGEMENT (PREM) NETWORK    www.worldbank.org/economicpremise ways that allow them to be utilized by already existing institu- control of mobile surveys is dynamic because issues identified tions and actors. in one round can be corrected in the next. Finally, the use of the results from the surveys has been Cost-Effectiveness of Mobile Surveys encouraging. Once results were disseminated systematically, How cost-effective are mobile surveys? World Bank data give civil society started to discuss them and media reported on a good sense of the marginal cost of the surveys. The call cen- them. At the same time, it is clear that if the objective is to use ter in Tanzania was originally contracted to implement 12 the results for accountability purposes, distinct efforts need survey rounds at a rate of $1,400 per round. If one adds the to be made to ensure that the information reaches the actors cost for consultants to maintain a Web site, supervise data col- in government and civil society who are most willing and able lection and to analyze the data, the marginal cost per round to make use of it. increases to $2,500. Given that these rounds averaged 343 About the Authors respondents, this comes to about $4.10–$7.30 per interview. Dillon (2012) notes a relatively similar marginal cost per sur- Kevin Croke is a Consultant with the World Bank’s Africa Re- vey: $6.98. In addition to these marginal costs, one needs to gion. Andrew Dabalen is a Lead Poverty Specialist, Gabriel De- include the cost of a baseline, which will often be between mombynes is a Senior Economist, Marcelo Giugale is Director for $50 and $150 per interview, depending on the complexity of Poverty Reduction and Economic Management, and Johannes the survey and the distances that have to be covered. Whether Hoogeveen is a Senior Economist, all with the World Bank’s Af- this is cost-effective or not depends on the purpose. The abil- rica Region. ity to carry out an entire survey round and report on results Notes for $2,500 is remarkably cost-effective. But if one keeps in mind that the typical round in the Tanzania survey asks 17 1. http://www.ippmedia.com/frontend/index.php?l=39618 questions (with a maximum of 44), then the cost per ques- and http://www.ippmedia.com/frontend/index. tion is about $0.42. This is relatively high, so if the intention php?l=39551. is to ask many questions, it may turn out to be more cost-effec- 2. http://www.newtimes.co.rw/news/index. tive to opt for a face-to-face interview. php?i=14892&a=11334. 3. http://www.thecitizen.co.tz/sunday-citizen/-/20105-govt- Conclusion figures-on-access-to-clean-water-inflated. Evidence demonstrates that mobile surveys have great poten- References tial to provide rapid feedback and address existing data gaps at limited expense. Mobile surveys should not be considered Baird, Sarah, and Berk Özler. 2011. “Examining the Reliability of substitutes for household surveys, but complements: mobile Self-Reported Data on School Participation.� Journal of Develop- ment Economics 98 (1): 89–93. surveys may rely on an existing household survey to serve as Blumberg, Stephen J., and Julian V. Luke. 2009. “Wireless Substi- baseline, and mobile surveys are not the right platform for tution: Early Release of Estimates from the National Health lengthy interviews. When interviews are lengthy, face-to-face Interview Survey, July-December 2008.� Centers for Disease interviews are probably more cost-effective. Control, Atlanta, GA. The evidence from the Tanzania and South Sudan surveys Croke, Kevin, Andrew Dabalen, Gabriel Demombynes, Marcelo suggests that mobile surveys can collect quality data in a very Giugale, and Johannes Hoogeveen. 2012. “Collecting High-Fre- timely manner that is of use to a wide range of data users: deci- quency Panel Data in Africa Using Mobile Phone Interviews.� World Bank Policy Research Working Paper 6097, Washington, sion makers, program managers, statisticians, and researchers. DC. The Tanzania survey highlighted the need for mechanisms Dillon, Brian. 2012. “Using Mobile Phones to Conduct Research in that avoid attrition and nonresponse right from implementa- Developing Countries.� Journal of International Development 24 tion of the baseline. Much of the attrition in the Tanzania sur- (4): 518–27. vey can be explained by choices made in the organization of Lynn, Peter, and Olena Kaminsha. 2011. “The Impact of Mobile the survey (such as not to distribute mobile phones). Findings Phones on Survey Measurement Error.� Working Paper No. 2011-7, Institute for Social and Economic Research, University suggest that once households are included in the mobile sur- of Essex. vey, they are likely to remain in the survey: respondent fatigue Smith, G., I. MacAuslan, S. Butters, and M. Tromme. 2011. “New was not found to be an issue. These pilot experiences also show Technology Enhancing Humanitarian Cash and Voucher Pro- that because of the high frequency of data collection, quality gramming.� Research paper, Oxford Policy Management. The Economic Premise note series is intended to summarize good practices and key policy findings on topics related to economic policy. They are produced by the Poverty Reduction and Economic Management (PREM) Network Vice-Presidency of the World Bank. The views expressed here are those of the authors and do not necessarily reflect those of the World Bank. The notes are available at: www.worldbank.org/economicpremise. 5 POVERTY REDUCTION AND ECONOMIC MANAGEMENT (PREM) NETWORK    www.worldbank.org/economicpremise