High Frequency Mobile Phone Surveys of Households to Assess the Impacts of COVID-19 Guidelines on CATI Implementation The guidelines have been prepared by Akuffo Amankwah (Economist, Development Data Group), John Ilukor (Economist, Development Data Group), Shelton Kanyanda (Consultant, Development Data Group), Sergiy Radyakin (Senior Economist, Development Data Group), Zurab Sajaia (Senior Economist, Development Data Group), Arthur Shaw (Survey Specialist, Development Data Group), Michael Wild (Senior Statistician, Development Data Group, Kazusa Yoshimura (Jr Professional Officer, Poverty and Equity Global Practice). Version: May 1, 2020. 1 Contents Motivation..................................................................................................................................................... 3 Overview ....................................................................................................................................................... 4 What needs to be done ................................................................................................................................ 5 Preparation ............................................................................................................................................... 5 Choose survey staff .............................................................................................................................. 5 Procure equipment .............................................................................................................................. 5 Create payment systems ...................................................................................................................... 6 Create CATI questionnaire ................................................................................................................... 6 Perform pilot ........................................................................................................................................ 8 Provide training .................................................................................................................................... 9 Implementation ...................................................................................................................................... 11 Create assignments ............................................................................................................................ 11 Manage survey ................................................................................................................................... 11 What can be done, by time horizon............................................................................................................ 15 Short term............................................................................................................................................... 15 Training............................................................................................................................................... 15 Survey workflow ................................................................................................................................. 16 Survey management .......................................................................................................................... 16 Resources available ............................................................................................................................ 16 Medium term.......................................................................................................................................... 17 Training............................................................................................................................................... 17 Survey workflow ................................................................................................................................. 18 Survey management .......................................................................................................................... 19 Resources available ............................................................................................................................ 19 Longer term ............................................................................................................................................ 20 Support short-term solutions ............................................................................................................. 20 Build longer-term solutions................................................................................................................ 20 Annex 1: Survey staffing ............................................................................................................................. 22 Annex 2: Procurement ................................................................................................................................ 23 Annex 3: Establishing a payment system .................................................................................................... 25 Creating a facility for sending phone/internet credits ....................................................................... 25 Creating a facility for paying interviewers ......................................................................................... 25 Annex 4: Training needs.............................................................................................................................. 26 2 Motivation The rapid spread of the COVID-19 globally has had a profound impact on the lives and activities of people around the world. Most countries have introduced restrictions on movement and contacts of the population, and even where not introduced, isolation is encouraged. This presents an extraordinary problem for data collection for face-to-face data collection, a foundation for the demographic, social, economic, and other types of surveys, which are the typical responsibility of the NSOs. The effect on the operation of the statistical offices has been on several fronts: • NSO office temporary closures; • Restrictions on the movement of interviewers in places where quarantine, curfew, or other isolation regulations were enacted; • Reduction of cooperation of the respondents not willing to expose themselves to contacts with strangers. Without any advanced warning the NSOs (as well as numerous NGOs, and other agencies involved in data collection, including The World Bank) have found themselves in the environment where they can no longer meet in the office to plan their activities; hire, train and directly supervise interviewers; have face-to-face interviews with respondents. Yet the demands for data have only risen: to the scheduled surveys, the surveys on the direct impact of COVID-19 (health surveys) and indirect effects (employment, businesses closures, etc) have been added with great urgency on the NSOs to deliver. As a coping strategy, remote work is adopted to the extent possible, with greater success where the remote presence at meetings, remote work/telecommuting were practiced before the epidemics. Still it is a unique problem for the interviewers, that will need to be retrained and diverted to conduct surveys remotely by telephone where the operations cannot be delayed. Notably, in contrast to telephone surveys made from the call centers equipped with specialized hardware, communications facilities and infrastructure, the new environment presents the challenges for: • the interviewers to work from their homes with minimal equipment setup; realistically, an interviewer will find himself working with only a single portable device (tablet, smartphone, or similar) issued by the office to conduct CAPI surveys, which needs to be repurposed in the new conditions. • the supervisory staff: to plan, coordinate, and monitor the quality of the data collection in such a distributed system without direct interaction. • for technical personnel: to ensure the smooth running of the whole infrastructure, as well as addressing security concerns. To meet these challenges at global scale and with rapid roll-out, the World Bank will need a Computer- Assisted Telephone Interview (CATI) system that fulfills the following requirements: • Familiarity. Used recently or regularly by both the implementing agency and by the World Bank task team. • Low training needs. Inherently simple to use. Immediately familiar to the data collection team. Supported through online training materials. 3 • Low IT requirements. Easily deployable within the implementing agency’s existing IT infrastructure. Usable on data collection devices available to survey teams. Supports cloud hosting. • Rapid setup and deployment. Deployable as an out-of-the-box, click-button solution. Little or no need for installation, setup, or customization. • Extensibility. Possibility to work alongside other systems and/or as part of a larger ecosystem of software products (e.g., R or Python scripts that can facilitate or automate processes related to survey management and data quality monitoring) Given all this we think that the World Bank’s Survey Solutions Computer Assisted Survey system is capable in dealing with these requirements all at once. Its global outreach and strong user base in many countries allows for rapid deployment with minimal training. Nevertheless, in cases where existing CATI data collection systems are already in place, it is recommended to use the existing system and not complicate matters through the introduction of a new technology. Overview This guidance note describes a CATI system to meet the needs described above. The first section describes what needs to be done—that is, how to prepare and implement a CATI survey system. The second section describes how the CATI system will be deployed, as a function of when the survey is deployed—that is, what can be done for surveys deployed in the short term (next 1-2 weeks), what can be done for surveys deployed in the medium term (3-6 weeks), and what can be done in the longer term. In this note, we will also discuss areas of other working groups like the one on sampling or on questionnaire design, as long as it has relevance for the smooth implementation of the data collection process. However, for more detailed discussion, we refer the reader to the corresponding subject document. One word of caution also in this respect. Even though the data collection process by phone may seem very similar to face-to-face surveys on a first glance, it is significantly different in many details. One particular worrisome detail, which will also have an impact on the resulting data is the mode effect, that is the effect of the change from CAPI to CATI on the resulting statistics, see for example here. These mode effect have different impacts on different variables, and normally require careful preparation including sufficient testing to have a good understanding of the resulting effect. Due to the time constraints though, this will not be feasible for the time being. Nevertheless, we also incorporate recommendations for monitoring of this effect (including non-response) and the option for modifications, as soon as first data collection rounds had been performed. To minimize the overall or so-called Total Survey Error (TSE, Biemer, 2010), as well as the mode effect in particular, our recommendations are also based on best practices available in this area. This starts already with the procurement of the minimal suitable equipment including a software with high user friendliness, by still meeting all the required professional constraints of such an activity, and continues with the actual survey process, which requires much closer (digital) monitoring due to the remote 4 locations of the field staff. An important part in this respect is also the close monitoring of the response rates, including the analysis of non-response data. For the latter analysis, analysis of paradata has proven to be a useful instrument (Biemer et al., 2013)1 Since Survey Solutions collects a lot of interview process data (aka paradata) as a byproduct of the actual data collection process, we will also illustrate how this data can be used, and point towards tools (I.e. scripts and applications, mostly STATA and open source) What needs to be done Preparation Choose survey staff Given the survey’s mode of data collection and unique operational context, survey staff should be chosen using two sets of criteria: 1. Skills 2. Access to essential infrastructure Skills:  Great survey track record  Past CAPI experience (required) with the software to be used for data collection (strongly preferred)  Past CATI experience (preferred)  Local language skills Access to essential infrastructure at home: • Electricity • Good phone coverage • Adequate and stable internet connection Further considerations for staffing—for example, the roles to fill—can be found in annex 1. Procure equipment To perform their job, each survey staffer needs equipment to conduct interviews, capture data, and connect with respondents and their team. Conducting interview • Phone. Call respondents. If possible, with a project-specific phone. If necessary, with a personal phone. 1 Biemer, P. P., Chen, P., & Wang, K. (2013). Using level‐of‐effort paradata in non‐response adjustments with application to field surveys. Journal of the Royal Statistical Society: Series A (Statistics in Society), 176(1), 147-168. 5 • Headset. Improve call sound quality for both interviewer and respondent. Make calls hands-free so that focus can be on tablet. Capturing data • Tablet. Administer CATI survey instrument and capture answers. • Power source. Electricity to power the tablet. Potentially a battery to recharge during any outages. Connecting • Mobile airtime. Contact respondents and members of the survey team. • Mobile internet. Send and receive interview assignments. Participate in survey-specific WhatsApp group chat, if desired. Further details on survey equipment, and on tips for procuring it before lockdown occurs, can be found in annex 2. Create payment systems For survey operations to work, payment systems are needed to pay:  Interviewer salary. To continue working, interviewers need to be paid regularly. Yet there are likely obstacles. Bureaucratic ones may arise if payroll/accounting is unavailable to process payments. Practical problems may arise if there is difficulty for interviewers in receiving payments.  Interviewer airtime. To place calls, interviewers need airtime. Yet purchasing it may not be straightforward—particularly if vendors are closed.  Respondent rewards. To both thank respondents for their time, survey operations need a way to send respondents airtime. These three payments system are vital for survey operations, but may require some resourcefulness to establish and maintain. For suggestions on how, see annex 3. Create CATI questionnaire To operate as a mobile panel survey, the questionnaire should have three components: • Preloaded information • Call attempt roster • Questionnaire content Preloaded information This information will be of a few types: • Household identifiers. Facilitate merging with the source survey data, and any analyses that involve comparisons between the last face-to-face survey and this phone survey. 6 • Household roster. Establish who lived in the household previously. Expedite the interview process. Provides opportunity to see how household composition has changed. Enables questions whose responses identify members of the household. • Phone numbers. Provides potential contacts, which can be updated, for reaching the household over the course of the survey. This information, it is worth noting, will mostly only be available where phone surveys rely on data from recent face-to-face surveys. Phone surveys that rely on random digit dialing or sampling from service provider lists may only have the phone number to call. Call attempt roster This roster will capture: • When the contact attempt was made. This will be the date and time of the call. • Which number was dialed. This will draw from previously known contacts as well as contacts discovered during earlier phone contact attempts. • Whether anyone answered the phone. This will be a yes/no question. • Why no one answered. Answers will include: no answer, invalid number, phone turned off/unreachable. • Who answered. Answers will be drawn from the household roster. • Outcome of the attempt. Answers will include: answered and available, answered but unavailable, no answer, wrong/invalid number, refused, etc. If a refusal, why refused. • When to call back. Through several questions, the interviewer will record when to contact the household next. • Notes. This text question will provide the interviewer to record any other useful information to facilitate the scheduling or conduct of future contact attempts (e.g., preferred number or contact times). This information will be used for survey management. For interviewers, it will provide a vital case history of what attempts have been made, and an indication of what needs to be done next (e.g., call back at 3PM on Saturday, consider calling at different time, etc.). For survey managers, it will provide metadata that inform key indicators of process (e.g., when most successful call attempts are being made) and of data quality (e.g., suspiciously high refusal or non-response rate for a given interviewer). Questionnaire content While the questionnaire content will be driven by public policy needs, the questionnaire style will be dictated by the mode of administration: 7 • Short interview. Length not to exceed 30 minutes2. • Simple questions and answer options. For respondents to understand, questions need to be simple and direct. For interviewers to correctly code answers, response options need to be clear and concise. • Scripted introductions. This concerns not only the introduction of the survey, but also the introduction of individual modules (e.g., “Now, I would I would like to talk about what you know about COVID-19, the coronavirus.”). These need to be scripted so that interviewers read them in language that respondents will understand. • Scripted protocols. The interviewer needs to know from the questionnaire itself what they must and must not do at all points of the interview. This concerns not only filling out sections properly (e.g., call attempt roster) but knowing how to answer a respondent’s questions (e.g., how to answer/direct questions about COVID-19 and ways to reduce risk of infection). • Available in local languages. The interviewer should have clear guidance to ask questions in the language of the respondent. Where possible, this will be local language translations such that one can toggle between translations as needed. Where not possible, efforts should still be made to translate key terms of the questions where field translation easy for interviewers (e.g., social distancing). CATI software, including Survey Solution, allows for the questionnaire to be translated into several languages, and for the interview to switch between languages as needed. These qualities are desirable for any survey, but are particularly needed for a phone survey, where respondents are more impatient and information needs to be clearly conveyed. Perform pilot Before interviewer training and survey operations begin, comprehensive piloting should be conducted in order to test all aspect of the survey machinery, including: • Questionnaire. Are questions clearly worded and well understood by interviewers and respondents? Do pre-coded response categories cover the most common responses? Are scripted or more detailed introductions needed to introduce individual questions or entire sections? Does survey the consent/introduction script provide respondents an adequate understanding of the survey? Does the interview-end script provide respondents with a clear understanding of next steps? At a minimum, this will require interview observation and respondent debriefs. If possible, it may be good to conduct cognitive interviewing. • CATI application. Does the application contain questionnaire content that is complete and correct? Does the application enable and disable the correct questions as a function of responses provided? Does the application correctly handle all expected and observed navigation paths? Should application include more guidance to the interviewer about what to do and say next? Are there any missed opportunities for quality control-checks? This requires a means for capturing observations about the CATI application’s deficiencies. During the pilot, interviewers 2 The 30-minute length should be a taken as a strict maximum. Implementers are strongly encouraged to adopt much shorter survey instruments in order to maintain higher response rates and better data quality. 8 should have a means of taking structured notes. After the pilot, the team should have debrief to discuss these issues and surface others that may not have been noted. • Survey protocols. Are survey protocols adequate for handling conditions in the field? Do some protocols need to be changed or streamlined? Do protocols need a manual or automated solution? As with the CATI application above, capturing this information requires a framework for recording observations and interviewer debriefs. • Monitoring. Do the monitoring tools work correctly form a strictly mechanical perspective? Do the monitoring tools provide adequate information for tracking survey progress and data quality? Do any monitoring tools need to be changed or streamlined? • Other supporting technology. Does the system for sending respondents credits work? Do interviewers have adequate connectivity to sync both interview and audio recordings? Does the system for sending SMS introductions work? Do SMS messages seem to help or hurt response and compliance rates? To be most effective, piloting should be iterative, if possible. When planning pilots, country teams should prefer short, frequent pilots over longer, one-off pilots. This approach both maximizes learning and minimizes time to deployment. The more pilots occur, the more issues will be observed and corrected before the survey. The shorter the pilots are, the less time needed to observe problems, devise solutions, and test those solutions. This, in effect, represents an Agile approach to survey implementation. Provide training Constraints Typically, trainings rely on face-to- face interaction between trainers and trainees. However, due to the COVID-19 pandemic, training for this survey faces a unique set of operational constraints: Venue: • Impossible/inadvisable to convene in the same physical space • Difficulty in finding a virtual space that is accessible to all participants. (e.g., bandwidth/reliability for VC, knowledge/ability to use new VC apps, etc.) Content: • Accessible on tablets/phones • Available in, or quickly translatable to, major languages (e.g., English, French, Spanish, Arabic, etc.) Availability: • Now/quickly • Format usable or adaptable in many countries with similar contexts Strategies The constraints above suggest two possible models for training. 9 1. Trainer-directed (videoconferencing) 2. Self-directed (website) The first model would involve both trainers and trainees connecting via VC at scheduled times for live, face-to-face training. This would provide a familiar mode of instruction and facilitate real-time trainer- trainee interaction. But this model would be fraught with several serious practical problems: poor connection quality, poor VC etiquette, low ability to recover from technical problems. Furthermore, this model would be quite difficult to scale up without massive manpower. The second model would entail compiling instructional materials, creating a website to host the materials, and developing frequent quizzes to test trainee progress and understanding. The advantages of this model are: accessibility with low bandwidth and basic tablets; extensibility with addition of new content or pairing with complementary modes of instruction (e.g., WhatsApp chatroom, short VC Q&A, etc.); and scalability and portability to multiple contexts. The disadvantages are that this model relies on the seriousness of trainees and that this model is new and may require trainee orientation. However, this downside risk could be easily mitigated through regular quizzing to gauge interviewer understanding and use of chatrooms to provide trainees a vehicle for raising questions. Content The training will need not only to address the typical topics—questionnaire content, survey protocols, and survey software—but also emphasize the special considerations for the mode of data collection and unique context in which it is being implemented. The interviewer training should address the following topics: • Survey content. This covers the survey objectives, the survey structure, and the meaning of questions and pre-coded answer options. • Survey protocols. This concerns the overall survey workflow how to observe good phone etiquette, how to introduce the survey, how to ask questions to respondents, how to answer questions posed by respondents, how to close the interview, and how to handle common practical actions, such as making and recording call attempts, managing workload of time, identifying next actions for each interview, scheduling follow-up interviews. • Ethics/data security. This concerns special considerations for the mode and context of data collection. This will include practical guidance on topics such as how to keep interviews private while working from home, how to prevent other members of the interviewer’s household from accessing interviews, how to explain the ways survey data will be used, and how to indicate that the interview may be recorded for quality assurance purposes. • Survey software. This will address how to use the survey software generally as well as how to use the software for common survey actions such as sending/receiving interviews, resuming interviews, leaving comments, and completing administrative parts of the questionnaire (e.g., call attempt roster, numbers associated with the household, interview results, etc.). The headquarters/supervisor/data editor training will address the following topics: • Reviewing interviews. This will cover both the mechanics of interview review as well as the things to check. 10 • Performing quality control functions. This will explain how to perform quality control actions that go beyond simply reviewing interviews. This may include: listening to interviews, conducting call-back interviews, reviewing monitoring reports. • Using software for actions above. If any actions above are supported by dedicated software, training will cover how to use that software and how to integrate its use into survey workflows. Implementation Create assignments For each sampled household, one needs to create an interview assignment. This consists of two pieces: information to identify the household (i.e., practical identifiers for phone survey and case identifiers for merging with the face-to-face survey) and information to be fed forward in order to facilitate the interview (e.g., list of members, attributes of members, etc.), Assignment creation consists of two broad steps. The first entails extracting required information from the face-to-face survey data sets. The second part involves transforming that information into the format(s) required by the CATI system. The formats are both human-readable components for facilitating content and the machine-readable component for feeding information forward into the phone interview application. Depending on the implementation strategy, the creation of interview assignments may be a one-time or ongoing process. If one-time, all assignments are created at once, and may either be distributed at once or distributed progressively. If an ongoing creation process, assignments are created as needed as data collection progresses and interviewers exhaust initial assignments. Manage survey Recruit respondents Before attempting to call a household, it is important to send an introduction message via SMS. This text message serves several purposes. First, it sensitizes the respondent to the survey and its objectives. Second, it informs respondents that someone will try to call them from a certain phone number during a certain window of time. Third, it may minimize the need for introducing the survey by shifting this introduction from the interview to the pre-interview phase. There are two ways for sending this SMS introduction: manually and automatically. The manual process would involve composing and sending an SMS, perhaps in the respondent’s language, a few days prior to the first contact attempt. This could be done either by the supervisor/survey manager before assigning the case, by the interviewer before making contact, or by a team tasked with this and other support functions. The automatic process would entail using bulk messaging services. This service, with a small amount of effort, could be linked to the CATI system such that messages precede the first phone contact. The automatic SMS system seems the preferred option for a few reasons. First, it automates a mundane, time-consuming task. This is clearly beneficial for a rapid data collection operation. Second, it notifies the respondent about the survey but, if the SMS is sent from a dedicated number, prevents respondents from contacting interviewers in advance of the interview. 11 The automatic SMS system may not, however, always be possible. The service may not be available or affordable in every country. Use of the service may also require more coordination—between those sending the SMS and the interviewer who will conduct the interview—to be practical. And while sending SMS introductions seems to increase both response and compliance rates3, this approach should be tested before being deployed at scale. It should also be noted that sending SMS will only make sense in settings where people use/read SMS and where literacy rates are sufficiently high. Distribute assignments A few days after sending an SMS introduction, the survey manager will allocate interview assignments based on:  Respondent’s language  Region origin (as proxy for language)  Gender  Respondent’s expected/known availability  Interviewer’s availability/workload Allocation based on language is particularly relevant in a country where multiple languages are spoken. In cases we can have prior information on the language spoken by the respondents, assigning the right interviewers to the specific households is essential. Afterwards, interviewers will use the CATI application to fetch their interview assignments. Make call attempt(s) Upon receiving an assignment, interviewers will attempt to contact their assigned households using the contact numbers associated with each assignment. After each call attempt, the interviewer will record: • When the attempt was made • What number was called • What was the outcome • Who answered • Whether the call was rescheduled, and if so when • Other relevant notes This will document their attempt as well as provide a basis for both the interviewer and the survey manager to decide on next actions (e.g., whether to call again, when to call again, etc.). 3 See for example the following studies: https://nces.ed.gov/FCSM/pdf/F3_DuBray_2013FCSM_AC.pdf and https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0150231. 12 Return interviews The survey protocol will dictate when interviewers should send their interviews for review. There are three clear cases:  Interview successfully completed  Respondent refuses to participate  Interviewer exhausts the number of call attempts dictated by the survey protocol The proper protocol for all other cases–for example, unsuccessful contact attempt, rescheduled interview–will be discussed in more detail in later sections. React to interviews received Once interviews have been received, the survey manager will need to react both to interviews individually (i.e., quality and scheduling) and to trends in interviews (i.e., progress, performance, and quality). Review Ideally, the survey manager will review each interview for completeness, correctness, and comments that require their attention. In other words, the survey manager will need to apply acceptance/rejection criteria to all incoming interviews. For example, completed interviews need all questions answered and all errors justified. If any of these conditions are not met, the interview should be returned to the interviewer for remediation. If all of these criteria are satisfied, it can be accepted. Audit Ideally, the survey manager would audit a certain percentage of interviews to confirm that the process being followed is correct and that the responses recorded are accurate. In a CATI call center, interview auditing is traditionally done through listing to part of the interview in order to ensure that interviewers observe proper interview etiquette, ask questions correctly, and record answers faithfully. In the current context, where interviews will be done from home by a distributed team of interviewers, interview auditing may need to be done through other means. Several approaches seem technologically possible, and implementers should select the best option—or combination of options—for their context and operational capacity. The first option is to identify and use an alternative means for listening to live interviews. This may involve routing calls through a switchboard or utilizing 3-way calls with the mobile service provider, for example. The viability of these options will be a function of the implementing agency’s pre-existing infrastructure or of its capacity to contract service providers. This option, thus, may not be available to most. The second option is to record the interview and listen to it later. The CATI data collection device, likely a tablet, would record the interview. The recordings, which are the audio files stored on the tablet, would be collected and reviewed alongside interview data. 13 This has many of the advantages of listening to live interviews. The quality assurance team can hear how interviewers conduct the interview (via the recordings) and see how they record answers (via the data). This enables tight supervision despite physical distance. But this option does have some practical limitations. The CATI data capture device would likely capture only one side of the interview: how the interview asks questions. The audio files would be potentially large files that might turn out to be difficult to upload. And effectively performing audio audits— distributing audio files, assigning audit responsibilities, acting on results—more staff and organizational capacity than may be available in some contexts (or within certain deployment timelines). Whatever the potential limitations, this option is easily accessible to most. Most CATI software packages, including Survey Solutions, can record audio during the interview. Meanwhile, the audio files can be played with the same ease as any other. The third option is to conduct call-back interviews, which involves following up with interviewed households to check whether the interview occurred, how it went, and how they answered some key questions. While this tool, in principle, allows insights on the interview, it should be used carefully. With CATI, response rates are typically low. Placing more burden on respondents may have a negative impact if not implemented carefully (e.g., lead with follow-up questions about receipt of phone credits and follow with a few questions on the interview) and selectively (e.g., perform call backs only for interviewers when there are quality concerns). Reassign Call attempts may need to be scheduled as a function of either known or expected respondent availability. After initial contact with respondents, the interviewer may learn when to call a particular respondent (e.g., call back on Saturday after 2PM). That information needs to be used in order to schedule follow-up calls. Additionally, the lessons on when is the best time to call in order to maximize the rate of contact/completion (e.g., weekend afternoons) will be accumulated over the course of time. In addition, survey managers may need to reassign interviews in order to redistribute the workload across members of the team based on their performance (e.g., move work to more productive interviewers) or to maximize chances of interview completion (e.g., move work to interviewers available in time windows with the chance of interview completion). Monitor progress The survey manager needs to monitor survey progress and interviewer effort using indicators like the following:  Contact attempts. Percentage of assignments that have been attempted.  Contact rates. Percentage of attempted assignments that resulted in contact with the household. Percentage that resulted in non-contact, broken down by outcome.  “Conversion” rate. Percentage of assignments with successful contact that resulted in agreement and refusal.  Attempts by time. Number of attempts made by time window. 14  Contacts by time. Number of successful contacts made by time window. These metrics need to be tracked both at the level of each interviewer and at the level of the survey overall. Monitor data quality The survey manager also needs to monitor indicators of data quality. For example:  Interview length. Whether interviews are suspiciously short or long.  Reasons households not reached. Whether interviewers have a comparatively high rate of unreached households, or a propensity to select certain reasons for non-contact (e.g., no answer, phone not on, etc.).  Reasons households unreachable. Whether interviewers have a comparatively high rate of unreachable households (e.g., invalid number, disconnected number) Like the progress metrics, these need to be tracked both overall and at the level of each interviewer. What can be done, by time horizon How to implement the CATI system will be a function of when the survey is deployed:  Now. That is, the survey will start 1-2 weeks from now  In 2 weeks. That is, data collection starting 2-3 weeks from now. Depending on the deployment date, the implementation models will differ in at least three areas:  Training  Survey workflow  Survey management The differences will reflect, simply put, what preparations can reasonably be done to support the high- frequency phone surveys. For surveys deployed now, little preparation can be done in the limited time available, and support will largely be ad hoc and reactive. For surveys deployed in 2 weeks, significant preparation can be done, and support will provide global solutions to address common problems. Short term To deploy immediately, one needs to move forward with resources readily at hand. Training Videoconference is the easiest training model to deploy on short notice. Beyond a questionnaire and a CATI application, it does not require many materials to be developed in advance. Through interaction with trainees, trainers can gauge trainees’ understanding of materials. And setup costs–which principally involve scheduling a series of VCs–are relatively low. For these reason, video-conference training is the training modality recommended for surveys to be deployed immediately. To staff the training, training teams in each country will need three sets of content expertise:  Survey protocols 15  Questionnaire content  Survey software tools In practical terms, TTLs will need to identify the necessary human resources, schedule training times that suit both trainers and trainees, and identify VC technology that can support the training (e.g., easy to use for trainees, low bandwidth requirements, screen sharing capability, etc.). If training materials already exist, they can be used as a complement or substitute for live, face-to-face training. For example, Survey Solutions has a video training series for interviewers and supervisors. But where training materials do not exist or are not adequately tailored to this survey’s needs, face-to-face VC training will be required. Survey workflow The optimal CATI workflow would allow survey managers to supervise interviewers’ work continuously throughout the process (e.g., after each call attempt). But the workflow that is easiest to deploy is essentially a CAPI one that can be described as follows: • Team sends SMS message manually • Managers create assignments • Managers distribute assignments to interviewers • Interviewers manage in decentralized way • Make call attempts • Conduct interviews • Return assignments only when completed (i.e., success, refusal, exhaust call attempts) • Managers react to returned interviews This workflow could be deployed immediately without any effort on the part of the implementing agency or of the CATI support team. Survey management Most CATI surveys require strong survey management. This is true for at least two reasons. First, survey managers need to observe bottlenecks in progress and adjust accordingly (e.g., intensify work on nights and weekends to complete interviews). This requires constant updates on progress and efforts made by interviewers. Second, survey managers need to identify and respond to worrisome patterns in data– particularly when interviews are being done remotely and there is limited scope for direct supervision. To meet these needs, each country survey team would need to develop their own survey management reports. For lack of time, the CATI team will only be able to provide limited support in this area. Resources available Country teams can expect the following resources: Training For training on survey content: • Excel format questionnaire 16 • Interviewer manual • Slides on questionnaire content and survey protocols For training on CATI software tools: • CATI questionnaire • Training videos on Survey Solutions’ website All the above—in particular, the questionnaire, CATI application, and manual—will need to be adapted to the country context and needs. Survey workflow • CATI system that uses CAPI functionality. For Survey Solutions, the system works out of the box. Survey management If country teams would like to follow the recommendations above, they will need to develop their own scripts. Of course, these scripts can be shared once available in a few weeks. Medium term For surveys to be deployed two weeks (or more) from now, much can be done to improve training, refine management tools, and streamline survey operations. Training With more time available for the task, the approach to training will consist of:  Developing/tailoring training materials for the survey (e.g., slides, short videos, how-to articles)  Offering instruction through training websites  Providing additional training support on an ad hoc basis as needed Development of training materials will entail:  Compiling/tailoring existing materials (e.g., YouTube SuSo interviewer training videos, content from the Survey Solutions support site)  Developing new materials to fill gaps (e.g., survey workflow, questionnaire content, using survey management scripts)  Developing quizzes (Survey Solutions questionnaires) that help check interviewers’ comprehension (and programs for scoring quizzes) Deployment of training will involve:  Creating websites with quick and simple tools (i.e., developing content in Markdown files, building a static site with Hugo) 17  Making content free and open source for easy adoption and adaptation (i.e., post site and its contents in a GitHub repository)  Hosting websites with free services (e.g., GitHub Pages, Netlify, etc.) Survey workflow The proposed workflow is as follows: • Managers create assignments  Managers distribute assignments to interviewers  Interviewers make call attempts for assigned interviews – Record call attempt metadata (i.e., timing, outcome) – Return interviews to manager at the end of each day  Managers react to returned interviews, and determine next steps – Assign interview to interviewer – Pause non-responding cases for later – Adopt different tactics to elicit response (e.g., send SMS to respondent)  Managers monitor and react to trends – In call attempts (e.g., best/worst call times) – In performance of interviewers (e.g., # call attempts, success rate, non-response rate) – In data (e.g., quality indicators, indications of shirking, etc.) This workflow is motivated principally by survey management needs–that is, to manage non-sampling error. By having interviewers return interviews at the end of each day, the system can observe what work has been done, and produce reports for survey managers on survey progress and data quality trends. The workflow will be supported by scripts to facilitate each stage in the process. For creation of assignments, scripts will be created to:  Evaluate whether phone numbers are valid  Implement sampling design (potentially)  Preload information from the face-to-face interview These scripts will be developed to meet the needs of the first surveys deployed in this wave (e.g., LSMS, WAEMU, etc.), and may need some adaptation for a given country deployment. For reacting to all the interviews returned daily, scripts to:  Evaluate completed interviews (e.g., complete, refusal, exhausted call attempts) – Reject if incomplete (i.e., questions left unanswered) or if contains critical errors (e.g., contradictory information) – Accept if complete and without critical errors  Return incomplete interviews For survey monitoring, scripts to create a report on:  Survey progress 18  Data quality These reports will provide information both at the level of the survey overall and of each interviewer. Survey management The survey management toolkit will consist of two reports. The first monitors indicators of survey progress and interviewer effort:  Contact attempts. Percentage of assignments that have been attempted.  Contact rates. Percentage of attempted assignments that resulted in contact with the household. Percentage that resulted in non-contact, broken down by outcome.  “Conversion” rate. Percentage of assignments with successful contact that resulted in agreement and refusal.  Attempts by time bin. When attempts are being made.  Contacts by time. When successful contacts are being made. The second report monitors indicators of data quality. For example:  Interview length. Whether interviews are suspiciously short or long.  Reasons households not reached. Whether interviewers have a comparatively high rate of unreached households, or a propensity to select certain reasons for non-contact (e.g., no answer, phone not on, etc.).  Reasons households unreachable. Whether interviewers have a comparatively high rate of unreachable households (e.g., invalid number, disconnected number) All metrics will also be tracked both overall and at the level of each interviewer Resources available Training For training on survey protocols, questionnaire content, and survey software: • Website to host training materials • Training materials posted on the website • Quizzes • Adaptation and deployment guide The materials above will be provided as a template. The content will be developed to reflect the recommendations made by the questionnaire and implementation working groups. Each country team will need to adapt them to their specific needs, using the adaptation guide to facilitate this process. This will entail at least the following actions. First, the country team will need to create their own copy for the training website—likely forking a GitHub repository. Second, the country team will need to make adaptations to content (e.g., changing the materials to reflect country-specific questions, translating them into the country’s language, etc.). Each country team may also need to deploy their own country website. The deployment guide will provide a detailed roadmap for this process. This will likely consist of creating their own GitHub repository and publishing their training side using GitHub Pages. 19 The expectation should be that each country will be responsible for their own deployment, but that human resources to facilitate the process may be available, subject to constraints of availability and financing. Survey workflow and management To help execute common, complex, time-consuming tasks in the survey workflow: • Website to that hosts scripts • Scripts that accomplish actions in workflow: o Validate phone numbers in source data o Send SMS message to respondent o Create interview assignments with preloaded information o Reject incomplete interviews o Evaluate whether to accept/reject completed interviews o Create monitoring reports on survey progress, interviewer effort, and data quality The materials above will be offered as a template. They will be designed to satisfy known needs, but may require adaptation to work in a particular context. At a best, they will require that the user provide parameters (e.g., address, login, and password for site where data are hosted). At worst, they may require some adaptation (e.g., changing output text from English to French). These materials and their supporting documentation will be made available on one or more GitHub repositories, and may be updated by their authors over time to fix bugs or add features. Depending on staff availability and funding for the development effort, these scripts can be provided either on an as-is basis or as supported and maintained software. Longer term Even after phone surveys begin, needs will persist. In the immediate term, country teams may need support for the tools being proposed in this document. This support will be around use, adaptation, and refinement of these software tools. In the longer term, the World Bank will need a robust, ready-to-use software toolkit for filling the data gap during similar crises and other emergency situations. Both these activities require additional funding. Support short-term solutions The implementation working group will develop several scripts to facilitate the survey process, but all code requires support, maintenance, and refinement over time. To support these scripts, the group proposes to develop vignettes that explain scripts through extended examples and highlight solutions for common use cases. This will supplement technical documentation (e.g., what is expected for each of a function’s arguments). To maintain these scripts, the group proposes to refine code as issues flagged in the field, to introduce unit tests to identify issues before they arise, and track changes over time in a user-friendly way. To refine the scripts, the group will add, modify, or extend scripts as a function of input from the field. Build longer-term solutions To meet pressing needs, the implementation group will be constructing short-term solutions based on its understanding of the implementation problems. During data collection, the group will have a unique opportunity to move from short-term solutions, quickly crafted for today, to long-term solutions, 20 carefully constructed for tomorrow. First, the group will be able to test ideas through short-term solutions and iteratively improve those solutions. For example, scripts used in Nigeria, the first phone survey, can be tested, revised, and improved for WAEMU, the next set of phone surveys. Second, the group will be able to improve its understanding of the problem to come up with the best solutions for addressing it. On a practical level, the group will be able to observe the problems implementers actually have, rather than those they imagine ex ante that they might have. These longer-term solutions are the object of a separate proposal. 21 Annex 1: Survey staffing To collect data at all, there are several roles that must be filled:  Survey manager. Oversees survey operations. Monitors survey progress, interviewer performance, and data quality. Manages distribution of work between interviewers.  Interviewer. Conducts interviews. To collect data either at larger scale or at higher quality, there are also a few roles to consider filling:  Supervisor. If the survey requires more interviewers than the survey manager can effectively monitor, supervisors may be needed to play a middle-management role.  Quality assurance. If quality assurance requires time-consuming manual actions—for example, conducting follow-up interviews or listening in on interviews—dedicated quality assurance may be required. 22 Annex 2: Procurement In the current context, when lockdowns can come into effect from one day to the next, there are two important sets of consideration for getting equipment: 1. What to get 2. When to get it The first point provides a list of equipment. The second point is a reminder of the urgency of action, and some tips on taking actions that anticipate a changing operational environment. What For interviewers: • Tablet. Conduct interview and capture answers. • Internet access. Send and receive interview assignments. Participate in survey-specific WhatsApp group chat, if desired. • Mobile router. Provide Wi-Fi internet access to tablet, unless tablet has a SIM card or connectivity is already available at location where the interviewers work. • Phone. Call respondents and survey manager(s). • Headset. Make calls hands-free so that focus can be on tablet. Improve sound quality for both interviewer and respondent. • SIM card. If possible, strongly prefer new SIM used exclusively for this operation. If not possible, personal SIM. • Phone credits. Needed for contacting respondents and survey managers. For supervisors: • Laptop. Manage survey operations from web-based platform. • Tablet. Conduct interview and capture answers, if needed. Replicate issues reported from interviewers. • Internet access. Connect to web-based platform(s), contact WB remote support. • Mobile router. Provide Wi-Fi internet access for laptop and tablet. • Phone. Call interviewers for updates/support, and respondents for interviews/call-backs. • SIM card. If possible, new SIM for this operation. If not possible, personal SIM. • Phone credits. Needed for contacting interviewers, WB in-country support, and respondents. Needed for mobile data plan. For the survey overall, it is prudent to secure additional survey equipment in order to guard against failure, breakage, or loss of essential devices. In particular: • Tablets • Phones • Headsets • Laptop(s) • Mobile routers • Power cords 23 When Because the situation may deteriorate rapidly–with the inability to access equipment stores, or the imposition of home-based work–undertake the following steps as soon as possible:  Take stock of existing equipment. Of the necessary equipment, see what is available and what is missing. In this stock-taking exercise, confirm that the equipment works properly.  Get equipment out of storage. If equipment is in storage or in offices, preemptively remove that equipment from its current location. Make equipment available for rapid distribution in the event that restrictions are imposed suddenly (e.g., safely stored with a trusted survey manager).  Identify methods of expedited procurement. In anticipation of the needing to procure additional equipment, explore several avenues for expedited procurement. These may include: local procurement through the country office, local procurement in a nearby country office, procurement and shipping from HQ.  Procure the equipment that is missing. If any necessary equipment is not available or not working, identify the fastest way to procure this. In some cases, this may be procurement in local markets. In others, this may be expedited procurement at and shipping from HQ. 24 Annex 3: Establishing a payment system Creating a facility for sending phone/internet credits Interviewers are working under extra-ordinary conditions, therefore support in equipment and resources is essential. To perform their work, survey teams need: • Mobile airtime • Mobile data bundle In many contexts, these means of communication need to be regularly replenished (e.g., purchase a new data bundle when the plan is exhausted or expires). Should some form of social distancing come into effect, this routine action may become difficult. The following problems may arise:  Impossible to transfer money physically, due to restrictions on movement  Points of sale for airtime and data bundles may shut down, due to public order or economic downturn Yet, in many contexts, there may be natural solutions to these practical problems: • Purchase service subscriptions. • Transfer credits directly to device SIM card. • Send mobile money to interviewer. And always provide enough buffer credit, so interviewers don’t reach the limit to early on. Creating a facility for paying interviewers To continue working, interviewers need to be paid regularly. Yet the same problems outlined above for sending phone/internet credits apply more acutely to paying interviewers. The problem is two-fold: 1. Developing a mechanism for transmitting payments to interviewers (e.g., send mobile money transfer) 2. Ensuring that payroll functions operate (e.g., institution receives requests, approves requests, and issues payment) The first problem is largely technical. The second problem is organizational. In anticipation of these problems:  Identify a method for transferring money  Start conversations with payroll/accounting to ensure there is mechanism for making payments Investigate the option of the World Bank paying interviewers directly 25 Annex 4: Training needs Training needs Like for any survey, there are three levels of training: 1. Trainers/facilitators 2. Interviewers 3. Supervisors/survey managers Each training has its own audience, objectives, and content. Trainers/facilitators The objective is to understand: • Instructional content • Instructional approach In other words, trainers need to know what they're teaching, and how to teach it through remote (virtual) training. The content will be: • Condensed content of interviewer and supervisor trainings • Approach to training, given the training format Interviewers The objectives are to understand and be able to apply: • Survey protocols • Questionnaire content • Survey software tools The content will cover each of the topics above. The focus will be on survey protocols and the questionnaire. The assumption is that interviewers will already be familiar with the CATI software chosen for data collection. The survey software tools section of the training will be a refresher, with emphasis on the parts of the system most needed for this assignment. The understanding of interviewers will be tested through regular quizzes and, if possible, practical exercises and simulations, including mock interviews, by conducting full interviews with dummy respondents. Supervisors/survey managers, The objectives are to understand: • Role in the survey process 26 • Software tool(s) for performing their role • How to take common actions • How to report issues The content will be cover each of the topics above. The emphasis of training will be on how to execute common actions needed for this assignment. The assumption is that the supervisors/survey managers will be familiar with the CATI software and need only a refresher training. The understanding of supervisors/survey managers will be tested through regular quizzes and, if possible, practical exercises and simulations. 27