Policy Research Working Paper 9660 Representativeness of Individual-Level Data in COVID-19 Phone Surveys Findings from Sub-Saharan Africa Joshua Brubaker Talip Kilic Philip Wollburg Development Economics Development Data Group May 2021 Policy Research Working Paper 9660 Abstract The COVID-19 pandemic has created urgent demand for general adult population. To improve the representativeness timely data, leading to a surge in mobile phone surveys for of individual-level phone survey data, the household-level tracking the impacts of and responses to the pandemic. This phone survey sampling weights are calibrated based on pro- paper assesses, and attempts to mitigate, selection biases pensity score adjustments that are derived from a model of in individual-level analyses based on phone survey data. an individual’s likelihood of being interviewed as a function The research uses data from (i) national phone surveys that of individual- and household-level attributes. Reweight- have been implemented in Ethiopia, Malawi, Nigeria, and ing improves the representativeness of the estimates for the Uganda during the pandemic, and (ii) the pre-COVID-19 phone survey respondents, moving them closer to those of national face-to-face surveys that served as the sampling the general adult population. This holds for women and frames for the phone surveys. The availability of pre- men and a range of demographic, education, and labor COVID-19 face-to-face survey data permits comparisons of market outcomes. However, reweighting increases the vari- phone survey respondents with the general adult population. ance of the estimates and fails to overcome selection biases. Phone survey respondents are more likely to be household Obtaining reliable data on men and women through phone heads or their spouses and non-farm enterprise owners, surveys requires random selection of adult interviewees and on average, are older and better educated vis-à-vis the within sampled households. This paper is a product of the Development Data Group, Development Economics. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at tkilic@worldbank.org or pwollburg@worldbank.org. The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Produced by the Research Support Team Representativeness of Individual-Level Data in COVID-19 Phone Surveys: Findings from Sub-Saharan Africa Joshua Brubaker †, Talip Kilic‡, and Philip Wollburg* 1 JEL Codes: C81, C83, O12. Keywords: COVID-19, survey methodology, survey sampling, weighting methods, phone surveys. 1 The authors are listed alphabetically. † Independent Consultant, Living Standards Measurement Study, Development Data Group, World Bank, jbrubaker@worldbank.org. ‡ Senior Economist, Living Standards Measurement Study, Development Data Group, World Bank, tkilic@worldbank.org. * Economist, Living Standards Measurement Study, Development Data Group, World Bank, pwollburg@worldbank.org. The authors would like to thank Kathleen Beegle, Calogero Carletto, Isabela Coelho, and Kristen Himelein for their comments on the earlier versions of this paper. 1. Introduction With the onset of the coronavirus disease 2019 (COVID-19) pandemic, governments, academic institutions, and international organizations have scrambled to measure and monitor the pandemic’s impacts on livelihoods and tailor policy responses. A global survey of National Statistical Offices (NSOs) shows that over 80 percent are involved in collecting data related to the COVID-19 pandemic, focusing predominantly on its socioeconomic and business impacts. However, prompted by lockdowns, travel restrictions and safety concerns, face-to-face (F2F) survey data collection was suspended in the overwhelming majority of countries at the onset of the pandemic, and since then, the movement to resume F2F surveys, even under strict COVID-19 fieldwork protocols, has been slow, with a considerable lack of clarity regarding the timeline for fully resuming activities under the “new normal” (UNSD and World Bank, 2020c). These developments have led to a proliferation of telephone surveys for collecting data on COVID-19 impacts, with the majority of NSOs relying on them for this purpose (Fu and Schweinfest, 2020; UNSD and World Bank, 2020a). Similarly, the World Bank has launched a global initiative to monitor COVID-19 impacts using phone surveys as has UN Women, Innovations for Poverty Action and Young Lives, just to name a few (Fu and Sanchez-Paramo, 2020; IPA, 2020; Young Lives, 2020; UN Women, 2020). Nationally representative phone surveys had previously been rather uncommon in low-income countries, so relatively little was known about their feasibility and about best practices for their design and implementation in these contexts. Phone surveys have several advantages that make them a suitable tool for data collection during a crisis, most importantly in terms of the cost and the speed at which they can be deployed. Phone surveys can also be fielded at higher frequencies than traditional F2F surveys, enabling flexible questionnaire designs to respond to evolving data needs. There is considerable flexibility also in the mode which phone surveys can employ: in Computer Assisted Telephone Interviews (CATI) an enumerator interviews respondents over the phone and enters their responses with the help of a computer-based questionnaire; in Interactive Voice Response (IVR) respondents receive the interview questions through automated, pre-recorded messages; and in SMS-based surveys, respondents receive survey question via text message (Lau et al., 2019a). Phone survey sampling may be based on phone numbers collected in a previous F2F survey; may use a list of phone numbers otherwise obtained, for example from a mobile network operator; or may create phone numbers using Random Digit Dialing (RDD; Himelein et al., 2020). However, phone surveys are more prone to various forms of selection biases than F2F surveys. First, phone surveys usually require phone ownership, which is not universal, particularly in low- income countries, and is skewed towards men and in general, individuals in wealthier, male- headed, urban, better-educated households (Henderson and Rosenbaum, 2020). As such, certain parts of the population are likely under-represented in phone surveys, leading to coverage bias. Second, phone survey response rates are lower than in face-to-face surveys because respondents do not pick up, refuse at higher rates, or phone numbers may be disconnected. This leads to non- response bias when responding households are systematically different from households that do 2 not respond The severity of these biases may vary depending both on the phone survey mode (CATI, IVR, SMS) and on the sampling strategy, with RDD-based phone surveys typically resulting in higher non-response rates. Third, respondent selection is a potential source of bias in individual-level phone survey data collection. In particular, the respondent selection protocol in phone surveys may target heads of households or “most knowledgeable” adult household members, such that the sample of respondents may not be representative of the general adult population. Selecting the “most knowledgeable” adult as a respondent is a common practice in household surveys, whether face- to-face or telephone, and concerns with individual-level representativeness arise from this choice in all cases. However, interviewing household members other than the main respondent, or asking the main respondent to report information on behalf of other household members, is considerably easier and more common in F2F surveys. RDD-based phone surveys, typically based on individual phone numbers, are more likely to avoid the respondent selection problem. Obtaining unbiased, representative data from phone surveys is, therefore, a significant challenge, affecting both household-level and individual-level analyses. In this paper, we assess, and attempt to mitigate, the selection biases that may be inherent in individual-level analyses conducted with phone survey data. 2 Individual-level analysis is important in this context because monitoring attitudes, knowledge of, and perceptions around the COVID-19 pandemic in phone surveys is in many instances an individual-level exercise, for example when asking about attitudes towards vaccinations or knowledge of measures to prevent the spread of the virus. Moreover, individual-level data is critical to properly understanding the differential impacts of COVID-19, by gender, age group, or other subpopulations of interest. Several recent studies suggest that COVID-19 impacted women differently than men along a host of dimensions (Okoro and Pretittore, 2020; Quisumbing et al., 2020). Since phone ownership is skewed away from women and vulnerable populations, surveying these groups well is a particular challenge, especially in the context of the COVID-19 pandemic (Alvi et al., 2020; Roy Mathur, 2020). Our analysis leverages data from two sources: (1) the national phone surveys on COVID-19 in Ethiopia, Malawi, Nigeria and Uganda, which were implemented through live CATI interviews on a monthly basis since the beginning of the pandemic, and (2) the nationally-representative, multi-topic longitudinal F2F surveys that had been implemented prior to the pandemic under the World Bank Living Standards Measurement Study – Integrated Surveys on Agriculture (LSMS- ISA) program and that served as the sampling frames for the phone surveys. The F2F surveys collected the phone numbers of at least one individual per household, and in some cases of all household members. When no one in the household owned a phone, survey teams had attempted to record the phone number of a contact person outside the household. F2F survey households with at least one phone number were called in each phone survey, with the first contact usually being the head of household and all household member contacts being called before reference contacts 2 In a similar analysis, Ambel et al. (forthcoming) quantify and correct for selection biases at the household level. 3 outside the household (section 2.1). The phone survey sampling weights in these data sets are recalibrated versions of the F2F household survey sampling weights and deal with coverage and non-response biases at the household-level, leveraging the rich, pre-COVID-19 face-to-face survey data on (i) households that participated in the HFPS and (ii) households that did not participate in the HFPS but participated in the pre-COVID-19 face-to-face survey.3 The availability of pre-COVID-19 face-to-face survey data for phone survey respondents as well as all adult household members that were not interviewed on the phone during the pandemic allow us to compare phone survey respondents with the general adult population. The analysis reveals that concerns regarding the representativeness of household-level phone survey data (due to lack of universal phone ownership and high non-response rates) are even greater for individual-level phone survey data. Selected phone survey respondents are most often household heads or their spouses, and on average, are older, better educated and more likely to own a non-farm enterprise vis-à-vis the general adult population. To account for these differences and improve the representativeness of individual-level phone survey data, we recalibrate the household-level phone survey sampling weights based on propensity score adjustments that are derived from a cross- country comparable model of an adult individual’s likelihood of being interviewed as a function of a rich set of individual- and household-level attributes. Reweighting generally improves the representativeness of the individual-level estimates, moving the variable means for phone survey respondents closer to those of the general adult population. This holds for both women and men and for a range of demographic, education, and labor market outcomes. However, reweighting increases the variance of the estimates and fails to fully overcome selection biases, with differences in means remaining statistically significant for the majority of the outcomes. Obtaining reliable individual-level data from these phone surveys, therefore, requires fundamental changes to the individual respondent selection protocols, with a focus on random selection of interviewees. Our paper is related to a growing literature on methodology and best practices for designing and conducting phone surveys in low-income countries (see Himelein et. al., 2020, and L’Engle et al., 2018 on sampling; Greenleaf et al., 2020, Henderson and Rosenbaum, 2020, and Lau et al., 2019a on survey mode; Ballivian et al., 2015, Dabalen et al., 2016, Gibson et al., 2019, Lau and di Tada, 2018, Leo et al., 2015, McKenzie, 2012, Özler and Cuevas, 2019 on survey cost, non-response, attrition , and use of incentives; Glazerman et al., 2020, Lau et al., 2019b, and Palacios-Lopez et al., 2020 on questionnaire design , among others).4 Closely related to our research are several studies focusing on the representativeness of phone survey data and the role of respondent selection. Ambel et al. (forthcoming) assess the household- 3 Households that did not participate in the HFPS are those for which no contact phone number could be collected in the pre-COVID-19 face-to-face survey as well as those with contact phone numbers that survey teams attempted to call but who did not pick up or whose numbers had been disconnected. 4 There are several guidebooks and syntheses that compile best practices and experiences regarding phone surveys, including Leo et al. (2015), Ballivian et al. (2015), Dabalen et al. (2016) from before the COVID-19 pandemic and Glazerman et al. (2020), Henderson and Rosenbaum (2020), and Gourlay et al. (forthcoming) in the context of the COVID-19 pandemic. 4 level coverage and non-response biases using data from the same phone surveys that inform our analysis. They show that recalibrating household weights is relatively successful at overcoming these biases at the household level – somewhat contrary to what we find for individual-level data. Both our paper and Ambel et al. (forthcoming) use a similar reweighting model as Etang and Himelein (2020) who recalibrated phone survey weights in the context of the Ebola crisis to improve the representativeness of survey respondents vis-à-vis face-to-face survey respondents in Liberia and Sierra Leone. Our paper differs by virtue of our attempt at assessing the individual- level phone survey data’s representativeness of the entire adult population. Larmarange et al. (2016) and Velthausz et al. (2016) find that mobile phone surveys represent a population that is younger, more urban, and more likely to be male and better educated. This is confirmed by Henderson and Rosenbaum (2020) who review 15 phone-based studies from 11 countries, and Lau et al. (2019b), who document that sampling for an SMS-based phone survey favors more technologically savvy respondents. Moreover, significant sample selection biases compared to face-to-face data collection have been documented in phone surveys, irrespective of phone survey modes (CATI, IVR or SMS) (Lau et al. 2019a). The remainder of the paper is structured as follows. Section 2 describes the data and methods we use to assess individual-level biases and the relative success of bias reduction techniques. Section 3 presents the main emerging findings. Section 4 concludes with a discussion of what the results mean for individual-level analysis and data collection using phone surveys. 2. Data and Methods The longitudinal survey data informing our analysis originate from (i) the national high-frequency phone survey (HFPS) that was implemented on a monthly basis in Ethiopia, Malawi, Nigeria and Uganda during the COVID-19 pandemic, and (ii) the pre-COVID-19 F2F household survey that served as a sampling frame for each HFPS. Each pre-COVID-19 F2F survey that was the source of the phone numbers in each country had been designed to be representative at the national, regional, and urban/rural levels. These F2F surveys are the Ethiopia Socio-economic Survey (ESS) 2018/19, the Malawi Integrated Household Panel Survey (IHPS) 2019, the Nigeria General Household Survey (GHS) - Panel 2018/19, and the Uganda National Panel Survey (UNPS) 2019/20. In Ethiopia, Malawi, and Uganda, the HFPS attempted to call all pre-COVID-19 F2F survey households for which at least one phone number was available. The Nigeria HFPS first drew a national sub-sample from the universe of F2F survey households with contact details, based on a balanced sampling approach using the cube method (Tille 2006), and then this sub-sample of households was contacted. In Ethiopia, we use data from the first round of the HFPS, which was implemented in April-May 2020, covering 3,249 households. In Malawi, we use data from the first and fifth rounds of the HFPS, which were implemented in May-June 2020 and October-November 2020, covering 1,729 and 1,589 households, respectively. Similarly, in Nigeria, we use data from the first and fifth rounds of the HFPS, which were implemented in April-May 2020 and September 2020, covering 5 1,950 and 1,774 households, respectively. In the specific case of Malawi and Nigeria, we use the fifth round of the HFPS to take advantage of the individual-level employment data that was collected on all adults in each household as reported by the main phone survey respondent. Lastly, in Uganda, we use data from the first round of the HFPS, which was implemented in June 2020, covering 2,227 households. The implementing agencies for the national phone surveys in Ethiopia, Malawi, Nigeria, and Uganda, are, respectively, Laterite Ethiopia, Malawi National Statistical Office, Nigeria Bureau of Statistics, and Uganda Bureau of Statistics. The anonymized, unit-record phone survey data associated with each monthly survey round, together with the questionnaire, basic information document and interviewer manual for that round, are made publicly available based on agreements with each national statistical office (NSO), within approximately four to six weeks of completion of phone interviews, through the World Bank Microdata Library, under the High-Frequency Phone Survey collection. 5 The approach to the phone survey questionnaire design and sampling is comparable across countries, and is informed by the template questionnaire, the phone survey sampling guidelines and the computer-assisted telephone interviewing (CATI) guidelines that have been made publicly available by the World Bank.6 Since the phone surveys build on the face-to-face surveys, and the phone survey respondent was recorded in the publicly available data, we can link the phone survey data with the pre-COVID-19 F2F survey data at the individual-level. This gives us two samples to compare: (i) the phone survey respondents and (ii) the general adult population (defined as individuals 15 and above 7), of which the phone surveys are a subsample. Our analysis assesses the differences between phone survey respondents and the general adult population as represented in the pre-COVID-19 F2F surveys and gauges the success in utilizing bias correction techniques to derive general adult population representative estimates for a core set of individual-level variables related to gender, age, marital status, relationship with the household head, education, and employment. Appendix Table A1 shows the unweighted means of these variables for the samples of interest.8 5 The World Bank Microdata Library HFPS collection can be accessed here: http://bit.ly/microdata-hfps.The World Bank HFPS is the preferred platform among the NSOs in Ethiopia, Malawi, Nigeria, and Uganda regarding the public dissemination of other NSO-owned surveys. 6 These can be accessed through the following links: Template Questionnaire: http://bit.ly/templateqx ; Manual: http://bit.ly/interviewermanual; Sampling Guidelines: http://bit.ly/samplingguidelines; and CATI Guidelines: https://bit.ly/phonesurveyCATI. 7 This cutoff was chosen because individuals 15 and older were eligible to be respondents in the HFPS and the F2F surveys. 8 The individual-level variables in Appendix Table A1 originate from the pre-COVID-19 F2F survey in each country; are all dichotomous; and include variables that identify, within the entire sample of F2F survey adults and separately within the sample of HFPS respondents, whether the individual (i) is a household head, spouse of head or child of head; (ii) male; (iii) is in the age category of 15-24, 25-49 or 50+; (iv) is married, (v) is literate (i.e. can read and write in any language), (vi) has no educational degree, a primary education degree, a secondary education degree, a post- secondary certificate or a post-secondary degree, (vii) was sick in the last 2-4 weeks; (viii) is chronically ill or disabled; 6 2.1. Sampling frames, contact protocols, and response rates The protocols for contacting the sampled households and subsequently selecting the respondent in each household were slightly different in each HFPS. In Malawi, the IHPS 2019 was the sampling frame for the HFPS. During the IHPS 2019, phone numbers were collected from the sampled households in two ways: First, each household member’s phone number was collected during the interview and recorded as part of the household roster, provided that the individual had a phone number. Second, phone numbers for up to three non-household reference contacts, such as neighbors or friends, were taken at the beginning of the interview. Prior to the implementation of the first round of the HFPS, the order of phone numbers for household members was randomized in each household and augmented with the randomized list of all reference contacts. During the first round of the HFPS, the enumerators called the phone numbers in accordance with the resulting order in each household. In the follow up rounds, the first phone number to be called was the one that the respondent of the first round indicated as the best number to reach him or her. The original list of phone numbers was retained in the event that the preferred phone number cannot be reached. Of the 3,181 IHPS 2019 households that were interviewed face-to-face, 2,337 provided at least one phone number and all of these households were attempted to be contacted by the HFPS. Of the attempted households, 1,729 households were fully interviewed in the first round, a response rate of 74 percent. In Ethiopia, the ESS 2018/19 was the sampling frame for the HFPS. The ESS 2018/19 interviewed 6,770 households which were asked to provide phone numbers for the head of household, up to three additional household members and up to two non-household reference individuals. At least one phone number was obtained for 5,374 ESS 2018/19 households. The enumerators called the available phone numbers for each household in the order in which they were recorded during the ESS 2018/19 interview. During the first round of the HFPS, all 5,374 households were attempted to be contacted, of whom 3,249 were successfully interviewed, for a final response rate of 60 percent. In Nigeria, the GHS-Panel 2018/19 was the sampling frame for the HFPS. The GHS-Panel 2018/19 interviewed 4,976 households, of whom 4,934 provided phone numbers, from which 3,000 were randomly selected to be contacted by the first round of the HFPS. The contact protocol targeted the household head, who was called first if his or her number was listed, followed by the other numbers of household members and the reference contacts, in the order in which they were captured by the GHS-Panel 2018/19. During the first round of the HFPS, 1,950 households were (ix) was employed for a wage/salary during the last 12 months; (x) was an owner of a non-farm household enterprise that was operational during the last 12 months; (xi) performed any piecemeal/casual work during the last 12 months; and (xii) owns a mobile phone. The household-level variables associated with the individuals underlying the estimates in Table A1 include (i) number of household members (i.e. household size), and (ii) five dichotomous variables identifying the pre-COVID-19 annual consumption expenditure per capita quintile for each household. 7 successfully interviewed out of 3,000 households attempted, equivalent to a 65 percent response rate. Finally, in Uganda, the UNPS 2019/20 was the sampling frame for the HFPS. The UNPS 2019/20 interviewed 3,098 households, of whom 2,386 provided a phone number for at least one household member or a reference contact. The HFPS attempted to contact all 2,386 households, of whom 2,227 were successfully interviewed, markedly the highest response rate in our sample at 93 percent. Like Nigeria, the Uganda HFPS contact protocol prioritized the household head, followed by other household members, and referenced contacts, in the order in which they were captured during the UNPS 2019/20. Table 1. Selection of HFPS Households Ethiopia Malawi Nigeria Uganda Sample Households (HHs) N % N % N % N % Face-to-face (F2F) HH sample 6,770 100 3,181 100 4,976 100 3,098 100 HHs with phone numbers 5,374 79.4 2,337 73.5 4,934 99.2 2,386 77.0 HHs called by HFPS 5,374 79.4 2,337 73.5 3,000 60.3 2,386 77.0 HHs reached by HFPS 3,357 49.6 1,743 54.8 2,057 41.3 2,246 72.5 HHs successfully interviewed by HFPS 3,249 48.0 1,729 54.4 1,950 39.2 2,227 71.9 HHs successfully interviewed by HFPS with the phone survey respondent also 3,196 47.2 1,701 53.5 1,910 38.4 2,128 68.7 appearing in the F2F survey In the F2F survey database, there are a total of 17,563 adults in Ethiopia, 8,588 in Malawi, 15,230 in Nigeria, and 8,763 in Uganda – across all households, irrespective of being contacted and/or interviewed by the HFPS rounds that are used in our analysis. Of these adults, 8,004 in Ethiopia, 4,670 in Malawi, 6,178 in Nigeria, and 6,361 belonged to F2F survey households that were also interviewed by the first round of the HFPS. Table 2 presents unweighted descriptive statistics for (i) individuals that were respondents in successfully interviewed HFPS households in round 1 (i.e. phone survey respondents), and (ii) all adults living in F2F survey households, irrespective of being contacted and/or interviewed by the HFPS (i.e. general adult population). In all HFPS rounds that inform our analysis, the majority of respondents were household heads, ranging from 74 percent in Uganda to 83 percent in Ethiopia – with Malawi and Nigeria standing at 79 and 82 percent, respectively.9 The share of household heads among the HFPS respondents is similar across the four countries. This is notable because Ethiopia, Nigeria, and Uganda implicitly or explicitly targeted the household head as the HFPS respondent, whereas in Malawi, the order of the contacted phone numbers was randomized for 9 The sample underlying the estimates that are presented in Table 2 exclude individuals that were HFPS respondents but that were not household members at the time of the pre-COVID19 F2F survey. In Ethiopia, 98.4 percent of successfully interviewed households in the first HFPS round had a respondent that was also present in the associated F2F survey. This rate was 98.3 percent in Malawi, 97.9 percent in Nigeria, and 93.9 percent in Uganda. 8 each household and yet the respondent was still similarly likely to be the head of the household. One reason for this is that household heads are likely to own phones and as a result more likely to be called. Another reason is that arguably individual phone owners other than the household head handed phones to the household head to respond on behalf of the household. The remaining HFPS respondents were predominantly spouses of the household heads in each country, and the majority of respondents were men, ranging from 73 percent in Nigeria to just slightly above the population average in Uganda at 52 percent – with Ethiopia and Malawi standing at 62 and 63 percent, respectively. The HFPS respondents were also much less likely to be among the youth (i.e. between the ages of 15 and 24 years) vis-à-vis the general adult population. The gap was most pronounced in Uganda where 6 percent of respondents versus 38 percent of adults fall in the 15-24 age range and was least pronounced in Ethiopia where 13 percent of respondents versus 34 percent of adults fall in the same age range. For Nigeria and Malawi, the comparable statistics were 6 versus 32 percent, and 12 versus 40 percent, respectively. This finding is somewhat contrary to previous studies, which often found youth to be overrepresented among phone survey respondents (see the summary in Henderson and Rosenbaum 2020). Table 2. Unweighted Descriptive Statistics for HFPS Respondents and Adult Population in F2F Survey Ethiopia Malawi Nigeria Uganda Phone Adult Phone Adult Phone Adult Phone Adult resp. pop. resp. pop. resp. pop. resp. pop. Women 37.6 52.7 36.9 52.4 27.2 51.7 48.3 51.8 Gender Men 62.4 47.3 63.1 47.6 72.8 48.3 51.7 48.2 15-24 12.9 34.3 11.8 39.6 5.7 31.6 5.9 37.7 Age Group 25-49 66.6 49.6 65.5 44.5 55.0 45.0 59.8 40.8 50+ 20.5 16.1 22.6 15.9 39.3 23.4 34.3 21.5 Head 82.8 38.5 78.7 37.0 82.7 32.7 74.1 35.1 Relationship Spouse 9.8 24.8 16.5 26.1 9.2 28.1 20.2 22.0 to HH Head Child 6.0 26.3 3.1 24.6 6.5 30.3 4.4 32.1 Other 1.5 10.3 1.8 12.3 1.7 9.0 1.4 10.8 Observations 3,196 17,563 1701 8,588 1910 15,230 2128 8,763 Note: Table 2 presents unweighted results. Phone resp. = phone survey respondents; Adult pop. = General adult population 2.2. Household and individual sampling weights There are several sampling weights that are used in our analysis. To start with, there are the pre- COVID-19 F2F household survey sampling weights (wb ). These sampling weights serve as the starting point for the computation of the HFPS household sampling weights in public use data sets (w1 ), which are the calibrated versions of wb to address coverage and non-response biases at the household-level, leveraging the rich, pre-COVID-19 F2F survey data on (i) households that do not own a mobile phone and are excluded from the sampling frame; (ii) households that participate in the HFPS, and (iii) households that are contacted but cannot not be reached. This latter scenario is 9 overwhelmingly due to non-working phone numbers or prospective respondents not answering calls as opposed to answering the phone call but then refusing to respond to the survey. The household-level bias adjustment to create w1 follows the methodology proposed by Himelein (2014) 10 and is further detailed by Josephson et al. (2020) for the HFPS rounds in Ethiopia, Malawi, Nigeria, and Uganda. The HFPS household sampling weights are also post-stratified to match the projected population totals at the highest spatial resolution possible, ranging from region to district, based on the data availability in each country. Finally, w1 does not account for the non-random selection of an individual to be an HFPS respondent. To address this and allow for the analysis of individual-level phone survey data in a way that is more representative of the general adult population, an additional individual-level sampling weight is needed. The objective of this paper is to assess the effectiveness of this recalibrated weight to correct for selection biases at the individual level. To create the individual-level weight (w2 ), we follow an adjustment procedure that is similar to the procedure used to create w1. First, using the sample of all adult members of HFPS households (respondents and non-respondents), we estimate an unweighted logit regression to model the individual-level probability of selection as an HFPS respondent: (1) ( = 1) = (0 + � ) =1 The dependent variable in this model is a binary variable indicating whether a given individual was the round 1 HFPS respondent. X is a vector containing K independent variables that originate from the F2F survey and that are expected to predict the likelihood of being an HFPS respondent.11 A cross-country consistent set of independent variables is used for equation 1, including an extensive range of individual and household attributes and spatial fixed effects. Equation 1 is then estimated separately for each country. Since the individual’s relationship to the household head is likely to impact respondent-ship due to the HFPS respondent selection protocols, dichotomous variables are included to identify household head, spouse of the household, and child/adopted child of the household head, with the omitted category being any other relationship to the household head. Additional dichotomous variables are included to identify (i) men; (ii) married individuals; (iii) those aged 25-49 and, separately, 50+, with individuals in the age range of 15-24 constituting the omitted category; (iv) individuals with completed primary education, completed secondary education, completed post-secondary certificate/training, and completed post-secondary degree, with individuals having less than completed primary education being the omitted category; and (v) individuals that can read and write in any language. Since individuals with different time use may 10 This methodology is also commonly used for the computation of sampling weights in longitudinal F2F surveys with tracking of individuals over time, as in the case of the longitudinal surveys that have been supported under the World Bank LSMS-ISA initiative. 11 The sample for equation 1 does not include “new” HFPS household members that were not part of the F2F survey. 10 have different incentives and availability to respond to a phone survey, a set of non-exclusive dichotomous variables are included to discern whether the individual had regular wage employment; was the owner of a household enterprise, and participated in casual labor (with the latter being restricted only to Ethiopia and Malawi, in view of data availability and importance of casual labor activities in these contexts). Finally, a dichotomous variable is included to identify an individual’s ownership of a mobile phone, 12 which is expected to increase likelihood of being an HFPS respondent. The household-level attributes in equation 1 include (i) household size, which is expected to decrease the probability of any single adult being an HFPS respondent; (ii) dichotomous variables identifying the household’s total annual per capita household consumption expenditure quintile, with the lowest quintile being the excluded category. The significance level and size of the regression coefficients (β ) associated with the binary independent variables can be interpreted as the change in likelihood of being a phone survey respondent as a result of having the respective individual characteristic. Following the estimation of equation 1, we predict the probability of being an HFPS respondent across the entire sample of adult household members in successfully interviewed HFPS households; create deciles for this variable; compute the average predicted probability within each decile; and take the reciprocal of this average to define the adjustment factor for each decile (=): 1 (2) = = ∑ � =1 where N is the number of individuals in each decile. The adjustment factor is then applied to w1, the HFPS household sampling weight in the public use phone survey data set: (3) , = = ∗ 1 , is in turn winsorized at the top and bottom 2 percent, and the winsorized weight is post- stratified to match the projected population totals at the highest spatial resolution available, following the approach to the post-stratification of w1 . In each country, the post-stratification adjustment () is produced at the level of the lowest administrative unit for which population projections are available (typically region or district, depending on the country). It is computed as (i) the weighted total number of households residing in each administrative unit of interest, as measured by the sum of winsorized , values in that unit, divided by (ii) the household population projection in that unit. 12 Mobile phone ownership is identified by specific questions within each survey. In Ethiopia and Malawi, the data comes from individual level roster questions on mobile phone ownership. The Nigeria data comes from an individual level roster question on whether the individual has access to a mobile phone. In Uganda, mobile ownership is covered in the asset roster and allows the respondent to identify up to two household members that own working mobile phones. 11 Once computed, for each administrative unit is associated with all surveyed household in that unit, and , is multiplied with to derive the final individual weight, w2: (4) 2 = ∗ , 2.3. Assessing differences between HFPS respondents and general adult population under different sampling weights using the F2F survey data To assess the effectiveness of the bias reduction techniques for the individual-level phone survey data analysis, we focus on the individual-level variables that are captured in the pre-COVID-19 F2F survey and that are related to gender, age, marital status, relationship with the household head, education, and employment (see Appendix Table A1). We derive estimates for these variables using two different samples: (1) all adult household members, as captured in the F2F survey, who are assumed to be representative of the general adult population with the use of F2F household sampling weights (wb ), and (2) HFPS respondents whom were also present in the F2F survey (i.e. 2 is a subsample of 1). The weighted estimates for the adult household members in the F2F survey, denoted as b , serve as the benchmark to which we compare the sample of HFPS respondents under three different scenarios: 1. unweighted (w0 ), 2. weighted by the HFPS household sampling weights in the public use data sets (w1 ), and, 3. weighted by our newly generated HFPS individual sampling weight (w2 ), which is the recalibrated version of w1, intended to account for the non-random selection to be an HFPS respondent among the adult household members residing in the successfully interviewed HFPS households. We use two different approaches to assess the effectiveness of HFPS household and individual sampling weights in reducing the bias in estimates for the HFPS respondents vis-à-vis the general adult population (as captured through the F2F survey). First, we take a graphical approach, where the estimates from the F2F and phone surveys are standardized by subtracting the F2F survey mean. This means that the F2F survey mean is always zero and all other estimates are standardized in relation to the F2F survey mean, allowing a comparison across the competing estimates. The graphs then present the weighted mean and 95 percent confidence interval estimated for a range of individual-level variables for the general adult population (b ), and the same set of statistics estimated for the HFPS respondents, without the use of any sampling weight (w0 ) and under the HFPS household (w1 ) and individual (w2 ) sampling weight. This allows us to assess how large the differences in the two populations are at the outset (b vs w0 ) and how well the HFPS household sampling weights (b vs w1 ), and the HFPS individual weights (b vs w2 ) perform in reducing the differences between the HFPS respondents and the general adult population. 12 Second, we rely on Wald tests to assess whether the HFPS-based estimates obtained under different weights are significantly different vis-à-vis the F2F survey-based estimates for the general adult population. This approach requires constructing an appended data set that includes: i. all adult household members in the F2F survey households and the F2F survey household sampling weight (wb ), ii. the HFPS respondents and the HFPS household sampling weight set to 1 (w0 ), iii. the HFPS respondents and the HFPS household sampling weight in the public use data sets (w1 ), and iv. the HFPS respondents with the HFPS individual sampling weight (w2 ). In this setup, the samples (ii) through (iv) are composed of identical individuals that are appended with different sampling weights and that constitute a subset of sample (i). A common name is used for the sampling weight variable across the appended data sets and each appended data set includes the same set of individual-level variables, as listed in the Appendix Table A1. Furthermore, a new categorical variable is defined to uniquely identify each appended sample (i through iv). A weighted linear regression is then estimated for each outcome of interest, with an identical set of independent variables that include the dichotomous variables identifying the samples (ii) through (iv), with the sample (i) (i.e., all adult household members in the F2F survey households) serving as the comparison category. The sampling weight value for each observation is either wb , w0 , w1 or w2 in accordance with the appended sample that the record belongs to. When presenting the results from this regression, the base category is shown on the top row and represents the mean from which all other estimates deviate. The values in rows other than the base category express the difference in mean from the base category. 2.4. Assessing differences between HFPS respondents and general adult population under different sampling weights using the phone survey data In the previous section, we presented the approach to (i) comparing pre-COVID-19 attributes of respondents to those of the general adult population as captured in the pre-COVID-19 F2F surveys, and (ii) understanding how well individual-level weight adjustments can reduce these differences. In this section, we lay out the approach to expanding the comparative assessment of phone survey respondents vis-à-vis the general adult population, this time as captured in the HFPS data. Specifically, we leverage the fifth HFPS round in Malawi and Nigeria, where individual-level data on labor market outcomes were collected for all adult household members through the main HFPS respondent. We weight these individual-level HFPS data on select employment outcomes using 13 the HFPS household sampling weights (w1 ) 13 and assume these to be the alternative benchmark estimates for the general adult population.14 The estimates for the same set of employment outcomes are derived for the main HFPS respondents using the HFPS household (w1 ) and individual (w2 ) sampling weight. The approach to gauging graphical and statistically significant mean differences between the three competing estimates for each employment outcome is identical to the approach that is detailed in section 2.3. While the HFPS data on all adult household members, beyond the HFPS respondents, is crucial to understanding the labor market impacts of COVID-19, collecting individual-level data through a proxy is considered second-best to self-reporting. This is because proxy response may lead to non- sampling error (See e.g. references in Kilic et al., 2020), which cannot be corrected through reweighting (Himelein et al., 2020). Ideally, an individual-level analysis would therefore rely on self-reported data only, provided that the direct respondents are, or can be made, representative of the general population – which is what we test in this part of the analysis. 3. Results In the following, we first discuss how phone survey respondents differ from the general adult population and then explore how well the different weight adjustment techniques perform in allowing the data on HFPS respondents to be more representative of the general adult population. 3.1. Phone survey respondents versus the general adult population Given the respondent selection protocols discussed above, it is expected that the two populations – phone survey respondents and the general adult population – differ along various dimensions. As a reminder, Appendix Table A1 shows a set of descriptive statistics for the individual-level variables of interest for both populations in each of the four countries. Table 3 presents the results (i.e. marginal effects) from the estimation of equation 1, i.e. the logit regression, that models the likelihood of being an HFPS respondent among adults in successfully interviewed HFPS households, as a function of a rich set of individual and household attributes. Several overarching results emerge. First, household heads are most likely to be respondents. In all surveys, being the household head has the largest effect on the conditional probability of being the phone survey respondent, increasing that probability by between 31.4 percent in Nigeria and 45.7 percent in Ethiopia (with 13 Since individual-level reweighting is possible only for respondents for whom there is baseline face-to-face survey data, we drop any respondent who was not present in the face-to-face survey. 14 The HFPS household sampling weights (w1) are readily calibrated to provide representative estimates for the general household population, as explained in section 2.2 and demonstrated by Ambel et al. (forthcoming). As such, the assumption is that the HFPS individual-level data on adult household members that are weighted by w1 are representative of the general adult population. 14 Malawi- and Uganda-specific impacts being estimated at 39.7 percent and 38.9 percent, respectively). Note that this result already accounts for phone ownership, which is one of the control variables. Being the spouse of the household head also has a large effect in all countries but Nigeria, ranging between 12.8 percent in Ethiopia and 18.3 percent in Uganda. These results are likely driven by the country-specific respondent selection protocols, which tend to favor the household head or their spouse, as discussed in section 2.1. Conditional on household headship and remaining control variables, men are less likely to be HFPS respondents in Malawi and Uganda, and just as likely as women in being HFPS respondents in Ethiopia and Nigeria. However, men make up the majority of respondents in all four countries (Table 2). This finding is due to household heads being men much more often than women combined with the strong effect headship has on being the respondent. The household head effect masks the gender dynamics of phone survey response. Second, ownership of a mobile phone increases the probability of being the respondent substantially, ranging from 7.7 percent in Nigeria to 15.4 percent in Malawi (with Ethiopia- and Uganda-specific impacts being estimated at 11.4 percent and 13.9 percent, respectively). This is not surprising in a phone survey context, though the effect is not as strong as the effect of household headship. It suggests that phones are handed over from one household member to another to complete the interview. Third, HFPS respondents are more educated than non- respondents in all countries except for Malawi. In Ethiopia and Nigeria, holding any of primary, secondary, post-secondary certificate, or post-secondary degrees increases probability of being an HFPS respondent vis-à-vis adults with no degree. In Uganda, there are effects specifically associated with having primary education and with having a post-secondary certificate. The effect sizes range from two to eight percent. Fourth, being in an age category older than 15-24 increases probability of being a phone survey respondent in all countries but Ethiopia, where individuals aged 50+ are not any more likely to be selected as HFPS respondents vis-à-vis individuals aged 15-24. The age effects are particularly pronounced in Uganda, where individuals aged 25-49 and those aged 50+ are 11.2 percent and 9.4 percent, respectively, more likely to be HFPS respondents vis-à-vis individuals aged 15-24. Fifth, owning a household enterprise increases the probability of being an HFPS respondent in all countries, with the effect sizes ranging from 2.6 to 5.7 percent. The data on participation in casual labor is only available for Malawi and Ethiopia, and the results show that it increases likelihood of being an HFPS respondent by 7.5 percent in Ethiopia and 5.5 percent in Malawi. Given the high prevalence of casual labor in Malawi, 15 this is a relatively strong effect in that country. Finally, greater household wealth (proxied by household consumption quintiles) leads to a decline in the probability of being an HFPS respondent. However, the second quintile is no different than the first. Differences arise in the third quintile in Nigeria and Uganda, the fourth quintile in Ethiopia, and in the top quintile in Malawi. This suggests that wealthier households are overall less likely to respond to the phone survey, possibly due to higher opportunity cost of their time. 38.6 percent of adults were estimated to be participating in casual labor according to the IHPS 2019 (i.e. the pre- 15 COVID-19 F2F survey that served as a sampling frame for the Malawi HFPS). 15 Table 3. Marginal Effects from Logit Regressions on Being an HFPS Respondent in Round 1 Ethiopia Malawi Nigeria Uganda Household Size -0.015 *** -0.013 *** -0.011 *** -0.012 *** (.002) (.002) (.001) (.002) Head † 0.457 *** 0.397 *** 0.314 *** 0.389 *** (.018) (.026) (.019) (.027) Spouse of head † 0.128 *** 0.140 *** -0.010 0.183 *** (.023) (.033) (.023) (.032) Child of head † 0.083 *** -0.006 0.026 0.000 (.019) (.026) (.021) (.027) Male † -0.005 -0.040 *** 0.013 -0.050 *** (.009) (.015) (.013) (.012) Ages 25-49 † 0.031 *** 0.040 ** 0.079 *** 0.112 *** (.011) (.016) (.016) (.019) Ages 50+ † -0.009 0.038 ** 0.060 *** 0.094 *** (.014) (.019) (.018) (.020) Married † -0.016 -0.021 0.033 ** -0.065 *** (.012) (.019) (.014) (.017) Primary † 0.030 *** 0.005 0.021 * 0.029 *** (.010) (.015) (.012) (.010) Secondary † 0.043 *** 0.014 0.031 *** 0.050 (.013) (.017) (.012) (.033) Certificate † 0.079 ** -0.002 0.057 *** 0.037 * (.037) (.016) (.017) (.022) Post-Secondary Degree † 0.063 *** 0.002 0.036 * -0.003 (.016) (.023) (.019) (.020) Employed for a wage/salary † -0.007 -0.005 0.039 *** 0.007 (.010) (.015) (.013) (.012) Owner of a household enterprise † 0.026 ** 0.047 *** 0.057 *** 0.029 *** (.011) (.012) (.009) (.010) Casual laborer † 0.075 *** 0.055 *** (.020) (.012) Consumption quintile 2 † -0.011 -0.007 -0.022 -0.024 (.017) (.021) (.015) (.015) Consumption quintile 3 † -0.018 -0.017 -0.034 ** -0.031 ** (.016) (.021) (.015) (.015) Consumption quintile 4 † -0.031 * -0.027 -0.042 ** -0.048 *** (.016) (.020) (.016) (.015) Consumption quintile 5 † -0.043 *** -0.048 ** -0.041 ** -0.055 *** (.017) (.021) (.017) (.017) Individual owns a mobile phone † 0.114 *** 0.154 *** 0.077 *** 0.139 *** (.009) (.012) (.014) (.010) Spatial Fixed Effects Region x Urban District State Subregion Number of Observations 8535 4959 6183 6647 Pseudo R-squared 0.456 0.437 0.484 0.386 Note: † denotes dichotomous variables. Standard errors are reported in parentheses. ***/**/* denote statistical significance at the 1/5/10 percent level, respectively. For each country the sample is all F2F survey household members age 15 and older for the set of households that were successfully interviewed in round 1 of the phone survey. 16 3.2. Assessing bias reduction through weight adjustments We now turn to assessing how well the various survey weights perform at counteracting the bias associated with phone survey respondent selection. The results of the graphical analysis are shown in Figures 1 to 4. The effectiveness of the bias reduction is mixed and depends on the outcome of interest. Compared to the estimates obtained under the HFPS household sampling weights, the estimates based on the HFPS individual weights move closer to those for the general adult population for most individual-level outcomes of interest. However, confidence intervals widen as well. Several points stand out. First, there are instances where the HFPS household weight (w1 ) increases the difference between the unweighted respondent data and the wb -weighted F2F survey sample. Notably, the incidence of headship moves further from the mean in all four countries, though the difference is easier to detect in Nigeria and Uganda. The incidence of being a spouse also shows this pattern across all countries but Uganda. Beyond headship, Ethiopia exhibits this pattern for the estimates for the dichotomous variables identifying men and women, those in youngest and oldest age categories and married individuals. Malawi shows the same pattern for the estimates for the dichotomous variables identifying individuals in the youngest and oldest age groups. Nigeria shows the same pattern for the estimates for dichotomous variables identifying men and women, and individuals that own a household enterprise. Finally, Uganda demonstrates this pattern for the estimates for dichotomous variables identifying individuals in the age group 25-49, those without an educational degree, individuals that are engaged in wage employment, those that own a household enterprise, and individuals that own a mobile phone. This broad set of instances provides evidence that the HFPS household weights (w1 ) do not adequately support the analysis of individual-level data on HFPS respondents in a way that is representative of the general adult population. Second, the over-representation of household heads and mobile phone owners among phone survey respondents cannot be corrected by the HFPS household weights (w1 ) but is addressed more effectively by individual weights (w2 ). However, the individual weights do not eliminate the difference from baseline adults and cause the confidence intervals to widen. Lastly, there are some cases of over-correction where the individual weights move the mean estimates for the HFPS respondents beyond those that are associated with the benchmark sample of adult household members in F2F surveys. This is true particularly regarding the estimates for being the spouse of the household head in Malawi and being a woman in Uganda. These are different sorts of failures, since they lead to biases that are not reflected in the unweighted data. Tables 4 and 5 present the results from the weighted linear regressions that are detailed in section 2.3 and that reveal whether the differences are statistically significant between the weighted estimates for the general adult population, as captured in the pre-COVID-19 F2F survey, and the unweighted and various weighted estimates for the HFPS respondents. The results show that the differences between the HFPS respondents and the general adult population are not fully addressed by HFPS individual weights (w2 ). However, there are a few cases where individual weights do 17 succeed in addressing the bias. In Malawi, the individual weights can deal with over-representation of age group 50+ and under-representation of females. In all countries except for Ethiopia, under- representation of respondents without an educational degree is also mitigated. The over-arching result remains that the individual weights applied to the data on the HFPS respondents move the estimates in the right direction, but they do not successfully eliminate bias. These results hold if the sample is broken down by gender and different age groups.16 16 Gender- and age-disaggregated results are presented in Appendix Table A2. 18 Figure 1. Graphical Inspection of Bias Adjustment, Ethiopia. Figure 2. Graphical Inspection of Bias Adjustment, Malawi. 19 Figure 3. Graphical Inspection of Bias Adjustment, Nigeria. Figure 4. Graphical Inspection of Bias Adjustment, Uganda. 20 Table 4. Tests of Mean Differences Between Face-To-Face Adults and Phone Respondents: Sex, Age, Relation to Head – As Measured in the F2F Survey Comparison Group Ethiopia Malawi Nigeria Uganda Variable Sample Weight Abbrev. Beta p-value Beta p-value Beta p-value Beta p-value Base, All F2F Adults F2F HH Weight b 0.518 0.513 0.515 0.518 Phone respondents Unweighted w0 -0.142 (.000) *** -0.143 (.000) *** -0.243 (.000) *** -0.035 (.003) *** Female Phone respondents HFPS HH Weight w1 -0.242 (.000) *** -0.118 (.000) *** -0.263 (.000) *** -0.036 (.020) ** Phone respondents HFPS Individual Weight w2 -0.146 (.000) *** 0.035 (.206) -0.068 (.004) *** 0.039 (.028) ** Base, All F2F Adults F2F HH Weight b 0.356 0.387 0.313 0.360 Phone respondents Unweighted w0 -0.227 (.000) *** -0.269 (.000) *** -0.256 (.000) *** -0.300 (.000) *** Ages 15-24 Phone respondents HFPS HH Weight w1 -0.238 (.000) *** -0.300 (.000) *** -0.255 (.000) *** -0.290 (.000) *** Phone respondents HFPS Individual Weight w2 -0.124 (.000) *** -0.193 (.000) *** -0.100 (.000) *** -0.205 (.000) *** Base, All F2F Adults F2F HH Weight b 0.478 0.427 0.469 0.450 Phone respondents Unweighted w0 0.188 (.000) *** 0.229 (.000) *** 0.082 (.000) *** 0.148 (.000) *** Ages 25-49 Phone respondents HFPS HH Weight w1 0.169 (.000) *** 0.208 (.000) *** 0.082 (.000) *** 0.183 (.000) *** Phone respondents HFPS Individual Weight w2 0.068 (.001) *** 0.176 (.000) *** -0.010 (.622) 0.152 (.000) *** Base, All F2F Adults F2F HH Weight b 0.166 0.186 0.218 0.190 Phone respondents Unweighted w0 0.039 (.000) *** 0.040 (.001) *** 0.175 (.000) *** 0.153 (.000) *** Ages 50+ Phone respondents HFPS HH Weight w1 0.069 (.000) *** 0.092 (.000) *** 0.173 (.000) *** 0.107 (.000) *** Phone respondents HFPS Individual Weight w2 0.056 (.001) *** 0.017 (.342) 0.111 (.000) *** 0.053 (.000) *** Base, All F2F Adults F2F HH Weight b 0.370 0.341 0.326 0.374 Phone respondents Unweighted w0 0.457 (.000) *** 0.446 (.000) *** 0.501 (.000) *** 0.366 (.000) *** Head Phone respondents HFPS HH Weight w1 0.486 (.000) *** 0.455 (.000) *** 0.507 (.000) *** 0.369 (.000) *** Phone respondents HFPS Individual Weight w2 0.209 (.000) *** 0.100 (.000) *** 0.120 (.000) *** 0.072 (.000) *** Base, All F2F Adults F2F HH Weight b 0.259 0.232 0.290 0.238 Phone respondents Unweighted w0 -0.161 (.000) *** -0.068 (.000) *** -0.199 (.000) *** -0.036 (.000) *** Spouse Phone respondents HFPS HH Weight w1 -0.192 (.000) *** -0.096 (.000) *** -0.211 (.000) *** -0.032 (.012) ** Phone respondents HFPS Individual Weight w2 -0.069 (.000) *** 0.095 (.001) *** -0.049 (.034) ** 0.110 (.000) *** Note: Base row reports the nationally representative mean among all adults in the face-to-face survey. Rows other than the base row report the difference from the base and the p-value from a test of significance for that difference. Sample: all adults in F2F surveys, of which phone survey respondents are a sub-sample. 21 Table 5. Tests of Mean Differences Between Face-To-Face Adults and Phone Respondents: Marital Status, Education, Employment – As Measured in the F2F Survey Comparison Group Ethiopia Malawi Nigeria Uganda Variable Sample Weight Abbrev. Beta p-value Beta p-value Beta p-value Beta p-value Base, All F2F Adults F2F HH Weight b 0.549 0.508 0.561 0.525 Phone respondents Unweighted w0 0.114 (.000) *** 0.250 (.000) *** 0.175 (.000) *** 0.204 (.000) *** Married Phone respondents HFPS HH Weight w1 0.176 (.000) *** 0.213 (.000) *** 0.175 (.000) *** 0.201 (.000) *** Phone respondents HFPS Individual Weight w2 0.116 (.000) *** 0.196 (.000) *** 0.050 (.034) ** 0.166 (.000) *** Base, All F2F Adults F2F HH Weight b 0.520 0.747 0.751 0.795 Phone respondents Unweighted w0 0.240 (.000) *** 0.128 (.000) *** 0.070 (.000) *** -0.020 (.046) ** Literate Phone respondents HFPS HH Weight w1 0.060 (.001) *** 0.026 (.206) 0.030 (.026) ** 0.007 (.488) Phone respondents HFPS Individual Weight w2 0.051 (.027) ** 0.019 (.433) 0.021 (.253) -0.018 (.203) Base, All F2F Adults F2F HH Weight b 0.768 0.676 0.355 0.465 Phone respondents Unweighted w0 -0.282 (.000) *** -0.191 (.000) *** -0.114 (.000) *** 0.011 (.366) No degree Phone respondents HFPS HH Weight w1 -0.045 (.003) *** -0.035 (.053) * -0.024 (.076) * -0.029 (.022) ** Phone respondents HFPS Individual Weight w2 -0.053 (.008) *** -0.040 (.124) -0.025 (.218) 0.010 (.544) Base, All F2F Adults F2F HH Weight b 0.090 0.089 0.100 0.219 Wage Phone respondents Unweighted w0 0.199 (.000) *** 0.161 (.000) *** 0.113 (.000) *** 0.017 (.056) * employment Phone respondents HFPS HH Weight w1 0.050 (.000) *** 0.089 (.000) *** 0.084 (.000) *** 0.076 (.000) *** Phone respondents HFPS Individual Weight w2 0.022 (.029) ** 0.039 (.002) *** 0.032 (.011) ** 0.035 (.008) *** Base, All F2F Adults F2F HH Weight b 0.098 0.160 0.281 0.185 Enterprise Phone respondents Unweighted w0 0.118 (.000) *** 0.181 (.000) *** 0.138 (.000) *** 0.134 (.000) *** owner Phone respondents HFPS HH Weight w1 0.078 (.000) *** 0.129 (.000) *** 0.179 (.000) *** 0.135 (.000) *** Phone respondents HFPS Individual Weight w2 0.037 (.004) *** 0.063 (.000) *** 0.070 (.001) *** 0.059 (.000) *** Base, All F2F Adults F2F HH Weight b 0.307 0.305 0.797 0.445 Phone respondents Unweighted w0 0.506 (.000) *** 0.518 (.000) *** 0.150 (.000) *** 0.317 (.000) *** Mobile owner Phone respondents HFPS HH Weight w1 0.351 (.000) *** 0.423 (.000) *** 0.133 (.000) *** 0.345 (.000) *** Phone respondents HFPS Individual Weight w2 0.227 (.000) *** 0.262 (.000) *** 0.060 (.001) *** 0.185 (.000) *** Note: Base row reports the nationally representative mean among all adults in the face-to-face survey. Rows other than the base row report the difference from the base and the p-value from a test of significance for that difference. Sample: all adults in F2F surveys, of which phone survey respondents are a sub-sample. 22 3.3. Individual-level employment outcomes measured in phone surveys We now turn to the analysis of individual-level employment outcomes during COVID-19, as measured in the fifth HFPS rounds in Malawi and Nigeria. There are three dichotomous outcomes of interest that pertain to the last 7 days and that identify whether: a. an individual worked to generate income for at least 1 hour, irrespective of type of employment (i.e. overall employment), b. an individual worked for a wage or salary (i.e. wage employment), and c. an individual worked at a household enterprise, as an owner, manager, or a contributing laborer (i.e. self-employment). The pool of HFPS respondents differ slightly in round 5 vis-à-vis round 1 due to attrition, as such we generate a round 5-specific HFPS individual weight, following the same steps outlined in section 2.2. Figure 5 shows the mean and confidence interval for each employment outcome of interest for: i. all adults that were interviewed in the F2F survey and that were residing in HFPS households that were successfully interviewed in round 5, weighted by the round 5 HFPS household sampling weight (w1 ) – assumed to be representative of the general adult population, ii. the main HFPS respondents interviewed in round 5, weighted by the round 5 HFPS household sampling weight (w1 ), and iii. the main HFPS respondents interviewed in round 5, weighted by the round 5 HFPS individual sampling weight (w2 ). The mean for (i), which is assumed to be the benchmark estimate in this portion of our analysis, is subtracted from all estimates, as such it is set at zero – following the approach in Figures 1 through 4. The HFPS individual weights again do succeed in moving the estimates for the HFPS respondents closer to those for the general adult population (except for the incidence of self- employment in Malawi), albeit with widened confidence intervals (Figure 5). When weighted with the HFPS household sampling weights (w1 ), the mean differences remain statistically significant between the estimates for the HFPS respondents and the estimates for all adults residing in HFPS households (Table 6). This result holds true irrespective of the country and employment outcome of interest. Once weighted with the HFPS individual weights (w2 ), the estimates wage employment and self-employment for the HFPS respondents in Nigeria are statistically indistinguishable from the benchmark estimates. However, the mean differences for the overall employment variable in Nigeria and for all three employment variables in Malawi remain between the w2 -weighted estimates for the HFPS respondents and the benchmark w1- weighted estimates for all adults residing in HFPS households. 23 Figure 5. HFPS Round 5 Employment Outcomes, with HFPS Household versus Individual Weights Table 6. Differences in HFPS Round 5 Employment Outcomes Between Phone Survey Respondents and Adult Population Comparison Group Malawi Nigeria Variable Sample Weight Abbrev. Beta p-value Beta p-value Adults (base) Final HH Weight w1 0.612 0.72 Any Employment Respondents Final HH Weight w1 0.204 (.000) *** 0.119 (.000) *** Respondents Individual Weight w2 0.146 (.000) *** 0.048 (.018) ** Adults (base) Final HH Weight w1 0.158 0.086 Wage Employment Respondents Final HH Weight w1 0.066 (.000) *** 0.026 (.003) *** Respondents Individual Weight w2 0.027 (.099) * 0.016 (.220) Adults (base) Final HH Weight w1 0.119 0.297 Self-Employment Respondents Final HH Weight w1 0.086 (.000) *** 0.024 (.050) * Respondents Individual Weight w2 0.094 (.000) *** -0.001 (.975) Note: Base row reports the nationally representative mean among all adults present in both the F2F and phone surveys. Rows other than the base row report the difference from the base in the sample of respondents present in both F2F and phone surveys, and a p-value from a test of significance for that difference. Employment = 1 if individual spent any time in the last seven days doing specified work, 0 otherwise. All employment data are from wave 5 post-COVID survey in Malawi or wave 5 post-COVID survey in Nigeria. The disaggregated employment results in the Appendix Table A3 are consistent with the findings presented in Table 6. In Nigeria, the individual weights remove the differences in estimates for the HFPS respondents versus the general adult population for wage employment and for self- employment, except among individuals aged 25-49 where a significant difference remains for wage employment. The differences in overall employment variable remain significant among the male sub-population, and individuals aged 25-49 in Nigeria, which are the largest sub-populations of HFPS respondents. In Malawi, the individual weighted estimates for the HFPS respondents are statistically indistinguishable from the benchmark estimates for wage employment among males and for overall and wage employment among individuals aged 25-49. The HFPS household weights also mitigate bias in some subpopulations, particularly in Nigeria, but there are no cases where the individual weights do not perform at least as well. Overall, these results suggest that while individual weights can be more effective than household weights in reducing the bias in the 24 analysis of individual-level data on the main HFPS respondents, they are still insufficient to eliminate the bias in full. 4. Conclusion Our analysis has shown that phone survey respondents in Ethiopia, Malawi, Nigeria, and Uganda are significantly different from the general adult population in a range of demographic, education, and labor market characteristics. On average, respondents are significantly more likely to be household heads or their spouses, and they tend to be older, more educated, and more likely to own a household enterprise. It is notable that these differences persist in Malawi, where the entries in the list of potential contact numbers for each household was randomized, although the HFPS did not follow any further respondent selection protocols once the contact was made with a household member. To address these selection biases, we have recalibrated the HFPS household sampling weights based on propensity score adjustments that are derived from a cross-country comparable model of an adult individual’s likelihood of being interviewed, as a function of both individual- and household-level attributes. The individual-level reweighting reduces the bias, consistently moving the estimates for the phone survey respondents closer to those for the general adult population for a range of variables. However, individual-level reweighting fails to fully overcome the biases in most cases, as the differences in means remain statistically significant for most of the outcomes of interest. Using individual-level phone survey data, we show that respondents’ labor market outcomes during the COVID-19 pandemic differ from the adult population living in phone survey households. Here, too, individual-level reweighting is a step in the right direction but is ultimately insufficient. At the same time, the data that are provided by the main phone survey respondent (i.e. the proxy) in each household on all other adult household members’ labor market activities can be subject to bias in and of itself. This is especially likely to be the case when it comes to questions about attitudes, expectations, and knowledge, which a proxy respondent should not be expected to answer on behalf of other individuals. Yet, these types of questions are of particular interest during the COVID-19 pandemic, as they can reveal the effectiveness of containment measures and vaccination policies, among others. The rapid design and successful implementation of high-frequency phone surveys during the COVID-19 pandemic has been an unprecedented learning experience on the part of national statistical offices in low- and middle-income countries and international agencies and donor organizations that have provided financial and technical support to these operations. Across Ethiopia, Malawi, Nigeria, and Uganda alone, a total of 33 national phone survey rounds have been implemented from April 2020 to April 2021, amounting to a total of over 68,000 interviews. The anonymized unit record data from each survey round have been disseminated through the 25 World Bank Microdata Library, and have fed into the World Bank COVID-19 High-Frequency Monitoring Dashboard; a total of 25 analytical survey reports, with a total download count of over 6,400 17; and several World Bank publications, including cross-country journal publications, working papers and policy briefs, with a total download count of over 6,500.18 In view of (a) the investments in phone survey data production during the COVID-19 pandemic, (b) the extensive remote technical assistance provision to improve statistical capacity in design and implementation of phone surveys in low- and middle-income countries, (c) the demonstrated utility of phone survey data to fill data and knowledge gaps, and (d) our findings regarding the limits of representativeness of the individual-level phone survey data in four African countries, survey implementers should think more critically about respondent selection protocols in future phone surveys. A desirable option is to randomly select an adult household member to be interviewed in each household on topics that are related to individuals/personal experiences. In the context of the on- going HFPS rounds and future phone surveys that use existing household surveys as sampling frames, the interview target can be selected at random (without replacement) in each household following a household roster update. Upon the selection of the interview target, the current phone survey respondent can be asked to either pass the phone to the selected individual, if he or she is available, or provide a phone number for the selected individual, if a person-specific phone number exists, or coordinate with the selected individual to converge on a date and time for an interview using the current respondent’s phone. The randomly selected household member can ultimately replace (depending on the objective of the study) or be in addition to the main phone survey respondent. Attempting to interview all adult members would be yet another option. There will obviously be additional financial requirements for two or more interviews per household and the scope for non-response will likely grow with the number of interview targets. Given limited prior experiences with such variations in respondent selection protocols, a sensible first step would be to pilot one or several of these improved options in a random subset of households in future phone surveys to better understand the subsequent impacts on consent, non-response and attrition. 17 The download count is as of April 16, 2021 and is sourced for each document from the Documents and Reports repository of the World Bank. The survey reports can be accessed via www.worldbank.org/lsms-covid19. 18 The download count is as of May 6, 2021. The count is, unless otherwise stated, sourced for each of the following documents from the Documents and Reports repository of the World Bank and is noted in parenthesis: Josephson et al., 2020 (1,137); Josephson et al., 2021 (3,303 – based on Access statistic from Nature Human Behavior) Furbush et al., 2021 (528), Amankwah and Gourlay, 2021a (910), Amankwah and Gourlay, 2021b (234), Khamis et al., 2021 (467). 26 References Alvi, M., Gupta, S., Meinzen-Dick, R., and Ringler, C. (2020). “Phone surveys to understand gendered impacts of COVID-19: a cautionary note.” IFPRI South Asia Office. Retrieved on February 25, 2021 from http://southasia.ifpri.info/2020/07/22/phone-surveys-to- understand-gendered-impacts-of-covid-19-a-cautionary-note/ Amankwah, A., Gourlay, S. (2021a). “Impact of COVID-19 crisis on agriculture: evidence from five Sub-Saharan African countries.” Washington, D.C.: World Bank. Amankwah, A., Gourlay, S. (2021b). “Food Security in the face of COVID-19: evidence from Africa.” Washington, D.C.: World Bank. Ambel, A., McGee, K., Tsegay, A. (Forthcoming). “Advantages of representative face-to-face surveys for adjusting phone survey biases.” Ballivian, A., Azevedo, J.P., Durbin, W., Rios, J., Godoy, J., and Borisova, C. (2015). “Using mobile phones for high-frequency data collection.” Mobile Research Methods, 21. Dabalen, A., Etang, A., Hoogeveen, J., Mushi, E., Schipper, Y., and von Engelhardt, J. (2016). Mobile phone panel surveys in developing countries: a practical guide for microdata collection. Washington, D.C.: World Bank. Etang, A., and Himelein, K. (2020). “Monitoring the Ebola crisis using mobile phone surveys.” In Data collection in fragile states: innovations from Africa and beyond , edited by J. Hoogeveen and U. Pape, pp. 15–31. Cham: Springer International Publishing. https://doi.org/10.1007/978-3-030-25120-8_2. Fu, H., and Sanchez-Paramo, C. (2020). “High-frequency monitoring of COVID-19 impacts.” World Bank Data Blog Post. Retrieved on February 25, 2021 from https://blogs.worldbank.org/opendata/high-frequency-monitoring-covid-19-impacts. Fu, H., and Schweinfest, S. (2020). “COVID-19 widens gulf of global data inequality, while national statistical offices step up to meet new data demands.” World Bank Data Blog Post. Retrieved on February 25, 2021 from https://blogs.worldbank.org/opendata/covid-19- widens-gulf-global-data-inequality-while-national-statistical-offices-step-up. Furbush, A., Josephson, A.L., Kilic, T., and Michler, J.D. (2021). “The Evolving socioeconomic impacts of COVID-19 in four African countries.” World Bank Policy Research Working Paper No. 9556. Gibson, D.G., Wosu, A.C., Pariyo, G.W., Ahmed, S., Ali, J., Labrique, A.B., Khan, I.A., Rutebemberwa, E., Flora, M.S., and Hyder, A. A. (2019). “Effect of airtime incentives on response and cooperation rates in non-communicable disease interactive voice response surveys: randomised controlled trials in Bangladesh and Uganda.” BMJ Global Health, 4, e001604. Glazerman, S., Rosenbaum, M., Sandino, R., and Shaughnessy, L. (2020). Remote surveying in a pandemic: handbook. Innovation for Poverty Action. Gourlay, S., Kilic, T., Zezza, A., Martuscelli, A., Wollburg, P., forthcoming. “High-frequency phone surveys on COVID-19: good practices, open questions.” Greenleaf, A.R., Gadiaga, A., Choi, Y., Guiella, G., Turke, S., Battle, N., Ahmed, S., and Moreau, C. (2020). “Automated and interviewer-administered mobile phone surveys in Burkina 27 Faso: sociodemographic differences among female mobile phone survey respondents and nonrespondents.” JMIR mHealth and uHealth, 8, e17891, https://doi.org/10.2196/17891. Henderson, S., and Rosenbaum, M. (2020). Remote surveying in a pandemic: research synthesis. Innovation for Poverty Action. Himelein K. (2014). “Weight calculations for panel surveys with subsampling and split‐off tracking.” Statistics and Public Policy, 1.1, pp. 40-45. Himelein, K. H., Eckman, S., Kastelic, J. G., McGee, K. R., Wild, M., Yoshida, N., and Hoogeveen, J. G. (2020). “High frequency mobile phone surveys of households to assess the impacts of COVID-19: guidelines on sampling design.” Washington, D.C.: World Bank. ILO (International Labour Organization) (2020). “ILO monitor: COVID-19 and the world of work. 6th edition.” Retrieved on February 25, 2021 from https://www.ilo.org/tokyo/information/publications/WCMS_755910/lang--en/index.htm. IPA (Innovations for Poverty Action) (2020). “IPA’s RECOVR Survey.” Retrieved on February 25, 2021 from http://www.poverty-action.org/recovr/recovr-survey. Josephson A., Kilic T., and Michler J. D. (2020). “Socioeconomic impacts of COVID-19 in four African countries.” World Bank Policy Research Working Paper No. 9466. Josephson, A., Kilic, T., and Michler, J.D. (2021). “Socioeconomic impacts of COVID-19 in low- income countries.” Nature Human Behaviour. https://doi.org/10.1038/s41562-021-01096- 7. Khamis, M., Prinz, D., Newhouse, D. L., Palacios-Lopez, A., Pape, U. J., and Weber, M. (2021). “The early labor market impacts of COVID-19 in developing countries: evidence from high-frequency phone surveys.” World Bank Policy Research Working Paper No. 9510. Kilic, T., Van den Broeck, G., Koolwal, G., and Moylan, H. (2020). “Are you being asked? Impacts of respondent selection on measuring employment.” World Bank Policy Research Working Paper No. 9152. Larmarange, J., Kassoum, O., Kakou, E., Fradier, Y., Sika, L., Danel, C., and Dutreuilh, C. (2016). “Feasibility and representativeness of a random sample mobile phone survey in Côte d’Ivoire.” Population, 71.1, 119–29. Lau, C., Cronberg, A., Marks, L., and Amaya, A. (2019a). “In search of the optimal mode for mobile phone surveys in developing countries. a comparison of IVR, SMS, and CATI in Nigeria.” Survey Research Methods, 13, pp. 305–318, https://doi.org/10.18148/srm/2019.v13i3.7375. Lau, C., and di Tada, N. (2018). “Identifying non-working phone numbers for response rate calculations in Africa. Surv Pract, 11. Lau, C., Sanders, H., and Lombaard, A. (2019b). “Questionnaire design in short message service (SMS) surveys.” Field Methods, 31, pp. 214–229, https://doi.org/10.1177/1525822X19862506. L’Engle, K., Sefa, E., Adimazoya, E.A., Yartey, E., Lenzi, R., Tarpo, C., Heward-Mills, N.L., Lew, K., and Ampeh, Y. (2018). “Survey research with a random digit dial national mobile phone sample in Ghana: methods and sample quality.” PLOS ONE, 13, e0190902, https://doi.org/10.1371/journal.pone.0190902. 28 Leo, B., Morello, R., Mellon, J., Peixoto, T., and Davenport, S.T. (2015). “Do mobile phone surveys work in poor countries?” Center for Global Development Working Paper No. 398. McKenzie, D. (2012). “Help for attrition is just a phone call away – a new bounding approach to help deal with non-response.” World Bank Development Impact Blog. Retrieved on February 25, 2021 from https://blogs.worldbank.org/impactevaluations/help-for-attrition- is-just-a-phone-call-away-a-new-bounding-approach-to-help-deal-with-non-response. Okoro, S., and Pretittore, P. (2020). “Will COVID-19 widen the gender justice gap?” World Bank Voices Blog Post. Retrieved on February 25, 2021 from https://blogs.worldbank.org/voices/will-covid-19-widen-gender-justice-gap . Özler, B., and Cuevas, P.F. (2019). “Reducing attrition in phone surveys.” World Bank Development Impact Blog Post. Retrieved on February 25, 2021 from: https://blogs.worldbank.org/impactevaluations/reducing-attrition-phone-surveys. Palacios-Lopez, A., Noumedem Temgoua, C., Kokas, D., Talbert, E.A., Siwatu, G.O., Ponzini, G., Ilukor, J., Phadera, L., Yoshida, N., and Wollburg, P.R. (2020). “High frequency mobile phone surveys of households to assess the impacts of COVID-19: overview of the questionnaire template.” Washington DC: World Bank. Quisumbing, A. R., Kumar, N., Meinzen-Dick, R.S., and Ringler, C. (2020). “Why gender matters in COVID-19 responses: now and in the future.” Washington, DC: International Food Policy Research Institute. https://doi.org/10.2499/p15738coll2.133762_20. Roy Mathur, M., (2020). “Reflections on reaching women over the phone in rural India.” IDInsight Blog Post. Retrieved on February 25, 2021 from https://medium.com/idinsight- blog/reflections-on-reaching-women-over-the-phone-in-rural-india-29fffdbe296a. Tillé, Y. (2006). Sampling algorithms. Springer: New York, NY. United Nations and World Bank (2020). “Monitoring the state of statistical operations under the COVID-19 pandemic.” The World Bank, Washington D.C. UN Women (2020). “Unlocking the lockdown. The gendered effects of COVID-19 on achieving the SDGs in Asia and the Pacific.” Retrieved on May 7, 2021 from https://data.unwomen.org/sites/default/files/documents/COVID19/Unlocking_the_lockdo wn_UNWomen_2020.pdf. Velthausz, D., Donco, R., Skelly, H., and Eichleay, M. (2016). “Mozambique mobile access and usage study: computer-assisted telephone interview (CATI) survey results.” United States Agency for International Development, Retrieved on February 25, 2021 from http://fsdmoc.com/wp-content/uploads/2017/08/Mozambique-Mobile-Access-and-Usage- Study.pdf. 29 Appendix Table A1. Descriptive Statistics Ethiopia Malawi Phone Respondents Phone Respondents All F2F Adults (N=17563) All F2F Adults (N=8582) (N=3196) (N=1700) Mean SE Mean SE Mean SE Mean SE Individual-Level Attributes Respondent † 0.18 0.003 1.00 0.000 0.20 0.004 1.00 0.000 Head † 0.39 0.004 0.83 0.007 0.37 0.005 0.79 0.010 Spouse of head † 0.25 0.003 0.10 0.005 0.26 0.005 0.16 0.009 Child of head † 0.26 0.003 0.06 0.004 0.25 0.005 0.03 0.004 Male † 0.47 0.004 0.62 0.009 0.48 0.005 0.63 0.012 15-24 † 0.34 0.004 0.13 0.006 0.40 0.005 0.12 0.008 25-49 † 0.50 0.004 0.67 0.008 0.45 0.005 0.66 0.012 50+ † 0.16 0.003 0.21 0.007 0.16 0.004 0.23 0.010 Married † 0.54 0.004 0.66 0.008 0.56 0.005 0.76 0.010 Literate † 0.61 0.004 0.76 0.008 0.78 0.004 0.87 0.008 No Degree † 0.65 0.004 0.49 0.009 0.63 0.005 0.48 0.012 Primary Education † 0.20 0.003 0.22 0.007 0.14 0.004 0.13 0.008 Secondary Education † 0.08 0.002 0.14 0.006 0.10 0.003 0.18 0.009 Certificate † 0.01 0.001 0.02 0.002 0.09 0.003 0.13 0.008 Post-Secondary Education † 0.06 0.002 0.14 0.006 0.04 0.002 0.08 0.006 Sick in last 2-4 weeks † 0.14 0.003 0.17 0.007 0.22 0.005 0.24 0.010 Chronically ill/Disabled † 0.11 0.002 0.12 0.006 0.09 0.003 0.11 0.008 Employed for a wage/salary † 0.14 0.003 0.29 0.008 0.11 0.003 0.25 0.011 Owner of a household enterprise† 0.11 0.002 0.22 0.007 0.18 0.004 0.34 0.011 Casual laborer † 0.03 0.001 0.06 0.004 0.39 0.005 0.35 0.012 Individual owns a mobile phone † 0.42 0.004 0.81 0.007 0.36 0.005 0.82 0.009 Household-Level Attributes Household Size 5.00 0.018 4.05 0.038 5.38 0.027 4.74 0.055 Consumption Quintile 1 (lowest) † 0.13 0.003 0.05 0.004 0.17 0.004 0.09 0.007 Consumption Quintile 2 † 0.15 0.003 0.10 0.005 0.17 0.004 0.14 0.009 Consumption Quintile 3 † 0.16 0.003 0.13 0.006 0.17 0.004 0.16 0.009 Consumption Quintile 4 † 0.21 0.003 0.22 0.007 0.22 0.004 0.23 0.010 Consumption Quintile 5 (highest) † 0.34 0.004 0.51 0.009 0.27 0.005 0.37 0.012 Notes: No weights are used. † denotes a dichotomous variable Respondent identifies whether the individual was an HFPS respondent – set to 1 for all individuals under the Phone Respondents column. 30 Table A1. Descriptive Statistics (Continued) Nigeria Uganda Phone Respondents Phone Respondents All F2F Adults (N=15230) All F2F Adults (N=8763) (N=1910) (N=2128) Mean SE Mean SE Mean SE Mean SE Individual-Level Attributes Respondent † 0.13 0.003 1.00 0.000 0.24 0.005 1.00 0.000 Head † 0.33 0.004 0.83 0.009 0.35 0.005 0.74 0.010 Spouse of head † 0.28 0.004 0.09 0.007 0.22 0.004 0.20 0.009 Child of head † 0.30 0.004 0.06 0.006 0.32 0.005 0.04 0.004 Male † 0.48 0.004 0.73 0.010 0.48 0.005 0.52 0.011 15-24 † 0.32 0.004 0.06 0.005 0.38 0.005 0.06 0.005 25-49 † 0.45 0.004 0.55 0.011 0.41 0.005 0.60 0.011 50+ † 0.23 0.003 0.39 0.011 0.22 0.004 0.34 0.010 Married † 0.55 0.004 0.74 0.010 0.50 0.005 0.73 0.010 Literate † 0.74 0.004 0.82 0.009 0.75 0.005 0.78 0.009 No Degree † 0.34 0.004 0.24 0.010 0.52 0.005 0.48 0.011 Primary Education † 0.18 0.003 0.21 0.009 0.36 0.005 0.37 0.010 Secondary Education † 0.36 0.004 0.33 0.011 0.02 0.001 0.02 0.003 Certificate † 0.07 0.002 0.11 0.007 0.04 0.002 0.05 0.005 Post-Secondary Education † 0.06 0.002 0.11 0.007 0.06 0.003 0.08 0.006 Sick in last 2-4 weeks † 0.27 0.004 0.34 0.011 0.28 0.005 0.38 0.011 Chronically ill/Disabled † 0.11 0.003 0.13 0.008 0.03 0.002 0.02 0.003 Employed for a wage/salary † 0.10 0.002 0.21 0.009 0.18 0.004 0.24 0.009 Owner of a household enterprise† 0.26 0.004 0.42 0.011 0.17 0.004 0.32 0.010 Casual laborer † Individual owns a mobile phone † 0.80 0.003 0.95 0.005 0.39 0.005 0.76 0.009 Household-Level Attributes Household Size 6.80 0.031 5.54 0.077 6.29 0.032 5.41 0.059 Consumption Quintile 1 (lowest) † 0.16 0.003 0.10 0.007 0.20 0.004 0.16 0.008 Consumption Quintile 2 † 0.17 0.003 0.13 0.008 0.20 0.004 0.19 0.008 Consumption Quintile 3 † 0.20 0.003 0.18 0.009 0.20 0.004 0.21 0.009 Consumption Quintile 4 † 0.21 0.003 0.22 0.009 0.21 0.004 0.22 0.009 Consumption Quintile 5 (highest) † 0.26 0.004 0.38 0.011 0.19 0.004 0.23 0.009 Notes: No weights are used. † denotes a dichotomous variable. Respondent identifies whether the individual was an HFPS respondent – set to 1 for all individuals under the Phone Respondents column. 31 Table A2. Ethiopia: Tests of difference between face-to-face adults and phone respondents, disaggregated by sex and age group. Comparison Group National Males Females Ages 15-24 Ages 25-49 Ages 50+ Variable Sample Weight Beta p-value Beta p-value Beta p-value Beta p-value Beta p-value Beta p-value Base, All F2F Adults F2F HH Weight 0.518 0.522 0.529 0.477 Female Phone respondents HFPS HH Weight -0.242 (.000) *** -0.116 (.012) ** -0.278 (.000) *** -0.198 (.000) *** Phone respondents HFPS Individual Weight -0.146 (.000) *** -0.096 (.042) ** -0.150 (.000) *** -0.182 (.000) *** Base, All F2F Adults F2F HH Weight 0.356 0.353 0.359 Ages 15-24 Phone respondents HFPS HH Weight -0.238 (.000) *** -0.256 (.000) *** -0.185 (.000) *** Phone respondents HFPS Individual Weight -0.124 (.000) *** -0.141 (.000) *** -0.093 (.003) *** Base, All F2F Adults F2F HH Weight 0.478 0.467 0.488 Ages 25-49 Phone respondents HFPS HH Weight 0.169 (.000) *** 0.203 (.000) *** 0.101 (.000) *** Phone respondents HFPS Individual Weight 0.068 (.001) *** 0.073 (.002) *** 0.070 (.038) ** Base, All F2F Adults F2F HH Weight 0.166 0.180 0.153 Ages 50+ Phone respondents HFPS HH Weight 0.069 (.000) *** 0.053 (.001) *** 0.084 (.001) *** Phone respondents HFPS Individual Weight 0.056 (.001) *** 0.069 (.001) *** 0.023 (.331) Base, All F2F Adults F2F HH Weight 0.370 0.566 0.188 0.074 0.475 0.703 Head Phone respondents HFPS HH Weight 0.486 (.000) *** 0.353 (.000) *** 0.502 (.000) *** 0.391 (.000) *** 0.411 (.000) *** 0.268 (.000) *** Phone respondents HFPS Individual Weight 0.209 (.000) *** 0.170 (.000) *** 0.127 (.000) *** 0.061 (.003) *** 0.170 (.000) *** 0.181 (.000) *** Base, All F2F Adults F2F HH Weight 0.259 0.024 0.477 0.118 0.387 0.190 Spouse Phone respondents HFPS HH Weight -0.192 (.000) *** -0.018 (.000) *** -0.254 (.000) *** -0.019 (.436) -0.311 (.000) *** -0.168 (.000) *** Phone respondents HFPS Individual Weight -0.069 (.000) *** -0.001 (.881) -0.006 (.860) 0.037 (.299) -0.147 (.000) *** -0.092 (.002) *** Base, All F2F Adults F2F HH Weight 0.549 0.554 0.545 0.171 0.786 0.680 Married Phone respondents HFPS HH Weight 0.176 (.000) *** 0.297 (.000) *** -0.152 (.000) *** 0.110 (.003) *** 0.020 (.189) 0.046 (.100) * Phone respondents HFPS Individual Weight 0.116 (.000) *** 0.172 (.000) *** 0.016 (.610) 0.052 (.155) 0.013 (.478) 0.116 (.000) *** Base, All F2F Adults F2F HH Weight 0.520 0.623 0.424 0.709 0.480 0.231 Literate Phone respondents HFPS HH Weight 0.060 (.001) *** 0.002 (.935) 0.038 (.175) 0.158 (.000) *** 0.143 (.000) *** 0.089 (.002) *** Phone respondents HFPS Individual Weight 0.051 (.027) ** 0.001 (.960) 0.057 (.115) 0.139 (.001) *** 0.092 (.000) *** 0.049 (.116) Base, All F2F Adults F2F HH Weight 0.768 0.732 0.802 0.689 0.768 0.937 No degree Phone respondents HFPS HH Weight -0.045 (.003) *** -0.016 (.381) -0.061 (.004) *** -0.235 (.000) *** -0.061 (.000) *** -0.034 (.005) *** Phone respondents HFPS Individual Weight -0.053 (.008) *** -0.025 (.296) -0.071 (.014) ** -0.201 (.000) *** -0.044 (.046) ** -0.005 (.676) Base, All F2F Adults F2F HH Weight 0.090 0.124 0.059 0.060 0.124 0.059 Wage Phone respondents HFPS HH Weight 0.050 (.000) *** 0.024 (.032) ** 0.059 (.000) *** 0.085 (.000) *** 0.041 (.001) *** 0.011 (.394) employment Phone respondents HFPS Individual Weight 0.022 (.029) ** -0.002 (.848) 0.037 (.022) ** 0.054 (.029) ** 0.013 (.314) -0.010 (.401) Base, All F2F Adults F2F HH Weight 0.098 0.103 0.092 0.049 0.144 0.067 Enterprise Phone respondents HFPS HH Weight 0.078 (.000) *** 0.056 (.000) *** 0.127 (.000) *** 0.147 (.000) *** 0.064 (.000) *** 0.011 (.430) owner Phone respondents HFPS Individual Weight 0.037 (.004) *** 0.017 (.181) 0.067 (.004) *** 0.061 (.025) ** 0.029 (.059) * -0.004 (.785) Base, All F2F Adults F2F HH Weight 0.307 0.402 0.218 0.229 0.396 0.217 Mobile Phone respondents HFPS HH Weight 0.351 (.000) *** 0.313 (.000) *** 0.289 (.000) *** 0.417 (.000) *** 0.329 (.000) *** 0.261 (.000) *** owner Phone respondents HFPS Individual Weight 0.227 (.000) *** 0.213 (.000) *** 0.177 (.000) *** 0.310 (.000) *** 0.207 (.000) *** 0.143 (.000) *** Notes: Base row reports the nationally representative mean among all adults in the face-to-face survey. Rows other than the base row report the difference from the base and a p-value from a test of significance for that difference. 32 Table A2. (Continued) Malawi: Tests of difference between face-to-face adults and phone respondents, disaggregated by sex and age group. Comparison Group National Males Females Ages 15-24 Ages 25-49 Ages 50+ Variable Sample Weight Beta p-value Beta p-value Beta p-value Beta p-value Beta p-value Beta p-valu Base, All F2F Adults F2F HH Weight 0.513 0.522 0.529 0.477 Female Phone respondents HFPS HH Weight -0.118 (.000) *** -0.116 (.012) ** -0.278 (.000) *** -0.198 (.000) Phone respondents HFPS Individual Weight 0.035 (.206) -0.096 (.042) ** -0.150 (.000) *** -0.182 (.000) Base, All F2F Adults F2F HH Weight 0.387 0.353 0.359 Ages 15-24 Phone respondents HFPS HH Weight -0.300 (.000) *** -0.256 (.000) *** -0.185 (.000) *** Phone respondents HFPS Individual Weight -0.193 (.000) *** -0.141 (.000) *** -0.093 (.003) *** Base, All F2F Adults F2F HH Weight 0.427 0.467 0.488 Ages 25-49 Phone respondents HFPS HH Weight 0.208 (.000) *** 0.203 (.000) *** 0.101 (.000) *** Phone respondents HFPS Individual Weight 0.176 (.000) *** 0.073 (.002) *** 0.070 (.038) ** Base, All F2F Adults F2F HH Weight 0.186 0.180 0.153 Ages 50+ Phone respondents HFPS HH Weight 0.092 (.000) *** 0.053 (.001) *** 0.084 (.001) *** Phone respondents HFPS Individual Weight 0.017 (.342) 0.069 (.001) *** 0.023 (.331) Base, All F2F Adults F2F HH Weight 0.341 0.566 0.188 0.074 0.475 0.703 Head Phone respondents HFPS HH Weight 0.455 (.000) *** 0.353 (.000) *** 0.502 (.000) *** 0.391 (.000) *** 0.411 (.000) *** 0.268 (.000) Phone respondents HFPS Individual Weight 0.100 (.000) *** 0.170 (.000) *** 0.127 (.000) *** 0.061 (.003) *** 0.170 (.000) *** 0.181 (.000) Base, All F2F Adults F2F HH Weight 0.232 0.024 0.477 0.118 0.387 0.190 Spouse Phone respondents HFPS HH Weight -0.096 (.000) *** -0.018 (.000) *** -0.254 (.000) *** -0.019 (.436) -0.311 (.000) *** -0.168 (.000) Phone respondents HFPS Individual Weight 0.095 (.001) *** -0.001 (.881) -0.006 (.860) 0.037 (.299) -0.147 (.000) *** -0.092 (.002) Base, All F2F Adults F2F HH Weight 0.508 0.554 0.545 0.171 0.786 0.680 Married Phone respondents HFPS HH Weight 0.213 (.000) *** 0.297 (.000) *** -0.152 (.000) *** 0.110 (.003) *** 0.020 (.189) 0.046 (.100) Phone respondents HFPS Individual Weight 0.196 (.000) *** 0.172 (.000) *** 0.016 (.610) 0.052 (.155) 0.013 (.478) 0.116 (.000) Base, All F2F Adults F2F HH Weight 0.747 0.623 0.424 0.709 0.480 0.231 Literate Phone respondents HFPS HH Weight 0.026 (.206) 0.002 (.935) 0.038 (.175) 0.158 (.000) *** 0.143 (.000) *** 0.089 (.002) Phone respondents HFPS Individual Weight 0.019 (.433) 0.001 (.960) 0.057 (.115) 0.139 (.001) *** 0.092 (.000) *** 0.049 (.116) Base, All F2F Adults F2F HH Weight 0.676 0.732 0.802 0.689 0.768 0.937 No degree Phone respondents HFPS HH Weight -0.035 (.053) * -0.016 (.381) -0.061 (.004) *** -0.235 (.000) *** -0.061 (.000) *** -0.034 (.005) Phone respondents HFPS Individual Weight -0.040 (.124) -0.025 (.296) -0.071 (.014) ** -0.201 (.000) *** -0.044 (.046) ** -0.005 (.676) Base, All F2F Adults F2F HH Weight 0.089 0.124 0.059 0.060 0.124 0.059 Wage Phone respondents HFPS HH Weight 0.089 (.000) *** 0.024 (.032) ** 0.059 (.000) *** 0.085 (.000) *** 0.041 (.001) *** 0.011 (.394) employment Phone respondents HFPS Individual Weight 0.039 (.002) *** -0.002 (.848) 0.037 (.022) ** 0.054 (.029) ** 0.013 (.314) -0.010 (.401) Base, All F2F Adults F2F HH Weight 0.160 0.103 0.092 0.049 0.144 0.067 Enterprise Phone respondents HFPS HH Weight 0.129 (.000) *** 0.056 (.000) *** 0.127 (.000) *** 0.147 (.000) *** 0.064 (.000) *** 0.011 (.430) owner Phone respondents HFPS Individual Weight 0.063 (.000) *** 0.017 (.181) 0.067 (.004) *** 0.061 (.025) ** 0.029 (.059) * -0.004 (.785) Base, All F2F Adults F2F HH Weight 0.305 0.402 0.218 0.229 0.396 0.217 Mobile Phone respondents HFPS HH Weight 0.423 (.000) *** 0.313 (.000) *** 0.289 (.000) *** 0.417 (.000) *** 0.329 (.000) *** 0.261 (.000) owner Phone respondents HFPS Individual Weight 0.262 (.000) *** 0.213 (.000) *** 0.177 (.000) *** 0.310 (.000) *** 0.207 (.000) *** 0.143 (.000) Notes: Base row reports the nationally representative mean among all adults in the face-to-face survey. Rows other than the base row report the difference from the base and a p-value from a test of significance for that difference. 33 Table A2 (Continued) Nigeria: Tests of difference between face-to-face adults and phone respondents, by sex and age group. Comparison Group National Males Females Ages 15-24 Ages 25-49 Ages 50+ Variable Sample Weight Beta p-value Beta p-value Beta p-value Beta p-value Beta p-value Beta p-value Base, All F2F Adults F2F HH Weight 0.515 0.522 0.529 0.477 Female Phone respondents HFPS HH Weight -0.263 (.000) *** -0.116 (.012) ** -0.278 (.000) *** -0.198 (.000) *** Phone respondents HFPS Individual Weight -0.068 (.004) *** -0.096 (.042) ** -0.150 (.000) *** -0.182 (.000) *** Base, All F2F Adults F2F HH Weight 0.313 0.353 0.359 Ages 15-24 Phone respondents HFPS HH Weight -0.255 (.000) *** -0.256 (.000) *** -0.185 (.000) *** Phone respondents HFPS Individual Weight -0.100 (.000) *** -0.141 (.000) *** -0.093 (.003) *** Base, All F2F Adults F2F HH Weight 0.469 0.467 0.488 Ages 25-49 Phone respondents HFPS HH Weight 0.082 (.000) *** 0.203 (.000) *** 0.101 (.000) *** Phone respondents HFPS Individual Weight -0.010 (.622) 0.073 (.002) *** 0.070 (.038) ** Base, All F2F Adults F2F HH Weight 0.218 0.180 0.153 Ages 50+ Phone respondents HFPS HH Weight 0.173 (.000) *** 0.053 (.001) *** 0.084 (.001) *** Phone respondents HFPS Individual Weight 0.111 (.000) *** 0.069 (.001) *** 0.023 (.331) Base, All F2F Adults F2F HH Weight 0.326 0.566 0.188 0.074 0.475 0.703 Head Phone respondents HFPS HH Weight 0.507 (.000) *** 0.353 (.000) *** 0.502 (.000) *** 0.391 (.000) *** 0.411 (.000) *** 0.268 (.000) *** Phone respondents HFPS Individual Weight 0.120 (.000) *** 0.170 (.000) *** 0.127 (.000) *** 0.061 (.003) *** 0.170 (.000) *** 0.181 (.000) *** Base, All F2F Adults F2F HH Weight 0.290 0.024 0.477 0.118 0.387 0.190 Spouse Phone respondents HFPS HH Weight -0.211 (.000) *** -0.018 (.000) *** -0.254 (.000) *** -0.019 (.436) -0.311 (.000) *** -0.168 (.000) *** Phone respondents HFPS Individual Weight -0.049 (.034) ** -0.001 (.881) -0.006 (.860) 0.037 (.299) -0.147 (.000) *** -0.092 (.002) *** Base, All F2F Adults F2F HH Weight 0.561 0.554 0.545 0.171 0.786 0.680 Married Phone respondents HFPS HH Weight 0.175 (.000) *** 0.297 (.000) *** -0.152 (.000) *** 0.110 (.003) *** 0.020 (.189) 0.046 (.100) * Phone respondents HFPS Individual Weight 0.050 (.034) ** 0.172 (.000) *** 0.016 (.610) 0.052 (.155) 0.013 (.478) 0.116 (.000) *** Base, All F2F Adults F2F HH Weight 0.751 0.623 0.424 0.709 0.480 0.231 Literate Phone respondents HFPS HH Weight 0.030 (.026) ** 0.002 (.935) 0.038 (.175) 0.158 (.000) *** 0.143 (.000) *** 0.089 (.002) *** Phone respondents HFPS Individual Weight 0.021 (.253) 0.001 (.960) 0.057 (.115) 0.139 (.001) *** 0.092 (.000) *** 0.049 (.116) Base, All F2F Adults F2F HH Weight 0.355 0.732 0.802 0.689 0.768 0.937 No degree Phone respondents HFPS HH Weight -0.024 (.076) * -0.016 (.381) -0.061 (.004) *** -0.235 (.000) *** -0.061 (.000) *** -0.034 (.005) *** Phone respondents HFPS Individual Weight -0.025 (.218) -0.025 (.296) -0.071 (.014) ** -0.201 (.000) *** -0.044 (.046) ** -0.005 (.676) Base, All F2F Adults F2F HH Weight 0.100 0.124 0.059 0.060 0.124 0.059 Wage Phone respondents HFPS HH Weight 0.084 (.000) *** 0.024 (.032) ** 0.059 (.000) *** 0.085 (.000) *** 0.041 (.001) *** 0.011 (.394) employment Phone respondents HFPS Individual Weight 0.032 (.011) ** -0.002 (.848) 0.037 (.022) ** 0.054 (.029) ** 0.013 (.314) -0.010 (.401) Base, All F2F Adults F2F HH Weight 0.281 0.103 0.092 0.049 0.144 0.067 Enterprise Phone respondents HFPS HH Weight 0.179 (.000) *** 0.056 (.000) *** 0.127 (.000) *** 0.147 (.000) *** 0.064 (.000) *** 0.011 (.430) owner Phone respondents HFPS Individual Weight 0.070 (.001) *** 0.017 (.181) 0.067 (.004) *** 0.061 (.025) ** 0.029 (.059) * -0.004 (.785) Base, All F2F Adults F2F HH Weight 0.797 0.402 0.218 0.229 0.396 0.217 Mobile Phone respondents HFPS HH Weight 0.133 (.000) *** 0.313 (.000) *** 0.289 (.000) *** 0.417 (.000) *** 0.329 (.000) *** 0.261 (.000) *** owner Phone respondents HFPS Individual Weight 0.060 (.001) *** 0.213 (.000) *** 0.177 (.000) *** 0.310 (.000) *** 0.207 (.000) *** 0.143 (.000) *** Note: Base row reports the nationally representative mean among all adults in the face-to-face survey. Rows other than the base row report the difference from the base and a p-value from a test of significance for that difference. 34 Table A2. (Continued) Uganda: Tests of difference between face-to-face adults and phone respondents, by sex and age group. Comparison Group National Males Females Ages 15-24 Ages 25-49 Ages 50+ Variable Sample Weight Beta p-value Beta p-value Beta p-value Beta p-value Beta p-value Beta p-value Base, All F2F Adults F2F HH Weight 0.518 0.522 0.529 0.477 Female Phone respondents HFPS HH Weight -0.036 (.020) ** -0.116 (.012) ** -0.278 (.000) *** -0.198 (.000) *** Phone respondents HFPS Individual Weight 0.039 (.028) ** -0.096 (.042) ** -0.150 (.000) *** -0.182 (.000) *** Base, All F2F Adults F2F HH Weight 0.360 0.353 0.359 Ages 15-24 Phone respondents HFPS HH Weight -0.290 (.000) *** -0.256 (.000) *** -0.185 (.000) *** Phone respondents HFPS Individual Weight -0.205 (.000) *** -0.141 (.000) *** -0.093 (.003) *** Base, All F2F Adults F2F HH Weight 0.450 0.467 0.488 Ages 25-49 Phone respondents HFPS HH Weight 0.183 (.000) *** 0.203 (.000) *** 0.101 (.000) *** Phone respondents HFPS Individual Weight 0.152 (.000) *** 0.073 (.002) *** 0.070 (.038) ** Base, All F2F Adults F2F HH Weight 0.190 0.180 0.153 Ages 50+ Phone respondents HFPS HH Weight 0.107 (.000) *** 0.053 (.001) *** 0.084 (.001) *** Phone respondents HFPS Individual Weight 0.053 (.000) *** 0.069 (.001) *** 0.023 (.331) Base, All F2F Adults F2F HH Weight 0.374 0.566 0.188 0.074 0.475 0.703 Head Phone respondents HFPS HH Weight 0.369 (.000) *** 0.353 (.000) *** 0.502 (.000) *** 0.391 (.000) *** 0.411 (.000) *** 0.268 (.000) *** Phone respondents HFPS Individual Weight 0.072 (.000) *** 0.170 (.000) *** 0.127 (.000) *** 0.061 (.003) *** 0.170 (.000) *** 0.181 (.000) *** Base, All F2F Adults F2F HH Weight 0.238 0.024 0.477 0.118 0.387 0.190 Spouse Phone respondents HFPS HH Weight -0.032 (.012) ** -0.018 (.000) *** -0.254 (.000) *** -0.019 (.436) -0.311 (.000) *** -0.168 (.000) *** Phone respondents HFPS Individual Weight 0.110 (.000) *** -0.001 (.881) -0.006 (.860) 0.037 (.299) -0.147 (.000) *** -0.092 (.002) *** Base, All F2F Adults F2F HH Weight 0.525 0.554 0.545 0.171 0.786 0.680 Married Phone respondents HFPS HH Weight 0.201 (.000) *** 0.297 (.000) *** -0.152 (.000) *** 0.110 (.003) *** 0.020 (.189) 0.046 (.100) * Phone respondents HFPS Individual Weight 0.166 (.000) *** 0.172 (.000) *** 0.016 (.610) 0.052 (.155) 0.013 (.478) 0.116 (.000) *** Base, All F2F Adults F2F HH Weight 0.795 0.623 0.424 0.709 0.480 0.231 Literate Phone respondents HFPS HH Weight 0.007 (.488) 0.002 (.935) 0.038 (.175) 0.158 (.000) *** 0.143 (.000) *** 0.089 (.002) *** Phone respondents HFPS Individual Weight -0.018 (.203) 0.001 (.960) 0.057 (.115) 0.139 (.001) *** 0.092 (.000) *** 0.049 (.116) Base, All F2F Adults F2F HH Weight 0.465 0.732 0.802 0.689 0.768 0.937 No degree Phone respondents HFPS HH Weight -0.029 (.022) ** -0.016 (.381) -0.061 (.004) *** -0.235 (.000) *** -0.061 (.000) *** -0.034 (.005) *** Phone respondents HFPS Individual Weight 0.010 (.544) -0.025 (.296) -0.071 (.014) ** -0.201 (.000) *** -0.044 (.046) ** -0.005 (.676) Base, All F2F Adults F2F HH Weight 0.219 0.124 0.059 0.060 0.124 0.059 Wage Phone respondents HFPS HH Weight 0.076 (.000) *** 0.024 (.032) ** 0.059 (.000) *** 0.085 (.000) *** 0.041 (.001) *** 0.011 (.394) employment Phone respondents HFPS Individual Weight 0.035 (.008) *** -0.002 (.848) 0.037 (.022) ** 0.054 (.029) ** 0.013 (.314) -0.010 (.401) Base, All F2F Adults F2F HH Weight 0.185 0.103 0.092 0.049 0.144 0.067 Enterprise Phone respondents HFPS HH Weight 0.135 (.000) *** 0.056 (.000) *** 0.127 (.000) *** 0.147 (.000) *** 0.064 (.000) *** 0.011 (.430) owner Phone respondents HFPS Individual Weight 0.059 (.000) *** 0.017 (.181) 0.067 (.004) *** 0.061 (.025) ** 0.029 (.059) * -0.004 (.785) Base, All F2F Adults F2F HH Weight 0.445 0.402 0.218 0.229 0.396 0.217 Mobile Phone respondents HFPS HH Weight 0.345 (.000) *** 0.313 (.000) *** 0.289 (.000) *** 0.417 (.000) *** 0.329 (.000) *** 0.261 (.000) *** owner Phone respondents HFPS Individual Weight 0.185 (.000) *** 0.213 (.000) *** 0.177 (.000) *** 0.310 (.000) *** 0.207 (.000) *** 0.143 (.000) *** Notes: Base row reports the nationally representative mean among all adults in the face-to-face survey. Rows other than the base row report the difference from the base and a p-value from a test of significance for that difference. 35 Table A3. Difference between adults and phone respondent employment outcomes, by sex. Malawi Nigeria Variable Sample Weight Abbrev. Beta p-value Beta p-value Adult Females (base) HFPS HH Weight w1 0.556 0.655 Any Employment Female Respondents HFPS HH Weight w1 0.151 (.000) *** 0.07 (.007) *** Female Respondents HFPS Individual Weight w2 0.108 (.003) *** 0.035 (.290) Adult Females (base) HFPS HH Weight w1 0.095 0.045 Wage Employment Female Respondents HFPS HH Weight w1 0.074 (.000) *** 0.03 (.061) * Female Respondents HFPS Individual Weight w2 0.039 (.044) ** 0.024 (.202) Adult Females (base) HFPS HH Weight w1 0.107 0.328 Self-Employment Female Respondents HFPS HH Weight w1 0.071 (.000) *** 0.007 (.805) Female Respondents HFPS Individual Weight w2 0.076 (.004) *** -0.021 (.509) Adult Males (base) HFPS HH Weight w1 0.667 0.788 Any Employment Male Respondents HFPS HH Weight w1 0.22 (.000) *** 0.093 (.000) *** Male Respondents HFPS Individual Weight w2 0.201 (.000) *** 0.052 (.018) ** Adult Males (base) HFPS HH Weight w1 0.219 0.128 Wage Employment Male Respondents HFPS HH Weight w1 0.039 (.022) ** -0.004 (.719) Male Respondents HFPS Individual Weight w2 0.024 (.345) 0.004 (.812) Adult Males (base) HFPS HH Weight w1 0.13 0.264 Self-Employment Male Respondents HFPS HH Weight w1 0.091 (.000) *** 0.051 (.000) *** Male Respondents HFPS Individual Weight w2 0.117 (.000) *** 0.022 (.320) Notes: Base row reports the HFPS-based nationally representative mean among all adults present in the face-to-face and phone surveys. Rows other than the base row report the difference from the base and a p-value from a test of significance for that difference. Employment = 1 if individual spent any time in the last seven days doing specified work, 0 otherwise. All data are from the fifth round of the HFPS in Malawi and Nigeria. 36 Table A3. (Continued) Difference between adults and phone respondent employment outcomes, by age group. Malawi Nigeria Variable Sample Weight Abbrev. Beta p-value Beta p-value Adults Ages 15-24 (base) HFPS HH Weight w1 0.376 0.597 Any Employment Respondents Ages 15-24 HFPS HH Weight w1 0.387 (.000) *** 0.149 (.002) *** Respondents Ages 15-24 HFPS Individual Weight w2 0.352 (.000) *** 0.06 (.292) Adults Ages 15-24 (base) HFPS HH Weight w1 0.094 0.097 Wage Employment Respondents Ages 15-24 HFPS HH Weight w1 0.141 (.001) *** 0.031 (.485) Respondents Ages 15-24 HFPS Individual Weight w2 0.08 (.096) * -0.002 (.956) Adults Ages 15-24 (base) HFPS HH Weight w1 0.024 0.167 Self-Employment Respondents Ages 15-24 HFPS HH Weight w1 0.166 (.001) *** 0.106 (.070) * Respondents Ages 15-24 HFPS Individual Weight w2 0.197 (.001) *** 0.061 (.222) Adults Ages 25-49 (base) HFPS HH Weight w1 0.751 0.772 Any Employment Respondents Ages 25-49 HFPS HH Weight w1 0.074 (.000) *** 0.096 (.000) *** Respondents Ages 25-49 HFPS Individual Weight w2 0.005 (.838) 0.063 (.011) ** Adults Ages 25-49 (base) HFPS HH Weight w1 0.212 0.086 Wage Employment Respondents Ages 25-49 HFPS HH Weight w1 0.039 (.004) *** 0.048 (.000) *** Respondents Ages 25-49 HFPS Individual Weight w2 0.006 (.738) 0.034 (.055) * Adults Ages 25-49 (base) HFPS HH Weight w1 0.188 0.387 Self-Employment Respondents Ages 25-49 HFPS HH Weight w1 0.046 (.000) *** -0.014 (.441) Respondents Ages 25-49 HFPS Individual Weight w2 0.053 (.026) ** -0.019 (.470) Adults Ages 50+ (base) HFPS HH Weight w1 0.75 0.772 Any Employment Respondents Ages 50+ HFPS HH Weight w1 0.057 (.009) *** 0.038 (.010) *** Respondents Ages 50+ HFPS Individual Weight w2 0.035 (.398) -0.024 (.537) Adults Ages 50+ (base) HFPS HH Weight w1 0.141 0.067 Wage Employment Respondents Ages 50+ HFPS HH Weight w1 0.007 (.679) 0.005 (.490) Respondents Ages 50+ HFPS Individual Weight w2 -0.03 (.073) * 0.009 (.662) Adults Ages 50+ (base) HFPS HH Weight w1 0.131 0.26 Self-Employment Respondents Ages 50+ HFPS HH Weight w1 -0.001 (.933) -0.013 (.418) Respondents Ages 50+ HFPS Individual Weight w2 0.007 (.830) -0.026 (.350) Notes: Base row reports the HFPS-based nationally representative mean among all adults present in the face-to-face and phone surveys. Rows other than the base row report the difference from the base and a p-value from a test of significance for that difference. Employment = 1 if individual spent any time in the last seven days doing specified work, 0 otherwise. All data are from the fifth round of the HFPS in Malawi and Nigeria. 37