Policy Research Working Paper 9676 Reducing Bias in Phone Survey Samples Effectiveness of Reweighting Techniques Using Face-to-Face Surveys as Frames in Four African Countries Alemayehu Ambel Kevin McGee Asmelash Tsegay Development Economics Development Research Group May 2021 Policy Research Working Paper 9676 Abstract Several developing countries are currently implementing standards. Left unaddressed, this bias would result in biased phone surveys in response to immediate data needs to mon- estimates from the interviewed sample that do not fully itor the socioeconomic impact of COVID-19. However, reflect the situation of poorer households in the country. phone surveys are often subject to coverage and non-re- However, phone survey biases can be substantially reduced sponse bias that can compromise the representativeness by applying survey weight adjustments using information of the sample and the external validity of the estimates from the representative survey from which the sample is obtained from the survey. Using data from high-frequency drawn. Applying these methods to the four surveys resulted phone surveys in Ethiopia, Malawi, Nigeria, and Uganda, in a substantial reduction in bias, although the bias was this study investigates the magnitude and source of biases not fully eradicated. This highlights one of the potential present in these four surveys and explores the effectiveness advantages of drawing phone survey samples from exist- of techniques applied to reduce bias. Varying levels of cov- ing face-to-face, representative surveys over random digit erage and non-response bias are found in all four countries. dialing or using lists from telecom providers where such The successfully contacted samples in these four countries adjustment methods can be more limited. were biased toward wealthier households with higher living This paper is a product of the Development Research Group, Development Economics. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at kmcgee@worldbank.org. The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Produced by the Research Support Team Reducing Bias in Phone Survey Samples Effectiveness of Reweighting Techniques Using Face-to-Face Surveys as Frames in Four African Countries 1 Alemayehu Ambel2, Kevin McGee 3, Asmelash Tsegay 4 JEL Codes: C81, C83. Keywords: COVID-19, survey methodology, survey sampling, nonresponse bias, coverage bias, sampling representativeness, weighting methods, Ethiopia, Malawi, Nigeria, Uganda, Sub- Saharan Africa. 1 We would like to thank Kristen Himelein, Gero Carletto, Diego Zardetto, Talip Kilic, Philip Wollburg, Josh Brubaker, and Isabela Coelho for valuable comments on earlier drafts. Any error or omission is our responsibility alone. 2 Development Data Group, World Bank, Washington, DC, aambel@worldbank.org 3 Development Data Group, World Bank, Washington, DC, kmcgee@worldbank.org 4 Development Data Group, World Bank, Washington, DC, atsegay@worldbank.org 1. Introduction The outbreak of the COVID-19 virus has had far-ranging impacts across the globe. In addition to the serious health effects, COVID-19 has caused widespread economic disruption. The epidemic has resulted in closure of schools and markets, disruption in public services, banning of social gatherings and restriction of mobility and transportation of goods and people. Given these wide-ranging impacts of the pandemic, governments, policy makers and researchers are in dire need of just-in-time and reliable data both to monitor the health and socio-economic impact of the epidemic on livelihoods and well-being. However, amid a pandemic it is difficult to deploy enumerators to the field to collect information from households and communities through face-to-face (F2F) interviews given the risk of transmission to both interviewer and respondent. As a solution to this challenge and to address the urgent demand for rapid and informative data, high frequency phone surveys (HFPS) of households are being implemented in many developing countries. The improvement of phone penetration in these counties in recent years, as well as the existence of recently conducted representative household surveys that contain re-contact information for some or all household members, has created opportunities to conduct phone surveys. While phone surveys have distinct advantages that are particularly relevant during the pandemic, they also suffer from drawbacks. One of the most substantial weaknesses of phone surveys is that they are often subject to several sources of bias that can compromise the representativeness of the sample and the external validity of the estimates obtained from the survey (Himelein et al, 2020). This study investigates the magnitude and source of bias in recent telephone surveys as well as exploring the effectiveness of techniques to reduce it. Using telephone surveys in Ethiopia, Malawi, Nigeria and Uganda that were sampled from recently completed representative F2F surveys, this study will first explore the sources of bias introduced at each stage of the phone survey selection and interview process, by comparing the profile of characteristics of the F2F and the phone survey samples. After examining the source and extent of bias, it will then assess the effectiveness of reweighting adjustment methods to correct for these biases. These adjustment methods are especially powerful when using samples drawn from existing representative face-to-face surveys, like the Living Standards Measurement Study (LSMS) surveys, which contain a wealth of detailed information on all sampled units. While similar adjustments can be performed under other sampling approaches, such as sampling from a list of numbers from a telecom provider or random digit dialing, the scope of the adjustments is more limited in those approaches since they often do not have detailed information on sampled units that is available in a representative F2F survey. This could present one advantage of sampling from representative F2F surveys over other sampling approaches. The remainder of this paper will proceed as follows. In Section 2, the benefits and limitations of phone surveys will be briefly reviewed including the different sources of bias and methods available to reduce the bias. In Section 3, data and methods will be discussed. Section 4 presents the results and Section 5 concludes. 2. Phone Surveys Even though F2F surveys have been the main mode of data collection in developing countries for many years, there is also a growing list of successfully implemented phone surveys (see Tomlinson, 2 M., Solomon, W., Singh, Y. et al, 2009, Ballivian et al 2015, Dabalen et al, 2016, Lau et al, 2019, (Fu et al, 2015, Himelein and Kastelic, 2015 and Leo et al, 2015). Phone surveys have an array of advantages over typical F2F surveys, primary among them is cost. F2F surveys require interviewers to travel to the household’s or individual’s location to conduct the interview which introduces substantial travel and logistical costs, particularly for national surveys. On the other hand, phone surveys require no travel and interviewers can conduct all their calls from one location. In the era of the coronavirus pandemic, another advantage of phone surveys has been realized in that it eliminates the need for contact with others (either through travel or conducting the F2F interview) and thereby has minimal risk of COVID-19 exposure for interviewers and respondents alike. This advantage contrasted with the COVID-19 risks associated with F2F data collection has led to a proliferation of phone surveys in the developing world during the coronavirus pandemic. However, phone surveys also suffer from some disadvantages when compared to F2F surveys. One of the main disadvantages (and the focus of this study) is that phone surveys often suffer from coverage and nonresponse bias that erode the representativeness of the interviewed sample. Coverage bias results from the typical exclusion of the segment of the population that does not have access to a telephone (either mobile or landline). Although mobile phone penetration is increasing in developing countries, in many countries there is still a sizable share of the population that lacks access and would not be represented in typical phone survey samples. There are likely substantial differences between this uncovered portion of the sample and the covered portion that will introduce bias in the results obtained. Phone surveys also commonly suffer from nonresponse bias whereby the pool of sampled households or individuals who are not successfully interviewed are substantially different from those interviewed. Nonresponse is a common feature of any survey including F2F surveys, but the potential for nonresponse is typically substantially higher for phone surveys. One reason nonresponse is often higher for phone surveys is the difficulties contacting respondents over the phone. Successful contact is subject to mobile network availability and quality, whether the respondent’s mobile phone is charged and operational, whether the respondent answers a call from an unknown number, etc. After successful contact, nonresponse can equally result from refusal of the respondent to participate in the interview or a breakoff mid-way through the interview. These two sources of nonresponse are also present in F2F surveys but could be more substantial in phone surveys since respondents might be less responsive to an unseen interviewer over the phone than an interviewer they meet in person. Bias due to this nonresponse will be introduced if the nonresponse is nonrandom with some segments of the population being more likely to not respond. Given the additional sources of nonresponse common to phone surveys, there is also a higher potential for systematic bias due to that nonresponse. 2.1 Methods to Counteract Bias While coverage and nonresponse bias are real concerns with phone surveys, there are methods to attempt to reduce the bias. The methods available will vary according to the type of frame used for the survey. There are three predominant types of frames used for telephone surveys: (1) existing representative F2F surveys that collected phone numbers of respondents (for example, LSMS-ISA, DHS, HBS, etc.), (2) a list of phone numbers from a telecom provider, and (3) a list of numbers generated randomly (random digit dialing, or RDD) (Himelein et al, 2020). While there are advantages 3 and disadvantages to each of these approaches, 5 there are some distinct advantages to using an existing F2F survey when it comes to counteracting coverage and nonresponse bias. The F2F survey that the phone sample is drawn from will typically contain detailed information across a wide variety of domains not just for interviewed households but also households that were ineligible (i.e. did not have access to a mobile phone) and those that were not successfully interviewed (i.e. those that did not respond or refused). This wealth of information can be harnessed to more effectively reduce bias in the phone survey sample. RDD surveys and surveys utilizing network provider frames often do not have the luxury of having detailed information on the ineligible and nonresponding sample to attempt to reduce the bias and thus any bias adjustments employed for those surveys are potentially inferior to those from representative F2F surveys. 6 The most prominent method to achieve a reduction in bias in telephone surveys is throug h adjustments to the survey weights. In order to implement these weighting adjustments, additional information is required to understand and attempt to correct for the bias. External information on known demographic composition of the general population (e.g. from population census) can be compared with that of the interviewed sample and the weights adjusted to counteract that balance. Such external information is usually limited to basic demographics. More detailed information from recent representative F2F surveys (e.g. LSMS, Demographic and Health Surveys, Household Budget Surveys, etc.) could similarly be compared with the interviewed sample. However, for RDD and telecom list-based phone surveys, information on these characteristics must also captured in the phone survey which is not always feasible in a phone survey format where the scope (in terms of question complexity) and length (in terms of interview time) are much reduced compared to a typical F2F surveys. For example, it is not feasible to collect all information necessary to construct an aggregate measure of household consumption expenditures, the prevailing welfare measure in many developing countries. Phone surveys utilizing existing representative surveys as a frame have much more scope for effective bias adjustments through reweighting as a result of the rich set of information available for the representative sample that serves as a frame. There are a variety of approaches to counteract the biases present in phone surveys utilizing the detailed information from the representative F2F survey, though three prominent methods are explored here. The first is through weighting class adjustments, whereby the phone survey sample is divided into cells that represent different “classes” of respondents (Little, 1986). These cells are typically constructed by crossing several characteristics that may be correlated with the likelihood of nonresponse (e.g. gender, age, education, location, etc.). The phone survey response rate is then calculated for each of these cells and the weights for interviewed households in the cell increased by the inverse of the response rate. 5 See Himelein et al (2020) for a more complete comparison of the advantages and disadvantages of these three sampling approaches. 6 One disadvantage of using a representative F2F survey as a frame (compared to RDD and list-based frames) is that F2F surveys are often a two-stage clustered design with census enumeration areas typically serving as the clusters from which households/respondents are selected. This clustered design introduces design effects that reduces the effective sample size and increases the size of the overall sample required to achieve the desired level of precision (relative to a simple random sample). These design effects are at least partially retained in a phone sample drawn from the clustered F2F survey sample. These imported design effects imply that a larger sample is required to achieve the same level of precision for a phone survey drawn from a clustered F2F survey than an RDD or list-based phone survey (which is not clustered). 4 Another method is propensity score adjustments , whereby the probability of response is modeled (typically using a probit or logit model) using the response status from the phone survey as well as observable household/individual characteristics for the entire sample (both responders and nonresponders) derived from the representative survey (Little, 1986). The probability of response for each sampled unit is then predicted using the model parameters and the inverse of the predicted response probability (or propensity) applied to the base weights. This approach could introduce extreme weight adjustments and thereby increase the variance of the weights (and estimates obtained). However, the risk of this can be reduced by forming response classes based on the predicted probability (e.g. probability deciles) and the average predicted probability within the class used as the adjustment factor. The propensity modeling approach is particularly attractive for telephone surveys drawn from representative surveys that contain a diverse set of characteristics that can be incorporated into the propensity model. A further method that can be employed is through calibration of nonresponse (Lundström & Särndal, 1999; Andersson & Särndal, 2016). Calibration is a weight adjustment technique utilized in many surveys whereby weights are adjusted so that the weighted sample composition matches known characteristics of the underlying population. Typically, auxiliary data from population censuses are utilized for these calibrations. However, this approach can be extended to also include sample-based estimates from a representative survey used as a frame (Andersson & Särndal, 2016). In this case, the weights for the interviewed sample will be calibrated across all considered characteristics such that the weighed profile of characteristics obtained from the interviewed sample closely match those from the full representative survey sample. Provided the characteristics included in the calibration are closely associated with nonresponse, the calibration will serve to counteract the nonrespons e bias. There are many complex calibration models that could be used (and are beyond the scope of this paper), but most are subject to an algorithm that minimizes the distance between the uncalibrated and calibrated weights and therefore might result in a lower increase in variation of the adjusted weights compared to the propensity modeling approach. 3. Data and Methods 3.1 High Frequency Phone Surveys – Ethiopia, Malawi, Nigeria, & Uganda In order to examine bias in phone surveys and effectiveness of bias adjustment methods, this study utilizes four recent high frequency phone surveys (HFPS) in Ethiopia, Malawi, Nigeria, and Uganda. These four surveys were implemented as part of a broader initiative by the World Bank to support implementation of high frequency phone surveys in several developing countries to monitor the socioeconomic impact of the COVID-19 pandemic. The baseline rounds of the Ethiopia and Nigeria HFPS were conducted in April shortly after the rapid spread of the pandemic and safety protocols and restrictions put in place. The baseline rounds of Malawi and Uganda were implemented shortly thereafter in May and June. The four HFPSs contacted households drawn from the most recent round of Living Standards Measurement Study – Integrated Surveys on Agriculture (LSMS-ISA). The LSMS-ISA are longitudinal, face-to-face, representative surveys conducted in partnership with National Statistics Offices in the respective countries. In Ethiopia, the fourth round of the Ethiopia Socioeconomic Survey (ESS) conducted in 2018/19 served as the frame for the HFPS. The fifth round of the Integrated Household Panel Survey (IHPS) 5 conducted in 2019 served as the frame for the Malawi HFPS. The sixth round of the Uganda National Panel Survey (UNPS) fielded in 2019/20 served as the frame for the Uganda HFPS. Lastly, the fourth round of the General Household Survey – Panel (GHS-Panel) in 2018/19 served as the frame for the Nigeria HFPS. The four LSMS-ISA surveys that served as the frames for the HFPS are panel surveys that follow the same households over time. To facilitate recontact and tracking in the subsequent wave of the survey, interviewers collected phone numbers of up to 4 household members and 2 non-household member reference persons. It is this contact information that enabled seamless implementation of the HFPS surveys in each country. However, the availability of contact details varies between the countries, largely due to much higher mobile phone penetration in Nigeria than in Ethiopia, Malawi, and Uganda. Figure 3.1 summarizes the LSMS-ISA samples, contact information availability, and selection process for the HFPS in each country. The share of LSMS-ISA households with phone numbers (i.e. the coverage rate in Figure 3.1) ranges from 73 percent in Malawi and Uganda up to 99 percent in Nigeria. This at least partially reflects the much higher mobile phone penetration in Nigeria compared with the other three countries. 7 Figure 3.2 presents estimates of phone ownership in the four countries. Nigeria has the highest share of households who own a mobile phone at 76 percent, followed by Uganda (69 percent), Malawi (59 percent), and Ethiopia (48 percent). In all four countries, the LSMS-ISA sample households with contact information (row B in Figure 3.1) served as the frame for the HFPS. Figure 3.1 Survey sample and selection process for HFPS Survey Sample and Selection Processs Ethiopia Malawi Nigeria Uganda F2F Survey Sample ESS 2018/19 IHPS 2019 GHS 2018/19 UNPS 2019/20 Size (A) 6,770 Households 3,181 Households 4,976 households 3,308 households With phone numbers 5,372 2,337 4,934 2,421 (B) Coverage Rate 79% 73% 99% 73% (B/A) Initial HFPS Sample 5,372 2,337 3,000 2,421 (C) Reached 3,350 1,743 2,057 2,274 (D) Interviewed 3,249 1,729 1,950 2,259 (E) Response Rate 6 60% 74% 65% 93% (E/C) 7Included in the share of the LSMS-ISA sample with phone numbers are households that did not provide a phone number of a household member but did provide the phone number of at least one reference person. As a result, the share with phone numbers does not necessarily reflect whether the household has a phone, but whether they provided any phone number of a member or reference person. 6 3.1.1 Survey Sample The selection of the HFPS sample of households proceeded differently in the four countries. In Nigeria, 3,000 households were selected from the frame of 4,934 GHS-Panel households with contact details. The target number of successfully interviewed households to produce nationally representative estimates with reasonable precision was 1,800. Given the large amount of auxiliary information available in the GHS-Panel for these households, a balanced sampling approach using the cube method (Tille, 2006) was adopted for the selection of the 3,000 households for the HFPS. 8 For the first-round of the Nigeria HFPS, all the 3,000 sampled households were contacted and 69 percent of them were successfully reached by the interviewers (2,057 households). Of those contacted, 94 percent or 1,950 households were successfully interviewed. This yielded an overall crude response rate 9 for the sample of 65 percent. Figure 3.2: Phone Ownership Share of Households Who Own a Phone 100.0 88.7 88.0 90.0 80.0 76.0 75.7 73.3 70.5 68.8 70.0 65.6 58.9 60.0 51.8 National 47.9 50.0 Urban 40.0 35.7 Rural 30.0 20.0 10.0 0.0 Ethiopia Malawi Nigeria Uganda Source: Authors’ calculations from the ESS 2018/19, IHPS 2019, GHS-Panel 2018/19, & UNPS 2019/20 For the Ethiopia HFPS, to obtain representative estimates at the national, urban, and rural level, the target sample size for the HFPS was 3,300 households: 1,300 in rural and 2,000 households in urban areas. To account for non-response and attrition all of the 5,374 households with contact information in the ESS were called in round 1 of the HFPS. In rural areas 1,413 households owning a phone and 771 households with reference phone were contacted. Whereas in urban areas 3,213 households owing phone and 24 households with a reference phone numbers were contacted. A total of 3,249 households (2,271 urban and 978 rural households were fully interviewed yielding an overall crude response rate of 60 percent (67% for urban and 50% for rural households). 8 The variables considered in the cube sampling approach are the same variables included in the bias correction calibration below (displayed in Table 3.2). 9 Response and contact rates reported here are “crude” in that they do not cleanly correspond to AAPOR standard definitions. This crude calculation is retained for the sake of simplicity. 7 For both the Malawi and Uganda, the entire sample of households from the latest round of the LSMS- ISA which had contact information served as the HFPS sample. In Malawi, the initial HFPS sample consisted of 2,337 households while for Uganda it consisted of 2,421 households. The crude contact and response rate was very high in Uganda where 93 percent (2,259) of sampled households were successfully interviewed. Malawi’s contact and response rates were slightly higher than Nigeria and Ethiopia with 74 percent (1,729) of sampled households being successfully interviewed. 3.1.2 Sampling Weight and Bias Adjustments Sampling weights were calculated in all four surveys following similar methods. The sampling weight for the most recent round of the LSMS-ISA survey served as the starting point in all four countries since these weights produce representative estimates from the full sample of households in those F2F surveys. These weights were then inflated through basic ratio adjustments at each stage of the selection and interview process from the frame to the interviewed sample (depicted in Figure 3.1). These ratio adjustments preserve the original sum of weights from the full sample of the LSMS-ISA survey. These naïve ratio adjustments do not take into account any nonrandom bias contained in the sample. The ratio adjustments were then followed by the weighting adjustments to reduce bias in the interviewed sample relative to the F2F representative sample. The weighting adjustment implemented in each country serves to counteract both coverage and nonresponse bias simultaneously. However, the approaches taken at this stage vary in the four countries. For the Ethiopia, Malawi, and Uganda HFPS, the bias adjustment was conducted using a propensity model approach following Himelein (2014). The probability of household response was modeled using a logistic regression. The characteristics included in the logit response probability module are presented in Table 3.2 and were selected due to observed bias in the interviewed sample along these dimensions. Full results of the logit model are provided in Appendix Table 3.1. The predicted probability of response from the model parameters was obtained for each household the inverse of which was the basis for the bias adjustment factor. However, in order to prevent extreme weights due to the correction factor, the predicted probabilities were sorted into deciles and the mean inverse response probability within each decile taken as the final weight adjustment factor. In Nigeria, the bias adjustment was performed using the calibration approach whereby the weights were adjusted (minimizing the distance between the original and calibrated weights) to achieve the same weighted estimates across selected characteristics in the GHS-Panel sample. Table 3.2 contains the list of characteristics that were used in the calibration model. These characteristics were selected for inclusion in the calibration due to observed bias in the interviewed sample across these dimensions. Adjusting the weights to match the same profile of characteristics obtained from the fully representative GHS-Panel sample will counteract the bias along these dimensions. However, one drawback to the calibration approach is that it can be difficult to consider a large set of characteristics and achieve convergence in the calibration model. 8 Table 3.2: Characteristics Included in Bias Adjustment Characteristics Ethiopia Malawi Nigeria Uganda Residence (Urban/Rural) X X Age of the household Head (age and age squared) X X Gender of the household head X X X Household Size X X X X Marital Status (Married=Yes) X Literacy of the household head X X Language spoken by household head X Education of the household head X X X Highest years of education among household members X Consumption quintile (Q1-Q5) X X X Wealth quintile (Q1-Q5) X Food consumption score X Household received remittance (domestic or international) X Household received cash transfer from government X Any household member has a financial account X Any household member had access to credit X Household owns dwelling X Number of people per room X Employment status of the household head X Household head engaged in agriculture X Household head engaged in casual labor X Sector of household head employment X Share of household members engaged in wage employment X Share of household members engaged in informal (ganyu) employment X Coping strategies index X Mobile phone ownership X Number of months household head was away from the household X Following the bias adjustments, the weights were trimmed at the 2nd and 98th percentiles to prevent outlier weights that would result in a substantial increase in the variance of the weights. Lastly, the weights were post-stratified to match population totals in each country (following the same post- stratification approach utilized in the LSMS-ISA weight calculation). For full details on the sampling and weight calculation for Ethiopia and Nigeria, see Ambel et al (2020) and McGee et al (2020), respectively. Full documentation as well as the microdata for these four surveys can be found on the World Bank’s Microdata Catalogue 10 or on the LSMS website. Ethiopia HFPS: https://microdata.worldbank.org/index.php/catalog/3716 10 Malawi HFPS: https://microdata.worldbank.org/index.php/catalog/3766 Nigeria HFPS: https://microdata.worldbank.org/index.php/catalog/3712 Uganda HFPS: https://microdata.worldbank.org/index.php/catalog/3765 9 3.1.3 Method of Analysis The analysis undertaken in this study involves two steps. First, the magnitude and different sources of bias introduced in the phone survey selection and interview process are identified. Several subsamples are compared, reflecting the eligibility, selection, and interview process for the HFPS in the four countries. To measure the bias, pairwise comparisons of weighted means across a wide variety of household characteristics is carried out. In order to assess the extent of coverage bias, differences between the full F2F survey sample and the eligible sample for the HFPS (i.e. households with contact information) are examined. To measure sample representativeness in Nigeria, the profile of the selected sample is compared with the frame of eligible households. Lastly, to estimate nonresponse bias, the sample of households successfully contacted and those successfully interviewed is compared with the overall HFPS sample. Comparing the profile of characteristics for these samples will provide a more refined impression of the bias introduced throughout the course of the phone survey process. This comparative analysis will also reveal at what stage in the selection process the bias is the most substantial. After conducting the comprehensive assessment of bias, an assessment of the reweighting adjustments to counteract bias is conducted to see how effective these methods were at reducing bias. Weighted estimates for the diverse set of characteristics examined in the first analysis is presented for the full representative F2F survey as well as the HFPS. The F2F survey characteristics serve as the benchmark and are compared with the profile of characteristics obtained from the HFPS sample applying the weights that include the bias adjustments (as described in Section 3.1.3). Comparing these two sets of weighted characteristics will show how much bias remains in the sample after the correction. However, the relative effectiveness of the bias adjustments is further analyzed by computing a second set of HFPS weights which exclude the bias adjustments. The process of calculating these “unadjusted” weights follows exactly the same steps as the “adjusted” weights and only excludes the bias adjustment. The unadjusted weights serve to simulate the extent of bias that would have been present without the bias adjustments. By comparing the relative deviations from the F2F sample profile, this will provide an indication of how effective the bias adjustment techniques employed were in reducing the extent of bias in the HFPS sample. Finally, the advantages of drawing phone survey samples from representative F2F surveys is highlighted based on the performance shown using adjusted weights when compared to unadjusted weights. All the pair-wise comparisons of characteristic means in the analysis below are conducted following the same approach. All means are weighted using either the LSMS-ISA weights or weights calculated from the HFPS and the sampling properties of each survey are accounted for (clustering, stratification, etc.). An adjusted Wald test is used to determine if differences in the means between the different subsamples are significant. 10 4. Results 4.1 Sources of Bias Before examining how to correct for bias, it is important to first understand the sources of bias that enter into the phone survey sample and how that bias affects the composition of the sample. Tables 4.1a to 4.3b present weighted means for a wide array of characteristics 11 from the four LSMS-ISA F2F surveys across several different subsamples ending ultimately with the sample of successfully interviewed households in the HFPS for each country. Comparing the means across these subsamples will indicate (1) at what stage any bias is entering into the samples and (2) across what set of characteristics the samples are biased in relation to the F2F surveys. Figure 4.1 also presents estimated bias (measured by deviation from LSMS-ISA mean) in graphical form for selected characteristics. 4.1.1 Coverage bias Table 4.1a presents results comparing characteristics for the full F2F sample (columns 1, 4, 7, and 10) as well as the portion of the F2F sample that has contacted information (columns 2, 5, 8, and 11) for the samples from all four countries. The portion of the sample with contact information serves as the frame of eligible households to be selected for the phone survey sample. Therefore, any differences between these two samples would indicate the presence of coverage bias brought on by exclusion of households who are ineligible (i.e. have no phone number to be contacted on). Columns 3, 6, 9, and 12 present differences between the two samples (with significant differences indicated with “*” and highlighted in red). Looking down the variety of characteristics contained in Table 4.1a for Nigeria (column 9), it is evident that there are no substantial differences between the full sample and phone survey frame, suggesting coverage bias is not a major concern for Nigeria. This is not surprising however, since the share of households without contact information is very small (less than 1%) reflecting the relatively high penetration of mobile phones in the country. However, the results presented in columns 3, 6 and 12 (Table 4.1a) provide a clear indication of coverage bias in the Ethiopia, Malawi and Uganda samples. Households with contact information are generally richer (as measured by per capita consumption expenditure), more likely to own key assets (TVs, refrigerators, and mobile phones), more likely to live in dwellings with improved features such as a modern roof and floor, improved water source and toilet facilities, and electricity, as well as more likely to have a financial account. In addition, the household heads in the eligible sample tend to be better educated and more likely to working in more formal wage employment (in Ethiopia and Malawi). However, when the Ethiopia sample is disaggregated into urban and rural areas (the two domains for that sample) in Table 4.1b, many of the indications of coverage bias are not observed, particularly for the urban sample. This suggests that coverage bias is not a serious concern in urban areas, but may be a more serious concern in rural areas where mobile phone penetration is much lower and as a result 38 percent of ESS rural households are ineligible (compared to just 7% of urban households). 11In order to save space, a subset of characteristics analyzed are presented in the results that follow. The results for the full set of characteristics considered can be found in the online appendix. 11 Figure 4.1: Sources of Bias Relative to LSMS-ISA 12 Note: Bias as measured by mean difference between indicated subsample and full LSMS-ISA sample (with 95% confidence intervals) 13 Table 4.1a: Coverage bias in HFPS Ethiopia (ESS) Malawi (IHPS) Nigeria (GHS-Panel) Uganda (UNPS) All With Contact info All With Contact info All With Contact info All With Contact info Coverage Coverage Coverage Coverage Mean Mean bias† Mean Mean bias† Mean Mean bias† Mean Mean bias† (2) - (1) (5) - (4) (8) - (7) (11) - (10) Characteristic (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) Sample size (number of households) 6,788 5,372 3,181 2,337 4,976 4,934 3,308 2,421 Rural 0.675 0.588 -0.088** 0.826 0.765 -0.060 0.686 0.683 -0.002 0.684 0.659 -0.026 Average household size 4.483 4.484 0.000 4.904 4.990 0.086 5.528 5.515 -0.013 4.515 4.613 0.098 Dependency ratio 0.955 0.915 -0.040 1.055 0.975 -0.079** 1.016 1.014 -0.002 1.064 1.062 -0.002 Female head (%) 0.263 0.262 -0.001 0.296 0.262 -0.034 0.186 0.187 0.001 0.342 0.332 -0.010 Age of head 42.912 41.962 -0.950 45.855 44.911 -0.945 48.797 48.775 -0.022 45.311 45.133 -0.178 Religion of head Orthodox 0.478 0.485 0.007 Muslim 0.293 0.284 -0.010 0.808 0.828 0.020 0.425 0.422 -0.003 Christian 0.211 0.217 0.006 0.145 0.133 -0.012 0.557 0.560 0.003 Marital status of head Married 0.739 0.740 0.001 0.716 0.744 0.028 0.750 0.749 -0.001 0.713 0.738 0.024 Formerly married 0.189 0.175 -0.014 0.265 0.232 -0.033 0.193 0.194 0.001 0.246 0.227 -0.019 Never married 0.072 0.085 0.013 0.018 0.023 0.005 0.057 0.057 0.000 0.041 0.036 -0.005 Education level of head Head Literate 0.520 0.586 0.067*** 0.921 0.944 0.023** 0.744 0.745 0.002 0.772 0.809 0.037** None 0.523 0.456 -0.067*** 0.675 0.591 -0.084** 0.375 0.371 -0.003 0.087 0.076 -0.011 Primary 0.296 0.310 0.015 0.207 0.251 0.043 0.241 0.242 0.000 0.584 0.555 -0.030 Secondary 0.102 0.129 0.027** 0.083 0.111 0.028* 0.225 0.226 0.001 0.290 0.307 0.017 Tertiary 0.079 0.105 0.025** 0.034 0.047 0.013 0.160 0.161 0.001 0.120 0.132 0.012 Working status of head Any 0.735 0.724 -0.011 0.819 0.829 0.010 0.827 0.828 0.001 0.855 0.865 0.009 Wage 0.133 0.165 0.032* 0.158 0.198 0.04* 0.177 0.178 0.001 0.324 0.326 0.002 Agriculture 0.551 0.492 -0.059* 0.469 0.445 -0.025 0.442 0.442 0.000 0.479 0.474 -0.005 Nonfarm enterprise 0.115 0.136 0.020 0.193 0.214 0.021 0.367 0.367 0.000 0.266 0.288 0.023 Ganyu 0.248 0.205 -0.043* †The results of an adjusted Wald test comparing weighted means are presented (*** p<0.01, ** p<0.05, * p<0.10). ` 14 Table 4.1a (continued): Coverage bias in HFPS Ethiopia (ESS) Malawi (IHPS) Nigeria (GHS-Panel) Uganda (UNPS) All With Contact info All With Contact info All With Contact info All With Contact info Coverage Coverage Coverage Coverage Mean Mean bias† Mean Mean bias† Mean Mean bias† Mean Mean bias† (2) - (1) (5) - (4) (8) - (7) (11) - (10) Characteristic (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) Consumption Mean expenditure (LCU per capita) 14,943 16,247 1303.597* 219408 220234 826 373,830 399,209 25379.198* Quintilesa Quintile 1 - poorest 0.200 0.166 -0.034 0.201 0.106 -0.096*** 0.201 0.197 -0.004 0.156 0.135 -0.021 Quintile 2 - poorer 0.200 0.186 -0.014 0.199 0.161 -0.037** 0.199 0.200 0.001 0.174 0.164 -0.010 Quintile 3 - middle 0.200 0.207 0.006 0.200 0.208 0.008 0.200 0.201 0.001 0.195 0.196 0.001 Quintile 4 - richer 0.200 0.214 0.014 0.200 0.250 0.05** 0.200 0.201 0.001 0.221 0.226 0.005 Quintile 5 - richest 0.200 0.228 0.028 0.200 0.275 0.075** 0.200 0.201 0.001 0.245 0.269 0.024 Housing Improved roof 0.689 0.739 0.050 0.589 0.681 0.092*** 0.886 0.887 0.001 0.797 0.833 0.036*** Improved floor 0.177 0.228 0.051** 0.333 0.421 0.088** 0.776 0.780 0.003 0.466 0.512 0.046** Improved toilet 0.313 0.367 0.053* 0.043 0.056 0.013 0.602 0.603 0.001 0.400 0.435 0.034* Improved water source 0.747 0.785 0.039 0.844 0.852 0.009 0.765 0.766 0.000 0.787 0.802 0.015 Access to electricity 0.559 0.634 0.074** 0.150 0.207 0.057* 0.554 0.558 0.004 0.257 0.288 0.031* Asset ownership Solar 0.153 0.148 -0.005 0.207 0.252 0.045* Generator 0.246 0.247 0.002 0.404 0.425 0.021 TV 0.178 0.234 0.056** 0.152 0.208 0.056** 0.451 0.454 0.003 0.257 0.291 0.034* Refrigerator 0.072 0.095 0.024** 0.080 0.112 0.032 0.173 0.174 0.001 0.071 0.076 0.005 Bicycle 0.012 0.016 0.004 0.416 0.453 0.037 0.157 0.157 0.000 0.261 0.275 0.014 Motorbike 0.009 0.012 0.003 0.044 0.061 0.016 0.306 0.305 0.000 0.114 0.126 0.012 Car or other vehicle 0.009 0.012 0.003 0.025 0.034 0.009 0.096 0.097 0.001 0.039 0.041 0.002 Mobile Phone 0.553 0.706 0.153*** 0.582 0.784 0.201*** 0.760 0.764 0.004 0.796 0.872 0.076*** Other activities Owns land 0.824 0.779 -0.045** 0.890 0.854 -0.036 0.669 0.667 -0.002 0.608 0.598 -0.010 Crop farming 0.658 0.611 -0.048 0.812 0.755 -0.057* 0.699 0.697 -0.002 0.725 0.712 -0.013 Owns any livestock 0.684 0.636 -0.048 0.476 0.498 0.022 0.469 0.466 -0.003 0.559 0.557 -0.002 Operates nonfarm enterprise 0.229 0.259 0.030 0.409 0.459 0.05* 0.595 0.596 0.001 0.023 0.023 0.000 Any member has financial account 0.457 0.547 0.09*** 0.339 0.419 0.08*** 0.493 0.496 0.004 0.595 0.643 0.047** Income from remittance (int'l & domestic) 0.145 0.161 0.016 0.528 0.532 0.004 0.319 0.321 0.002 0.340 0.347 0.006 aForMalawi, the quintiles are based on a wealth index, not consumption expenditures. †The results of an adjusted Wald test comparing weighted means are presented (*** p<0.01, ** p<0.05, * p<0.10). 15 Table 4.1b: Coverage bias in Ethiopia HFPS by Urban-Rural domain Ethiopia (ESS - Urban) Ethiopia (ESS - Rural) All With Contact info All With Contact info Coverage Coverage Mean Mean bias† Mean Mean bias† (2) - (1) (5) - (4) Characteristic (1) (2) (3) (4) (5) (6) Sample size (number of households) 3,655 3,410 3,115 1,962 Average household size 3.647 3.651 0.004 4.885 5.067 0.182 Dependency ratio 0.651 0.635 -0.016 1.103 1.113 0.010 Female head (%) 0.346 0.339 -0.008 0.223 0.208 -0.015 Age of head 38.728 38.584 -0.144 44.924 44.332 -0.592 Religion of head Orthodox 0.574 0.579 0.006 0.431 0.418 -0.013 Muslim 0.207 0.201 -0.006 0.335 0.341 0.007 Christian 0.213 0.214 0.001 0.210 0.220 0.010 Marital status of head Married 0.618 0.621 0.003 0.798 0.824 0.026 Formerly married 0.220 0.214 -0.006 0.174 0.148 -0.026 Never married 0.162 0.164 0.003 0.029 0.028 0.000 Education level of head Head Literate 0.753 0.775 0.023 0.408 0.454 0.046* None 0.287 0.263 -0.024 0.637 0.592 -0.045 Primary 0.296 0.301 0.005 0.296 0.317 0.022 Secondary 0.215 0.226 0.010 0.047 0.060 0.013 Tertiary 0.202 0.210 0.008 0.020 0.031 0.010 Working status of head Any 0.592 0.594 0.002 0.804 0.816 0.011 Wage 0.302 0.310 0.008 0.051 0.063 0.012 Agriculture 0.124 0.115 -0.009 0.756 0.756 -0.001 Nonfarm enterprise 0.219 0.220 0.001 0.066 0.077 0.011 Consumption Mean expenditure (LCU per capita) 22,876 23,088 212 11,130 11,449 319 Quintiles Quintile 1 - poorest 0.059 0.053 -0.006 0.268 0.245 -0.023 Quintile 2 - poorer 0.107 0.100 -0.007 0.245 0.245 0.001 Quintile 3 - middle 0.158 0.159 0.001 0.221 0.240 0.019 Quintile 4 - richer 0.274 0.277 0.002 0.164 0.170 0.006 Quintile 5 - richest 0.402 0.411 0.009 0.103 0.100 -0.003 †The results of an adjusted Wald test comparing weighted means are presented (*** p<0.01, ** p<0.05, * p<0.10). 16 Table 4.1b (continued): Coverage bias in Ethiopia HFPS by Urban-Rural domain Ethiopia (ESS - Urban) Ethiopia (ESS - Rural) All With Contact info All With Contact info Coverage Coverage Mean Mean bias† Mean Mean bias† (2) - (1) (5) - (4) Characteristic (1) (2) (3) (4) (5) (6) Housing Improved roof 0.954 0.961 0.008 0.562 0.583 0.022 Improved floor 0.482 0.497 0.015 0.030 0.040 0.010 Improved toilet 0.593 0.606 0.013 0.179 0.199 0.020 Improved water source 0.976 0.978 0.002 0.637 0.650 0.013 Access to electricity 0.905 0.919 0.014 0.393 0.433 0.040 Asset ownership Solar 0.051 0.051 0.000 0.202 0.217 0.015 Generator TV 0.505 0.525 0.019 0.021 0.030 0.009 Refrigerator 0.213 0.223 0.010 0.004 0.006 0.002 Bicycle 0.032 0.033 0.002 0.002 0.003 0.001 Motorbike 0.019 0.020 0.001 0.004 0.007 0.002 Car or other vehicle 0.028 0.030 0.001 0.000 0.000 0.000 Mobile Phone 0.866 0.904 0.038** 0.403 0.568 0.164*** Other activities Owns land 0.962 0.957 -0.006 Crop farming 0.876 0.899 0.022 Owns any livestock 0.882 0.895 0.013 Operates nonfarm enterprise 0.380 0.381 0.001 0.157 0.174 0.017 Any member has financial account 0.773 0.799 0.026 0.305 0.370 0.065** Income from remittance (int'l & domestic) 0.215 0.213 -0.001 0.111 0.124 0.013 †The results of an adjusted Wald test comparing weighted means are presented (*** p<0.01, ** p<0.05, * p<0.10). 4.1.2 Sample representativeness The Nigeria HFPS included a sample selection performed on the frame (GHS-Panel households with a phone number) to reach the phone survey sample (see Section 3.1.1 above for more details on the Nigeria sample selection). This is in contrast to Ethiopia, Malawi, and Uganda where the entire frame served as the phone survey sample in those countries. As a result of this additional sampling step in Nigeria, it is important to examine just how representative the sample selected is. Unlike the issues of coverage and response, no distortions in the representativeness of the sample are expected from this sampling selection, but it is nonetheless useful to confirm this is the case. Table 4.2 presents the mean characteristics for the full GHS-Panel sample, the frame (households with contact information), and the selected sample for the Nigeria HFPS (columns 1, 2, and 3, respectively). Column 4 also contains the difference between the mean characteristics observed in the sample and the frame (with significance 17 indicated as above). There are no significant differences for any characteristics between the sample and the frame, suggesting that the balanced sampling approach performed well, and the Nigeria HFPS sample retains the representativeness present in the frame. Table 4.2: Sample representativeness in Nigeria HFPS Nigeria (GHS-Panel) With HFPS Sample All Contact info Difference † Mean Mean Mean (3) - (2) Characteristic (1) (2) (3) (4) Sample size (number of households) 4,976 4,934 3,000 Rural 0.686 0.683 0.687 -0.001 Average household size 5.528 5.515 5.530 -0.002 Dependency ratio 1.016 1.014 1.028 -0.012 Female head (%) 0.186 0.187 0.186 0.001 Age of head 48.797 48.775 48.646 0.151 Religion of head Orthodox Muslim 0.425 0.422 0.425 0.000 Christian 0.557 0.560 0.556 0.001 Marital status of head Married 0.750 0.749 0.749 0.000 Formerly married 0.193 0.194 0.192 0.001 Never married 0.057 0.057 0.058 -0.001 Education level of head Head Literate 0.744 0.745 0.748 -0.005 None 0.375 0.371 0.364 0.011 Primary 0.241 0.242 0.253 -0.012 Secondary 0.225 0.226 0.231 -0.006 Tertiary 0.160 0.161 0.152 0.007 Working status of head Any 0.827 0.828 0.830 -0.003 Wage 0.177 0.178 0.186 -0.009 Agriculture 0.442 0.442 0.430 0.012 Nonfarm enterprise 0.367 0.367 0.375 -0.007 Consumption Mean expenditure (LCU per capita) 219408 220234 220372 -964 Quintiles Quintile 1 - poorest 0.201 0.197 0.201 0.000 Quintile 2 - poorer 0.199 0.200 0.196 0.004 Quintile 3 - middle 0.200 0.201 0.201 -0.001 Quintile 4 - richer 0.200 0.201 0.197 0.003 Quintile 5 - richest 0.200 0.201 0.206 -0.006 †The results of an adjusted Wald test comparing weighted means are presented (*** p<0.01, ** p<0.05, * p<0.10). 18 Table 4.2 (continued): Sample representativeness in Nigeria HFPS Nigeria (GHS-Panel) With HFPS Sample All Contact info Difference† Mean Mean Mean (3) - (2) Characteristic (1) (2) (3) (4) Housing Improved roof 0.886 0.887 0.884 0.002 Improved floor 0.776 0.780 0.777 -0.001 Improved toilet 0.602 0.603 0.609 -0.007 Improved water source 0.765 0.766 0.764 0.001 Access to electricity 0.554 0.558 0.565 -0.010 Asset ownership Solar Generator 0.246 0.247 0.237 0.009 TV 0.451 0.454 0.462 -0.011 Refrigerator 0.173 0.174 0.178 -0.005 Bicycle 0.157 0.157 0.151 0.006 Motorbike 0.306 0.305 0.310 -0.004 Car or other vehicle 0.096 0.097 0.094 0.002 Mobile Phone 0.760 0.764 0.752 0.008 Other activities Owns land 0.669 0.667 0.666 0.004 Crop farming 0.699 0.697 0.696 0.003 Owns any livestock 0.469 0.466 0.464 0.005 Operates nonfarm enterprise 0.595 0.596 0.602 -0.007 Any member has financial account 0.493 0.496 0.500 -0.007 Income from remittance (int'l & domestic) 0.319 0.321 0.308 0.011 †The results of an adjusted Wald test comparing weighted means are presented (*** p<0.01, ** p<0.05, * p<0.10). 4.1.3 Non-response bias The last source of bias to examine is for non-response. Non-response as considered here includes cases from the HFPS sample that were either (1) not successfully contacted or (2) contacted but not successfully interviewed (refused or breakoff mid-interview). Both of these sources of non-response (non-contact and non-interview) will be considered separately here. Table 4.3a presents the comparison of the weighted mean characteristics for the HFPS sample (columns 1, 6, 11, and 16), sample of households successfully contacted in the HFPS (columns 2, 7, 12, and 17), and the sample of households successfully interviewed in the HFPS (columns 4, 9, 14, and 19) for the four 19 countries. Columns 3, 8, 13, and 18 present the differences in characteristics between the successfully contacted sample and the overall HFPS sample, which represents nonresponse bias due to the inability to contact sampled households. This reflects nonresponse due to poor network availability; respondent mobile phones being switched off or non-functioning, respondent’s unwillingness to answer a call from an unknown number, etc. For all four countries there appears to be some substantial bias introduced at this stage. Compared to the HFPS sample, successfully contacted households were more likely to reside in urban areas (for Ethiopia and Malawi), are wealthier (in terms of consumption expenditures, asset ownership, and housing), and have heads that are better educated. For Ethiopia, Malawi, and Uganda, the bias due to noncontact appears to further magnify the coverage bias with many of the same indicators showing significant bias and in the same direction as that for coverage bias. Non-response bias due to non-contact does appear to be less substantial for Uganda with fewer significant differences in Column 18 compared to the other countries’ samples. However, this is largely a product of the much higher successful contact rate in Uganda compared to the others. When disaggregating between the urban and rural HFPS sample in Ethiopia (shown in Table 4.3b), the extent of the detected bias due to noncontact is reduced for both samples. However, bias due to noncontact is detected across more indicators in the urban sample suggesting that bias due to noncontact could be more prevalent for that sample. However, the direction of the bias (in favor of wealthier households) is consistent across the urban and rural samples in Ethiopia. Lastly, columns 5, 10, 15, and 20 in Table 4.3a present estimates of bias due to non-interview of successfully contacted households in the four countries’ samples. This represents any bias due to respondent refusal to participate in the survey or a break-off during the interview. For all four countries, the bias introduced at this stage appears to be minimal with no significant difference observed for any indicators. This is unsurprising however since refusal and breakoffs were relatively uncommon in Ethiopia, Malawi, Uganda, and Nigeria (representing only 3%, 1%, 5%, and 1% of successfully contacted households, respectively). Minimal bias due to non-interview was also observed in both the urban and rural HFPS samples in Ethiopia in Table 4.3b. 20 Table 4.3a: Non-response bias in HFPS Ethiopia Malawi Sample Contacted Interviewed Sample Contacted Interviewed Bias due to Bias due to Bias due to Bias due to non- non- Mean Mean Mean non-interview† Mean Mean Mean non-interview† contact† contact† (4) - (2) (9) - (7) (2) - (1) (7) - (6) Characteristic (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) Sample size (number of households) 5,372 3,353 3,247 2,337 1,743 1,729 Rural 0.588 0.533 -0.055*** 0.535 0.002 0.765 0.749 -0.016* 0.749 0.000 Average household size 4.484 4.461 -0.023 4.469 0.008 4.990 5.008 0.017 5.026 0.018 Dependency ratio 0.915 0.866 -0.049** 0.865 -0.001 0.975 0.933 -0.042*** 0.939 0.006 Female head (%) 0.262 0.244 -0.017 0.244 -0.001 0.262 0.256 -0.007* 0.250 -0.006 Age of head 41.962 41.042 -0.92** 41.049 0.007 44.911 45.234 0.324 45.246 0.012 Religion of head Orthodox 0.485 0.522 0.038 0.528 0.005 Muslim 0.284 0.266 -0.017 0.268 0.002 0.828 0.840 0.012 0.838 -0.002 Christian 0.217 0.201 -0.016 0.194 -0.007 0.133 0.123 -0.011 0.124 0.001 Marital status of head Married 0.740 0.742 0.002 0.741 0.000 0.744 0.746 0.002 0.751 0.004 Formerly married 0.175 0.170 -0.005 0.171 0.000 0.232 0.228 -0.004 0.228 0.000 Never married 0.085 0.088 0.003 0.088 0.000 0.023 0.024 0.001 0.020 -0.005 Education level of head Head Literate 0.586 0.640 0.054*** 0.643 0.003 0.944 0.953 0.008*** 0.952 -0.001 None 0.456 0.397 -0.059*** 0.395 -0.002 0.591 0.545 -0.046*** 0.547 0.002 Primary 0.310 0.322 0.012 0.324 0.002 0.251 0.260 0.009* 0.262 0.002 Secondary 0.129 0.151 0.023*** 0.153 0.001 0.111 0.136 0.025*** 0.138 0.001 Tertiary 0.105 0.129 0.024*** 0.128 -0.001 0.047 0.059 0.012* 0.054 -0.005 Working status of head Any 0.724 0.722 -0.002 0.723 0.001 0.829 0.842 0.012 0.841 -0.001 Wage 0.165 0.194 0.029*** 0.192 -0.002 0.198 0.228 0.03*** 0.225 -0.003 Agriculture 0.492 0.449 -0.043*** 0.450 0.001 0.445 0.435 -0.010 0.439 0.004 Nonfarm enterprise 0.136 0.147 0.011* 0.147 0.000 0.214 0.219 0.005 0.217 -0.002 Ganyu 0.205 0.185 -0.02** 0.187 0.002 †The results of an adjusted Wald test comparing weighted means are presented (*** p<0.01, ** p<0.05, * p<0.10). 21 Table 4.3a (continued): Non-response bias in HFPS Ethiopia Malawi Sample Contacted Interviewed Sample Contacted Interviewed Bias due to Bias due to non- Bias due to Bias due to non- Mean Mean non-contact† Mean interview† Mean Mean non-contact† Mean interview† (2) - (1) (4) - (2) (7) - (6) (9) - (7) Characteristic (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) Consumption Mean expenditure (LCU per capita) 16,247 17,200 953.183*** 17,186 (14) Quintilesa Quintile 1 - poorest 0.166 0.134 -0.032*** 0.133 -0.001 0.106 0.081 -0.025*** 0.082 0.001 Quintile 2 - poorer 0.186 0.164 -0.022** 0.164 0.000 0.161 0.139 -0.022*** 0.141 0.002 Quintile 3 - middle 0.207 0.209 0.003 0.210 0.001 0.208 0.189 -0.018 0.191 0.002 Quintile 4 - richer 0.214 0.241 0.028** 0.242 0.001 0.250 0.265 0.015*** 0.266 0.001 Quintile 5 - richest 0.228 0.251 0.023** 0.251 -0.001 0.275 0.326 0.051*** 0.320 -0.006 Housing Improved roof 0.739 0.792 0.053*** 0.798 0.006 0.681 0.725 0.044*** 0.722 -0.003 Improved floor 0.228 0.264 0.036*** 0.264 0.000 0.421 0.478 0.057*** 0.472 -0.006 Improved toilet 0.367 0.394 0.028** 0.398 0.004 0.056 0.064 0.008 0.063 0.000 Improved water source 0.785 0.824 0.039** 0.828 0.004 0.852 0.838 -0.014 0.837 -0.001 Access to electricity 0.634 0.702 0.069*** 0.707 0.005 0.207 0.248 0.041*** 0.242 -0.006 Asset ownership Solar 0.148 0.151 0.003 0.154 0.003 0.252 0.262 0.01** 0.265 0.002 Generator TV 0.234 0.289 0.055*** 0.291 0.002 0.208 0.247 0.038*** 0.241 -0.005 Refrigerator 0.095 0.118 0.023*** 0.118 0.000 0.112 0.136 0.023** 0.131 -0.004 Bicycle 0.016 0.022 0.006 0.022 0.000 0.453 0.476 0.023* 0.479 0.003 Motorbike 0.012 0.011 -0.001 0.011 0.000 0.061 0.073 0.012** 0.073 0.000 Car or other vehicle 0.012 0.014 0.001 0.014 0.000 0.034 0.043 0.009 0.043 0.000 Mobile Phone 0.706 0.793 0.086*** 0.797 0.004 0.784 0.852 0.068*** 0.850 -0.002 Other Activities Owns land 0.779 0.754 -0.026*** 0.752 -0.001 0.854 0.851 -0.004 0.857 0.006 Crop farming 0.611 0.570 -0.04** 0.572 0.002 0.755 0.742 -0.013** 0.748 0.006 Owns any livestock 0.636 0.602 -0.035** 0.600 -0.001 0.498 0.490 -0.007 0.494 0.004 Operates nonfarm enterprise 0.259 0.279 0.02** 0.280 0.001 0.459 0.481 0.022** 0.482 -0.003 Any member has financial account 0.547 0.620 0.073*** 0.624 0.004 0.419 0.461 0.042*** 0.458 0.002 Income from remittance (int'l & domestic) 0.161 0.150 -0.011 0.151 0.001 0.532 0.539 0.007 0.541 0.003 aForMalawi, the quintiles are based on a wealth index, not consumption expenditures. †The results of an adjusted Wald test comparing weighted means are presented (*** p<0.01, ** p<0.05, * p<0.10). 22 Table 4.3a (continued): Non-response bias in HFPS Nigeria Uganda Sample Contacted Interviewed Sample Contacted Interviewed Bias due to Bias due to non- Bias due to Bias due to non- Mean Mean non-contact† Mean interview† Mean Mean non-contact† Mean interview† (12) - (11) (14) - (12) (17) - (16) (19) - (17) Characteristic (11) (12) (13) (14) (15) (16) (17) (18) (19) (20) Sample size (number of households) 3,000 2,057 1,950 2,421 2,274 2,259 Rural 0.687 0.629 -0.058 0.626 -0.003 0.659 0.661 0.002 0.662 0.002 Average household size 5.530 5.731 0.201 5.749 0.018 4.613 4.627 0.014 4.647 0.020 Dependency ratio 1.028 1.023 -0.005 1.023 0.000 1.062 1.047 -0.015 1.046 -0.001 Female head (%) 0.186 0.180 -0.006 0.178 -0.002 0.332 0.325 -0.006 0.324 -0.002 Age of head 48.646 48.830 0.184 48.706 -0.124 45.133 45.314 0.182 45.334 0.019 Religion of head Orthodox Muslim 0.425 0.407 -0.018 0.414 0.007 Christian 0.556 0.576 0.020 0.570 -0.007 Marital status of head Married 0.749 0.765 0.016 0.766 0.001 0.738 0.739 0.002 0.743 0.004 Formerly married 0.192 0.177 -0.015 0.174 -0.003 0.227 0.223 -0.004 0.220 -0.003 Never married 0.058 0.058 0.000 0.060 0.002 0.036 0.038 0.002 0.037 -0.001 Education level of head Head Literate 0.748 0.801 0.052*** 0.803 0.002 0.809 0.819 0.01*** 0.821 0.002 None 0.364 0.295 -0.069** 0.291 -0.003 0.076 0.076 0.000 0.077 0.001 Primary 0.253 0.260 0.006 0.260 0.001 0.555 0.541 -0.013* 0.540 -0.001 Secondary 0.231 0.256 0.025 0.256 0.000 0.307 0.313 0.006 0.315 0.002 Tertiary 0.152 0.190 0.038* 0.192 0.003 0.132 0.139 0.007 0.138 -0.001 Working status of head Any 0.830 0.838 0.009 0.840 0.001 0.865 0.866 0.002 0.869 0.003 Wage 0.186 0.213 0.027 0.217 0.004 0.326 0.330 0.004 0.330 0.000 Agriculture 0.430 0.376 -0.053* 0.375 -0.002 0.474 0.466 -0.008 0.470 0.005 Nonfarm enterprise 0.375 0.413 0.039 0.412 -0.001 0.288 0.295 0.007 0.296 0.001 Ganyu †The results of an adjusted Wald test comparing weighted means are presented (*** p<0.01, ** p<0.05, * p<0.10). 23 Table 4.3a (continued): Non-response bias in HFPS Nigeria Uganda Sample Contacted Interviewed Sample Contacted Interviewed Bias due to Bias due to non- Bias due to Bias due to non- Mean Mean non-contact† Mean interview† Mean Mean non-contact† Mean interview† (12) - (11) (14) - (12) (17) - (16) (19) - (17) Characteristic (11) (12) (13) (14) (15) (16) (17) (18) (19) (20) Consumption Mean expenditure (LCU per capita) 219,408 220,234 11422 232,707 913 399,209 403,870 4661.465** 404,428 557.44 Quintiles Quintile 1 - poorest 0.201 0.165 -0.036* 0.164 -0.001 0.135 0.128 -0.006** 0.130 0.001 Quintile 2 - poorer 0.196 0.187 -0.008 0.187 0.000 0.164 0.163 -0.001 0.163 0.000 Quintile 3 - middle 0.201 0.200 -0.001 0.200 0.000 0.196 0.191 -0.005 0.190 -0.001 Quintile 4 - richer 0.197 0.221 0.025 0.219 -0.002 0.226 0.232 0.006 0.234 0.002 Quintile 5 - richest 0.206 0.227 0.020 0.230 0.004 0.269 0.276 0.007 0.273 -0.002 Housing Improved roof 0.884 0.912 0.028 0.911 -0.001 0.833 0.846 0.013*** 0.844 -0.002 Improved floor 0.777 0.830 0.052** 0.828 -0.001 0.512 0.527 0.015*** 0.522 -0.005 Improved toilet 0.609 0.687 0.078** 0.692 0.004 0.435 0.443 0.008** 0.441 -0.002 Improved water source 0.764 0.820 0.056** 0.821 0.001 0.802 0.802 0.000 0.800 -0.002 Access to electricity 0.565 0.650 0.086** 0.651 0.000 0.288 0.297 0.009** 0.297 -0.001 Asset ownership Solar 0.425 0.439 0.013* 0.441 0.002 Generator 0.237 0.282 0.045** 0.283 0.001 TV 0.462 0.536 0.074** 0.540 0.003 0.291 0.300 0.008** 0.299 -0.001 Refrigerator 0.178 0.221 0.043** 0.223 0.002 0.076 0.079 0.003 0.077 -0.002 Bicycle 0.151 0.143 -0.008 0.144 0.001 0.275 0.279 0.005 0.282 0.002 Motorbike 0.310 0.335 0.025 0.336 0.001 0.126 0.126 0.000 0.126 0.000 Car or other vehicle 0.094 0.116 0.022 0.115 0.000 0.041 0.043 0.002 0.042 -0.001 Mobile Phone 0.752 0.813 0.061** 0.810 -0.002 0.872 0.883 0.011*** 0.888 0.005 Land & crop farming Owns land 0.666 0.642 -0.023 0.643 0.001 0.598 0.597 -0.002 0.603 0.006 Crop farming 0.696 0.657 -0.039 0.656 -0.001 0.712 0.707 -0.005 0.709 0.002 Owns any livestock 0.464 0.451 -0.013 0.453 0.002 0.557 0.555 -0.002 0.560 0.004 Operates nonfarm enterprise 0.602 0.647 0.045* 0.652 0.005 0.023 0.024 0.001 0.024 0.001 Any member has financial account 0.500 0.591 0.091*** 0.591 0.000 0.643 0.658 0.015*** 0.659 0.000 Income from remittance (int'l & domestic) 0.308 0.330 0.022 0.330 0.000 0.347 0.348 0.002 0.348 0.000 †The results of an adjusted Wald test comparing weighted means are presented (*** p<0.01, ** p<0.05, * p<0.10). 24 Table 4.3b: Non-response bias in Ethiopia HFPS by Urban-Rural domain Ethiopia HFPS - Urban Ethiopia HFPS - Rural Sample Contacted Interviewed Sample Contacted Interviewed Bias due to Bias due to Bias due Bias due to non- non- to non- non- Mean Mean Mean Mean Mean Mean contact† interview† contact† interview† (2) - (1) (4) - (2) (7) - (6) (9) - (7) Characteristic (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) Sample size (number of households) 3,410 2,340 2,270 1,962 1,013 977 Average household size 3.651 3.696 0.045 3.695 -0.001 5.067 5.131 0.063 5.142 0.011 Dependency ratio 0.635 0.618 -0.017 0.615 -0.003 1.113 1.085 -0.028 1.083 -0.002 Female head (%) 0.339 0.320 -0.019 0.324 0.005 0.208 0.178 -0.029** 0.174 -0.005 Age of head 38.584 38.422 -0.162 38.359 -0.063 44.332 43.337 -0.995 43.387 0.050 Religion of head Orthodox 0.579 0.604 0.024 0.607 0.004 0.418 0.451 0.033 0.458 0.007 Muslim 0.201 0.178 -0.023 0.178 -0.001 0.341 0.344 0.002 0.347 0.004 Christian 0.214 0.214 0.000 0.210 -0.003 0.220 0.190 -0.029 0.180 -0.011 Marital status of head Married 0.621 0.646 0.024 0.641 -0.004 0.824 0.826 0.002 0.828 0.002 Formerly married 0.214 0.207 -0.007 0.209 0.002 0.148 0.139 -0.009* 0.137 -0.001 Never married 0.164 0.148 -0.017 0.150 0.002 0.028 0.035 0.007 0.035 -0.001 Education level of head Head Literate 0.775 0.809 0.034* 0.813 0.003 0.454 0.492 0.038*** 0.495 0.003 None 0.263 0.228 -0.035* 0.225 -0.003 0.592 0.545 -0.047*** 0.542 -0.003 Primary 0.301 0.302 0.002 0.306 0.004 0.317 0.340 0.023 0.340 0.000 Secondary 0.226 0.241 0.015 0.243 0.002 0.060 0.073 0.013 0.074 0.001 Tertiary 0.210 0.228 0.018 0.226 -0.003 0.031 0.042 0.011* 0.043 0.001 Working status of head Any 0.594 0.606 0.012 0.605 -0.001 0.816 0.825 0.009 0.827 0.002 Wage 0.310 0.326 0.016 0.325 -0.001 0.063 0.078 0.016 0.076 -0.002 Agriculture 0.115 0.108 -0.007 0.105 -0.003 0.756 0.747 -0.009 0.749 0.002 Nonfarm enterprise 0.220 0.220 0.000 0.220 0.001 0.077 0.083 0.006 0.083 0.000 †The results of an adjusted Wald test comparing weighted means are presented (*** p<0.01, ** p<0.05, * p<0.10). 25 Table 4.3b: Non-response bias in Ethiopia HFPS by Urban-Rural domain Ethiopia HFPS - Urban Ethiopia HFPS - Rural Sample Contacted Interviewed Sample Contacted Interviewed Bias due to Bias due to Bias due to Bias due to non- non- Mean Mean non-contact† Mean non-interview† Mean Mean Mean contact† interview† (2) - (1) (4) - (2) (7) - (6) (9) - (7) Characteristic (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) Consumption Mean expenditure (LCU per capita) 23087.555 22959.014 -128.540 22976.492 17.478 11448.641 12156.554 707.913 12155.499 -1.055 Quintiles Quintile 1 - poorest 0.053 0.044 -0.009 0.041 -0.002 0.245 0.213 -0.032 0.213 0.000 Quintile 2 - poorer 0.100 0.095 -0.005 0.096 0.001 0.245 0.224 -0.021 0.224 -0.001 Quintile 3 - middle 0.159 0.155 -0.004 0.158 0.003 0.240 0.256 0.017 0.254 -0.002 Quintile 4 - richer 0.277 0.300 0.023 0.299 -0.002 0.170 0.190 0.020 0.194 0.004 Quintile 5 - richest 0.411 0.406 -0.005 0.406 0.000 0.100 0.116 0.016 0.115 -0.001 Housing Improved roof 0.961 0.971 0.010 0.976 0.004 0.583 0.635 0.052 0.643 0.008 Improved floor 0.497 0.512 0.015 0.513 0.002 0.040 0.047 0.007 0.048 0.001 Improved toilet 0.606 0.619 0.013 0.625 0.006 0.199 0.197 -0.002 0.201 0.003 Improved water source 0.978 0.985 0.006 0.988 0.003 0.650 0.684 0.034 0.689 0.006 Access to electricity 0.919 0.939 0.020 0.944 0.005 0.433 0.495 0.062** 0.502 0.006 Asset ownership Solar 0.051 0.045 -0.005 0.046 0.001 0.217 0.244 0.027 0.247 0.004 Generator TV 0.525 0.573 0.048* 0.578 0.005 0.030 0.041 0.011* 0.042 0.001 Refrigerator 0.223 0.244 0.021 0.245 0.000 0.006 0.008 0.002 0.008 0.000 Bicycle 0.033 0.043 0.009 0.044 0.001 0.003 0.003 0.000 0.003 0.000 Motorbike 0.020 0.019 -0.001 0.020 0.001 0.007 0.004 -0.002 0.004 0.000 Car or other vehicle 0.030 0.029 -0.001 0.029 0.000 0.000 0.000 0.000 0.000 0.000 Mobile Phone 0.904 0.949 0.045*** 0.952 0.003 0.568 0.656 0.088*** 0.662 0.006 Other Activities Owns land 0.957 0.947 -0.010 0.947 0.001 Crop farming 0.899 0.901 0.002 0.904 0.003 Operates nonfarm enterprise 0.381 0.381 0.001 0.383 0.002 0.174 0.190 0.016 0.191 0.002 Any member has financial account 0.799 0.846 0.047*** 0.850 0.003 0.370 0.422 0.052*** 0.429 0.006 Income from remittance (int'l & domestic) 0.213 0.195 -0.018 0.198 0.003 0.124 0.111 -0.013 0.111 0.000 †The results of an adjusted Wald test comparing weighted means are presented (*** p<0.01, ** p<0.05, * p<0.10). 26 4.1.4 Overall bias The results so far have examined the separate sources of bias introduced at each stage of the phone survey selection process. Column 3, 6, 9, and 12 in Table 4.4a present the cumulative bias for the Ethiopia, Malawi, Nigeria, and Uganda HFPS interviewed samples. It is clear that the successfully interviewed HFPS samples in all four countries are biased across many different household characteristics. Overall, the bias is skewed towards urban households (except in Uganda) as well as households that are relatively better off in terms of material well-being. Household heads that are successfully interviewed are better educated and more likely to working in formal (wage) employment compared to the overall LSMS-ISA sample. These results are mimicked in Table 4.4b when disaggregating the Ethiopia sample into the urban and rural components. 4.2 Testing Effectiveness of Weighting Adjustments The results above have clearly demonstrated the bias present in the sample of successfully interviewed households in the HFPS in the four countries. This bias would compromise the external validity of the results obtained from the HFPS which would not be fully representative of the general population. Furthermore, the bias in all four countries skews towards households that are generally better off and therefore would not accurately reflect the situation for poorer households, a segment of the population that is likely most vulnerable to the COVID-19 crisis. However, since the samples for these two surveys are derived from representative samples in the ESS, IHPS, GHS-Panel and UNPS, there is potential to reduce the bias by harnessing the detailed information available in the F2F surveys to conduct the weighting adjustments described in Section 3.1.2. Implementing these weighting adjustments will likely counteract the bias but will not fully eliminate it. This section presents results from an assessment of the effectiveness of the adjustments performed on the HFPS in each country. Table 4.5a presents three different characteristic profiles representing: (i) the full representative LSMS- ISA samples applying usual survey weights (columns 1, 6, 11, and 16) , (ii) the HFPS samples applying weights that include the bias reweighting adjustment (columns 2, 7, 12, and 17), and (iii) the HFPS samples applying weights that do not include the bias reweighting adjustment (columns 4, 9, 14, and 18). Comparing the relative gaps between the representative profile in (i) with the profile of characteristics obtained in (ii) and (iii) will provide an indication both of the effectiveness of the bias adjustment as well as the extent of bias that remains after the adjustment. The difference between the profile of characteristics in the representative sample and the HFPS sample using the adjusted and unadjusted weights was estimated and presented in Table 4.5a. Examining first the difference between the representative sample and the HFPS sample using the unadjusted weights (columns 5, 10, 15, and 20), in Ethiopia and Malawi there is substantial bias across the majority of characteristics. For Nigeria and Uganda, the bias is still present, though across fewer different indicators than in Ethiopia and Malawi. The results here using the HFPS unadjusted weights largely mimic the results from above (using the LSMS-ISA weights) indicating that there is substantial bias. 27 Table 4.4a. Overall bias in HFPS Ethiopia Malawi Nigeria Uganda HFPS HFPS HFPS HFPS ESS IHPS GHS-P UNPS Interviewed Interviewed Interviewed Interviewed Overall Overall Overall Overall Mean Mean bias† Mean Mean bias† Mean Mean bias† Mean Mean bias† (2) - (1) (5) - (4) (8) - (7) (11) - (10) Characteristic (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) Sample size (number of households) 6,788 3,247 3,181 1,729 4,976 1,950 3,308 2,259 Rural 0.675 0.535 -0.14*** 0.826 0.749 -0.077* 0.680 0.613 -0.067*** 0.684 0.662 -0.022 Average household size 4.483 4.469 -0.014 4.904 5.026 0.121 5.332 5.520 0.188 4.515 4.647 0.132 Dependency ratio 0.955 0.865 -0.091** 1.055 0.939 -0.115*** 0.980 0.984 0.004 1.064 1.046 -0.018 Female head (%) 0.263 0.244 -0.019 0.296 0.250 -0.046** 0.201 0.191 -0.011 0.342 0.324 -0.019 Age of head 42.912 41.049 -1.863** 45.855 45.246 -0.609 49.757 49.402 -0.355 45.311 45.334 0.023 Religion of head Orthodox 0.478 0.528 0.050 Muslim 0.293 0.268 -0.025 0.808 0.838 0.030 0.397 0.385 -0.012 0.000 0.000 0.000 Christian 0.211 0.194 -0.017 0.145 0.124 -0.021 0.585 0.599 0.014 0.000 0.000 0.000 Marital status of head Married 0.739 0.741 0.002 0.716 0.751 0.035 0.740 0.756 0.016 0.713 0.743 0.03* Formerly married 0.189 0.171 -0.018 0.265 0.228 -0.036 0.209 0.188 -0.021 0.246 0.220 -0.026 Never married 0.072 0.088 0.016 0.018 0.020 0.001 0.051 0.056 0.005 0.041 0.037 -0.004 Education level of head Head Literate 0.520 0.643 0.123*** 0.921 0.952 0.031*** 0.728 0.794 0.066*** 0.772 0.821 0.049*** None 0.523 0.395 -0.128*** 0.675 0.547 -0.129*** 0.365 0.275 -0.09*** 0.087 0.077 -0.010 Primary 0.296 0.324 0.029 0.207 0.262 0.055* 0.246 0.252 0.005 0.584 0.540 -0.044** Secondary 0.102 0.153 0.051*** 0.083 0.138 0.054*** 0.221 0.256 0.035** 0.290 0.315 0.025 Tertiary 0.079 0.128 0.048*** 0.034 0.054 0.020 0.167 0.217 0.05*** 0.120 0.138 0.018 Working status of head Any 0.735 0.723 -0.012 0.819 0.841 0.022 0.808 0.828 0.019 0.855 0.869 0.014 Wage 0.133 0.192 0.059*** 0.158 0.225 0.067** 0.179 0.219 0.04*** 0.324 0.330 0.006 Agriculture 0.551 0.450 -0.101*** 0.469 0.439 -0.030 0.447 0.391 -0.055*** 0.479 0.470 -0.009 Nonfarm enterprise 0.115 0.147 0.032* 0.193 0.217 0.023 0.329 0.366 0.037** 0.266 0.296 0.031 Ganyu 0.248 0.187 -0.061** †The results of an adjusted Wald test comparing weighted means are presented (*** p<0.01, ** p<0.05, * p<0.10). 28 Table 4.4a (continued). Overall bias in HFPS Ethiopia Malawi Nigeria Uganda HFPS HFPS HFPS HFPS ESS IHPS GHS-P UNPS Interviewed Interviewed Interviewed Interviewed Overall Overall Overall bias† Overall bias† Mean Mean Mean Mean bias† Mean Mean bias† Mean Mean (2) - (1) (11) - (10) (5) - (4) (8) - (7) Characteristic (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) Consumption Mean expenditure (LCU per capita) 14,943 17,186 2242.546*** 221123 236970 15847** 373,830 404,428 30598.098** Quintilesa Quintile 1 - poorest 0.200 0.133 -0.067*** 0.201 0.082 -0.12*** 0.205 0.164 -0.041*** 0.156 0.130 -0.026* Quintile 2 - poorer 0.200 0.164 -0.036** 0.199 0.141 -0.058*** 0.210 0.196 -0.014 0.174 0.163 -0.011 Quintile 3 - middle 0.200 0.210 0.009 0.200 0.191 -0.009 0.191 0.197 0.006 0.195 0.190 -0.005 Quintile 4 - richer 0.200 0.242 0.042*** 0.200 0.266 0.066*** 0.195 0.210 0.015 0.221 0.234 0.013 Quintile 5 - richest 0.200 0.251 0.051** 0.200 0.320 0.12*** 0.200 0.234 0.034** 0.245 0.273 0.028 Housing Improved roof 0.689 0.798 0.109*** 0.589 0.722 0.133*** 0.881 0.916 0.035*** 0.797 0.844 0.047*** Improved floor 0.177 0.264 0.088*** 0.333 0.472 0.139*** 0.768 0.825 0.057*** 0.466 0.522 0.056*** Improved toilet 0.313 0.398 0.085** 0.043 0.063 0.021 0.581 0.673 0.092*** 0.400 0.441 0.041** Improved water source 0.747 0.828 0.081** 0.844 0.837 -0.007 0.761 0.817 0.056*** 0.787 0.800 0.014 Access to electricity 0.559 0.707 0.148*** 0.150 0.242 0.092*** 0.545 0.642 0.096*** 0.257 0.297 0.04** Asset ownership Solar 0.153 0.154 0.001 0.207 0.265 0.057** 0.404 0.441 0.037* Generator 0.262 0.324 0.062*** TV 0.178 0.291 0.113*** 0.152 0.241 0.089*** 0.455 0.553 0.098*** 0.257 0.299 0.042** Refrigerator 0.072 0.118 0.046*** 0.080 0.131 0.051** 0.180 0.234 0.054*** 0.071 0.077 0.006 Bicycle 0.012 0.022 0.010 0.416 0.479 0.063* 0.147 0.138 -0.009 0.261 0.282 0.021 Motorbike 0.009 0.011 0.002 0.044 0.073 0.029** 0.302 0.333 0.031* 0.114 0.126 0.012 Car or other vehicle 0.009 0.014 0.004 0.025 0.043 0.018 0.098 0.125 0.026** 0.039 0.042 0.003 Mobile Phone 0.553 0.797 0.243*** 0.582 0.850 0.267*** 0.769 0.839 0.07*** 0.796 0.888 0.092*** Other activities Owns land 0.824 0.752 -0.072*** 0.890 0.857 -0.033 0.665 0.636 -0.029* 0.608 0.603 -0.006 Crop farming 0.658 0.572 -0.086** 0.812 0.748 -0.064* 0.695 0.656 -0.039** 0.725 0.709 -0.016 Owns any livestock 0.684 0.600 -0.083** 0.476 0.494 0.019 0.479 0.454 -0.025 0.559 0.560 0.000 Operates nonfarm enterprise 0.229 0.280 0.051** 0.409 0.482 0.073** 0.556 0.607 0.051*** 0.023 0.024 0.001 Any member has financial account 0.457 0.624 0.167*** 0.339 0.458 0.118*** 0.502 0.609 0.107*** 0.595 0.659 0.064*** Income from remittance (int'l & domestic) 0.145 0.151 0.006 0.528 0.541 0.013 0.327 0.334 0.008 0.340 0.348 0.008 aForMalawi, the quintiles are based on a wealth index, not consumption expenditures. †The results of an adjusted Wald test comparing weighted means are presented (*** p<0.01, ** p<0.05, * p<0.10). 29 Table 4.4b: Overall bias in Ethiopia HFPS by Urban-Rural domain Ethiopia- Urban Ethiopia- Rural ESS HFPS ESS HFPS Urban Interviewed Rural Interviewed Overall Overall Mean Mean bias† Mean Mean bias† (2) - (1) (5) - (4) Characteristic (1) (2) (3) (4) (5) (6) Sample size (number of households) 3,655 2,270 3,115 1,013 Average household size 3.647 3.695 0.048 4.885 5.131 0.063 Dependency ratio 0.651 0.615 -0.036 1.103 1.085 -0.028 Female head (%) 0.346 0.324 -0.022 0.223 0.178 -0.029** Age of head 38.728 38.359 -0.369 44.924 43.337 -0.995 Religion of head Orthodox 0.574 0.607 0.034 0.431 0.458 0.027 Muslim 0.207 0.178 -0.030 0.335 0.347 0.012 Christian 0.213 0.210 -0.003 0.210 0.180 -0.030 Marital status of head Married 0.618 0.641 0.023 0.798 0.828 0.031 Formerly married 0.220 0.209 -0.011 0.174 0.137 -0.037* Never married 0.162 0.150 -0.012 0.029 0.035 0.006 Education level of head Head Literate 0.753 0.813 0.06** 0.408 0.495 0.088*** None 0.287 0.225 -0.061** 0.637 0.542 -0.094*** Primary 0.296 0.306 0.010 0.296 0.340 0.045 Secondary 0.215 0.243 0.028 0.047 0.074 0.027 Tertiary 0.202 0.226 0.023 0.020 0.043 0.022** Working status of head Any 0.592 0.605 0.013 0.804 0.827 0.022 Wage 0.302 0.325 0.023 0.051 0.076 0.025 Agriculture 0.124 0.105 -0.019 0.756 0.749 -0.007 Nonfarm enterprise 0.219 0.220 0.001 0.066 0.083 0.017 Consumption Mean expenditure (LCU per capita) 22875.803 22976.492 100.690 11130.133 12155.499 1025.366 Quintiles Quintile 1 - poorest 0.059 0.041 -0.018 0.268 0.213 -0.055 Quintile 2 - poorer 0.107 0.096 -0.011 0.245 0.224 -0.021 Quintile 3 - middle 0.158 0.158 0.001 0.221 0.254 0.034 Quintile 4 - richer 0.274 0.299 0.024 0.164 0.194 0.029 Quintile 5 - richest 0.402 0.406 0.004 0.103 0.115 0.013 †The results of an adjusted Wald test comparing weighted means are presented (*** p<0.01, ** p<0.05, * p<0.10). 30 Table 4.4b (continued): Overall bias in Ethiopia HFPS by Urban-Rural domain Ethiopia- Urban Ethiopia- Rural ESS HFPS ESS HFPS Urban Interviewed Rural Interviewed Overall Overall Mean Mean bias† Mean Mean bias† (2) - (1) (5) - (4) Characteristic (1) (2) (3) (4) (5) (6) Housing Improved roof 0.954 0.976 0.022 0.562 0.643 0.081* Improved floor 0.482 0.513 0.032 0.030 0.048 0.018 Improved toilet 0.593 0.625 0.032 0.179 0.201 0.022 Improved water source 0.976 0.988 0.011 0.637 0.689 0.053 Access to electricity 0.905 0.944 0.039* 0.393 0.502 0.108** Asset ownership Solar 0.051 0.046 -0.005 0.202 0.247 0.045 Generator TV 0.505 0.578 0.073* 0.021 0.042 0.021* Refrigerator 0.213 0.245 0.032 0.004 0.008 0.004 Bicycle 0.032 0.044 0.012 0.002 0.003 0.001 Motorbike 0.019 0.020 0.001 0.004 0.004 0.000 Car or other vehicle 0.028 0.029 0.001 0.000 0.000 0.000 Mobile Phone 0.866 0.952 0.086*** 0.403 0.662 0.258*** Other activities Owns land 0.962 0.947 -0.015 Crop farming 0.876 0.904 0.028 Operates nonfarm enterprise 0.380 0.383 0.003 0.157 0.191 0.034 Any member has financial account 0.773 0.850 0.076*** 0.305 0.429 0.124*** Income from remittance (int'l & domestic) 0.215 0.198 -0.017 0.111 0.111 -0.001 †The results of an adjusted Wald test comparing weighted means are presented (*** p<0.01, ** p<0.05, * p<0.10). Looking at the results in columns 3, 8, 13, and 18 which present the difference between the F2F sample and the HFPS sample using the HFPS weight with the bias adjustment, there is a substantial improvement in the bias. For Nigeria (column 13) there are zero indicators which are significantly different from the GHS-Panel sample, suggesting that the bias correction using the calibration approach has been very effective at reducing the bias. In Ethiopia, Malawi, and Uganda, there are still some indications of bias, however the bias has been substantially reduced following the weighting adjustments. Only a handful of the characteristics that were significantly biased when using the unadjusted weights are also biased when using the adjusted weights. For those characteristics that do still exhibit bias after applying the adjustment, the bias is reduced in all cases. This illustrates the effectiveness of the bias reduction methods employed in the HFPS in all four countries, but also highlights that these adjustments do not fully counteract the bias, particularly in the Ethiopia, Malawi, and Uganda HFPS. 31 The results in Table 4.5a have provided evidence that the bias adjustment methods employed in the HFPS in the four countries have been effective at reducing the bias present in the interviewed sample. The benefits can be further illustrated by examining not just the point estimates for the considered characteristics but also the confidence intervals. Figures 4.2 to 4.8 plot the point estimates and 95 percent confidence intervals for selected characteristics for the full LSMS-ISA (F2F) sample, HFPS sample with adjusted weights, and HFPS sample with unadjusted weights. In the figures, all estimates are standardized by subtracting the F2F survey mean (so the F2F mean will always be zero) in order to allow comparison across indicators. For household size and head age, the estimates were further standardized by dividing by the F2F mean. Thus, the confidence intervals represent the percent deviation from the mean for these two variables. Looking through the figures, it is clear that the adjustments for bias result in substantially improved alignment with the LSMS-ISA survey point estimates and confidence intervals. Although the level of reduction in the bias achieved varies across indicators, the adjustment for nearly all indicators does result in a shift of HFPS estimates towards the LSMS-ISA. For many indicators, the reduction is the bias from the adjustments is substantial, for example literacy in Figure 4.2. As expected, the bias adjustment was especially effective for the indicators that were included in the bias adjustments in each country (listed in Table 3.2). However, for other indicators, the reduction in bias is more modest. For example, the housing characteristics in Figure 4.5 show substantial bias without the adjustment to the weights and the bias is significantly reduced after applying the adjustments, but there still remains a gap between the point estimates for the F2F and HFPS samples (though not statistically significant). Comparing the results for the four countries, there is remarkable consistency in the direction of the bias across the indicators, though the magnitude of the bias varies. The bias is always towards households that are better off in terms of consumption expenditures and general living standards. This is especially evident in Figure 4.4 for consumption expenditure quintiles where the bias follows a linear pattern with the largest bias among the poorest and richest households and in opposing directions (negatively biased for the poorest and positively biased for the richest). In this case the bias adjustment is quite effective in all four countries (and was included in the adjustment model). While Figures 4.2 to 4.8 illustrate the effectiveness of bias adjustments for dichotomous variables, Figure 4.9 goes a step further to show how effective the adjustments are for continuous welfare measures: consumption expenditures for Ethiopia, Nigeria and Uganda and wealth index for Malawi. Comparing the distribution for the full LSMS-ISA sample with that of the HFPS without the bias adjustments, the modest bias can be seen in all four countries, particularly in Ethiopia and Malawi. However, the HFPS sample distribution with the bias weighting adjustments aligns much more closely with the LSMS-ISA distribution, demonstrating the effectiveness of the bias corrections across the distribution of continuous measures. Although the bias adjustments have broadly been shown to substantially reduce bias in the estimated mean of the examined indicators, one concern is that the adjustments to weights will result in a higher variance in the weights and thus larger standard errors. Large differences between the LSMS-ISA and HFPS interviewed sample could result in extreme outlier weights following the weighting adjustments. Larger standard errors would compromise the precision of the estimates and, if the standard errors are too large, will limit the usefulness of the data. In order to counteract this potential, an additional step (outlined in Section 3.1.2) is implemented in the response propensity correction factors by sorting into deciles according to the correction factor value and applying the mean correction factor within each decile. The impact of the adjustments on standard errors can be observed in Figures 4.2 to 4.8 by comparing the width of the confidence intervals of the HFPS sample estimates with and without the bias weighting 32 adjustments. Although some expansion of the width of the confidence intervals can be observed, in all cases the difference is minor or imperceptible. Therefore, it appears the bias corrections implemented in the four countries has not come at the cost of substantially larger standard errors. Overall, the results from this analysis have demonstrated the effectiveness of the weight adjustment techniques employed in the Ethiopia, Malawi, Nigeria, and Uganda HFPS for reducing bias in the interviewed samples. Absent these adjustments, the phone survey samples would suffer from substantial bias that would compromise the representativeness of the results obtained from the phone survey sample. The reduction in the bias from these methods highlights the distinct advantage of phone surveys taken from representative F2F household survey samples. Although difficult to effectively simulate here, the adjustments possible for RDD and telecom list-based phone surveys would likely be less effective at reducing bias than adjustments made using the rich information from the F2F representative surveys used in these four countries. Therefore, it might be expected that effectiveness of bias adjustments in RDD and telecom list-based surveys to fall between the unadjusted and adjusted corrections considered here. The effectiveness of adjustments under these other phone survey sample approaches will vary depending on the availability of external information on demographic and economic characteristics of the general population and the extent to which corresponding information can be captured in the phone survey itself. 33 Table 4.5a: Effectiveness of Bias Correction Ethiopia Malawi Full ESS HFPS (n=3,247) Full IHPS HFPS (n=1,729) (n=6,774) Adjusted Weights Unadjusted Weights (n=3,181 Adjusted Weights Unadjusted Weights Difference† Difference† Difference† Difference† Mean Mean Mean Mean Mean Mean (2) - (1) (4) - (1) (7) - (6) (9) - (6) Characteristic (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) Rural 0.675 0.670 -0.006 0.528 -0.147*** 0.826 0.774 -0.015 0.797 -0.029 Average household size 4.483 4.686 0.203 4.504 0.021 4.904 4.751 0.022 5.097 0.193 Dependency ratio 0.955 0.922 -0.033 0.845 -0.111*** 1.055 1.000 -0.015 0.986 -0.069 Female head (%) 0.263 0.242 -0.021 0.246 -0.016 0.296 0.269 -0.003 0.246 -0.05** Age of head 42.912 42.132 -0.780 41.114 -1.799** 45.855 44.756 -0.488 44.887 -0.968 Religion of head Orthodox 0.478 0.490 0.013 0.513 0.035 Muslim 0.293 0.293 0.000 0.259 -0.034 0.808 0.754 -0.007 0.813 0.005 Christian 0.211 0.208 -0.003 0.220 0.009 0.145 0.090 0.007 0.147 0.002 Marital status of head Married 0.739 0.753 0.014 0.736 -0.003 0.716 0.688 0.007 0.760 0.044** Formerly married 0.189 0.178 -0.011 0.171 -0.018 0.265 0.237 -0.005 0.219 -0.045** Never married 0.072 0.068 -0.003 0.093 0.021* 0.018 0.011 -0.004 0.018 0.000 Education level of head Head Literate 0.520 0.536 0.016 0.652 0.132*** 0.921 0.907 0.008 0.959 0.038*** None 0.523 0.512 -0.011 0.384 -0.139*** 0.675 0.632 -0.023 0.573 -0.103*** Primary 0.296 0.296 0.001 0.320 0.025 0.207 0.171 0.010 0.247 0.040 Secondary 0.102 0.107 0.005 0.159 0.057*** 0.083 0.064 0.012 0.132 0.049*** Tertiary 0.079 0.084 0.005 0.137 0.058*** 0.034 0.020 0.001 0.049 0.015 Working status of head Any 0.735 0.741 0.005 0.720 -0.015 0.819 0.794 0.007 0.838 0.019 Wage 0.133 0.136 0.004 0.196 0.064*** 0.158 0.129 0.024 0.209 0.051** Agriculture 0.551 0.560 0.009 0.444 -0.107*** 0.469 0.424 -0.026 0.441 -0.029 Nonfarm enterprise 0.115 0.118 0.003 0.152 0.036** 0.193 0.168 0.006 0.225 0.032 Ganyu 0.248 0.216 -0.014 0.199 -0.05** †The results of an adjusted Wald test comparing weighted means are presented (*** p<0.01, ** p<0.05, * p<0.10). 34 Table 4.5a (continued): Effectiveness of Bias Correction Ethiopia Malawi Full ESS HFPS (n=3,247) Full IHPS HFPS (n=1,729) (n=6,774) Adjusted Weights Unadjusted Weights (n=3,181 Adjusted Weights Unadjusted Weights Difference† Difference† Difference† Difference† Mean Mean Mean Mean Mean Mean (2) - (1) (4) - (1) (7) - (6) (9) - (6) Characteristic (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) Consumption Mean expenditure (LCU per capita) 14,943 14,400 (543) 17,437 2493.996*** Quintilesa Quintile 1 - poorest 0.200 0.214 0.013 0.144 -0.056** 0.201 0.165 -0.048* 0.093 -0.109*** Quintile 2 - poorer 0.200 0.199 -0.001 0.166 -0.034** 0.199 0.177 -0.010 0.147 -0.051*** Quintile 3 - middle 0.200 0.212 0.012 0.204 0.003 0.200 0.171 0.006 0.195 -0.005 Quintile 4 - richer 0.200 0.199 -0.001 0.234 0.034** 0.200 0.174 0.033 0.273 0.073*** Quintile 5 - richest 0.200 0.176 -0.024 0.252 0.052** 0.200 0.162 0.018 0.292 0.092*** Housing Improved roof 0.689 0.728 0.039 0.795 0.106*** 0.589 0.540 0.006 0.691 0.102*** Improved floor 0.177 0.183 0.006 0.274 0.097*** 0.333 0.290 0.012 0.435 0.102*** Improved toilet 0.313 0.339 0.026 0.415 0.102*** 0.043 0.025 0.002 0.057 0.014 Improved water source 0.747 0.774 0.027 0.831 0.084*** 0.844 0.808 -0.005 0.848 0.004 Access to electricity 0.559 0.633 0.074* 0.711 0.152*** 0.150 0.112 0.013 0.208 0.058* Asset ownership Solar 0.153 0.174 0.021 0.148 -0.005 0.207 0.178 0.044* 0.291 0.084*** Generator TV 0.178 0.195 0.017 0.296 0.118*** 0.152 0.121 0.012 0.224 0.072*** Refrigerator 0.072 0.075 0.003 0.122 0.05*** 0.080 0.055 0.004 0.110 0.031 Bicycle 0.012 0.013 0.001 0.021 0.010 0.416 0.374 0.035 0.504 0.087*** Motorbike 0.009 0.009 0.000 0.013 0.004 0.044 0.032 0.012 0.075 0.031*** Car or other vehicle 0.009 0.009 -0.001 0.015 0.005* 0.025 0.013 0.004 0.040 0.015 Mobile Phone 0.553 0.736 0.183*** 0.800 0.247*** 0.582 0.544 0.178*** 0.839 0.257*** Other activities Owns land 0.824 0.827 0.003 0.753 -0.071*** 0.890 0.861 0.000 0.865 -0.025 Crop farming 0.658 0.683 0.024 0.561 -0.097*** 0.812 0.772 -0.018 0.787 -0.025 Owns any livestock 0.684 0.705 0.021 0.594 -0.089*** 0.476 0.431 0.008 0.510 0.035 Operates nonfarm enterprise 0.229 0.244 0.015 0.287 0.058** 0.409 0.372 0.028 0.484 0.075** Any member has financial account 0.457 0.545 0.088*** 0.634 0.177*** 0.339 0.300 0.007 0.422 0.083*** Income from remittance (int'l & domestic) 0.145 0.145 0.000 0.161 0.016 0.528 0.489 0.003 0.521 -0.007 aForMalawi, the quintiles are based on a wealth index, not consumption expenditures. †The results of an adjusted Wald test comparing weighted means are presented (*** p<0.01, ** p<0.05, * p<0.10). 35 Table 4.5a (continued): Effectiveness of Bias Correction Nigeria Uganda HFPS (n=1,950) Full HFPS (n=2,259) Full GHS-P UNPS (n=4,976) Adjusted Weights Unadjusted Weights Adjusted Weights Unadjusted Weights (n=3,308) Difference† Difference† Difference† Difference† Mean Mean Mean Mean Mean Mean (12) - (11) (14 - (11) (17) - (16) (19) - (16) Characteristic (11) (12) (13) (14) (15) (16) (17) (18) (19) (20) Rural 0.686 0.686 0.000 0.634 -0.051 0.684 0.685 0.001 0.672 -0.012 Average household size 5.528 5.528 0.000 5.825 0.297 4.515 4.544 0.029 4.684 0.169* Dependency ratio 1.016 1.033 0.017 1.029 0.013 1.064 1.033 -0.031 1.050 -0.014 Female head (%) 0.186 0.186 0.000 0.173 -0.014 0.342 0.330 -0.012 0.323 -0.019 Age of head 48.797 49.153 0.356 48.579 -0.218 45.311 45.370 0.059 45.484 0.173 Religion of head Orthodox Muslim 0.425 0.427 0.002 0.424 -0.001 Christian 0.557 0.555 -0.002 0.560 0.003 Marital status of head Married 0.750 0.754 0.004 0.772 0.022 0.713 0.732 0.019 0.748 0.035** Formerly married 0.193 0.189 -0.004 0.169 -0.025 0.246 0.231 -0.015 0.218 -0.028* Never married 0.057 0.058 0.001 0.059 0.002 0.041 0.037 -0.004 0.034 -0.007 Education level of head Head Literate 0.744 0.744 0.000 0.803 0.06*** 0.772 0.787 0.015 0.816 0.044*** None 0.375 0.369 -0.005 0.296 -0.078*** 0.087 0.090 0.003 0.079 -0.008 Primary 0.241 0.240 -0.001 0.256 0.015 0.584 0.585 0.000 0.549 -0.035 Secondary 0.225 0.231 0.006 0.255 0.030 0.290 0.291 0.001 0.311 0.021 Tertiary 0.160 0.160 0.000 0.193 0.033 0.120 0.117 -0.003 0.132 0.012 Working status of head Any 0.827 0.831 0.004 0.842 0.015 0.855 0.857 0.002 0.864 0.008 Wage 0.177 0.191 0.014 0.220 0.043* 0.324 0.323 -0.001 0.323 -0.001 Agriculture 0.442 0.407 -0.035 0.379 -0.063** 0.479 0.480 0.001 0.474 -0.005 Nonfarm enterprise 0.367 0.393 0.025 0.412 0.045 0.266 0.281 0.016 0.293 0.028 Ganyu †The results of an adjusted Wald test comparing weighted means are presented (*** p<0.01, ** p<0.05, * p<0.10). 36 Table 4.5a (continued): Effectiveness of Bias Correction Nigeria Uganda Full GHS-P HFPS (n=1,950) Full UNPS HFPS (n=2,259) (n=4,976) Adjusted Weights Unadjusted Weights (n=3,308) Adjusted Weights Unadjusted Weights Difference† Difference† Difference† Difference† Mean Mean Mean Mean Mean Mean (12) - (11) (14 - (11) (17) - (16) (19) - (16) Characteristic (11) (12) (13) (14) (15) (16) (17) (18) (19) (20) Consumption Mean expenditure (LCU per capita) 219408 220010 601 230199 10791 373,830 375,612 1,782 402,574 28743.854** Quintiles Quintile 1 - poorest 0.201 0.198 -0.002 0.167 -0.034* 0.156 0.155 -0.001 0.135 -0.020 Quintile 2 - poorer 0.199 0.199 0.000 0.191 -0.008 0.174 0.175 0.001 0.169 -0.005 Quintile 3 - middle 0.200 0.199 -0.001 0.200 0.000 0.195 0.196 0.001 0.194 -0.001 Quintile 4 - richer 0.200 0.199 0.000 0.217 0.017 0.221 0.223 0.002 0.231 0.010 Quintile 5 - richest 0.200 0.204 0.004 0.225 0.025 0.245 0.251 0.006 0.270 0.025 Housing Improved roof 0.886 0.886 0.000 0.907 0.021 0.797 0.806 0.009 0.821 0.024* Improved floor 0.776 0.794 0.017 0.823 0.047* 0.466 0.481 0.014 0.507 0.041** Improved toilet 0.602 0.640 0.038 0.688 0.086*** 0.400 0.404 0.004 0.426 0.026 Improved water source 0.765 0.792 0.027 0.815 0.05* 0.787 0.798 0.012 0.804 0.018 Access to electricity 0.554 0.596 0.042 0.646 0.092** 0.257 0.268 0.011 0.283 0.026 Asset ownership Solar 0.404 0.434 0.030 0.448 0.044** Generator 0.246 0.244 -0.002 0.280 0.034 TV 0.451 0.481 0.030 0.534 0.083*** 0.257 0.267 0.010 0.288 0.031 Refrigerator 0.173 0.187 0.014 0.221 0.048** 0.071 0.067 -0.004 0.074 0.002 Bicycle 0.157 0.141 -0.016 0.146 -0.011 0.261 0.289 0.028 0.298 0.037** Motorbike 0.306 0.333 0.028 0.340 0.035 0.114 0.116 0.002 0.128 0.013 Car or other vehicle 0.096 0.094 -0.002 0.115 0.018 0.039 0.037 -0.002 0.043 0.004 Mobile Phone 0.760 0.760 0.000 0.807 0.047* 0.796 0.875 0.079*** 0.889 0.093*** Land & crop farming Owns land 0.669 0.663 -0.007 0.648 -0.022 0.608 0.621 0.013 0.616 0.007 Crop farming 0.699 0.692 -0.007 0.663 -0.036 0.725 0.725 0.000 0.721 -0.004 Owns any livestock 0.469 0.475 0.006 0.457 -0.011 0.559 0.565 0.006 0.575 0.015 Operates nonfarm enterprise 0.595 0.630 0.035 0.653 0.058** 0.023 0.025 0.002 0.024 0.001 Any member has financial account 0.493 0.523 0.030 0.586 0.094*** 0.595 0.637 0.042** 0.663 0.068*** Income from remittance (int'l & domestic) 0.319 0.319 0.000 0.327 0.008 0.340 0.348 0.008 0.349 0.009 †The results of an adjusted Wald test comparing weighted means are presented (*** p<0.01, ** p<0.05, * p<0.10). 37 Table 4.5b: Effectiveness of Bias Correction in Ethiopia HFPS by Urban-Rural domain Ethiopia - Urban Ethiopia - Rural ESS HFPS (n=2,270) HFPS (n=1,013) ESS Rural Urban Adjusted Weights Unadjusted Weights (n=3,115) Adjusted Weights Unadjusted Weights (n=3,655) Difference† Difference† Difference† Difference† Mean Mean Mean Mean Mean Mean (2) - (1) (4) - (1) (7) - (6) (9) - (6) Characteristic (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) Average household size 3.647 3.786 0.139 3.761 0.114 4.885 5.130 0.245 5.167 0.282* Dependency ratio 0.651 0.653 0.002 0.626 -0.025 1.103 1.055 -0.049 1.041 -0.063 Female head (%) 0.346 0.343 -0.004 0.321 -0.026 0.223 0.192 -0.031 0.180 -0.043** Age of head 38.728 38.993 0.265 38.590 -0.138 44.924 43.680 -1.243 43.365 -1.558 Religion of head Orthodox 0.574 0.577 0.003 0.587 0.013 0.431 0.447 0.016 0.447 0.015 Muslim 0.207 0.203 -0.004 0.194 -0.014 0.335 0.337 0.003 0.318 -0.017 Christian 0.213 0.215 0.002 0.215 0.002 0.210 0.204 -0.006 0.224 0.014 Marital status of head Married 0.618 0.631 0.013 0.644 0.026 0.798 0.814 0.016 0.819 0.021 Formerly married 0.220 0.222 0.002 0.205 -0.015 0.174 0.156 -0.017 0.141 -0.033* Never married 0.162 0.146 -0.016 0.151 -0.011 0.029 0.030 0.001 0.040 0.012 Education level of head Head Literate 0.753 0.743 -0.010 0.813 0.06** 0.408 0.434 0.026 0.508 0.1*** None 0.287 0.308 0.021 0.225 -0.062** 0.637 0.613 -0.024 0.525 -0.112*** Primary 0.296 0.295 -0.001 0.299 0.003 0.296 0.297 0.001 0.340 0.044 Secondary 0.215 0.212 -0.003 0.247 0.031 0.047 0.056 0.008 0.081 0.033* Tertiary 0.202 0.185 -0.017 0.230 0.027 0.020 0.035 0.014 0.055 0.034*** Working status of head Any 0.592 0.567 -0.025 0.601 0.010 0.804 0.827 0.022 0.827 0.022 Wage 0.302 0.283 -0.019 0.326 0.024 0.051 0.064 0.013 0.080 0.029* Agriculture 0.124 0.131 0.007 0.104 -0.020 0.756 0.771 0.015 0.747 -0.009 Nonfarm enterprise 0.219 0.198 -0.021 0.215 -0.004 0.066 0.079 0.013 0.095 0.029 †The results of an adjusted Wald test comparing weighted means are presented (*** p<0.01, ** p<0.05, * p<0.10). 38 Table 4.5b (continued): Effectiveness of Bias Correction in Ethiopia HFPS by Urban-Rural domain Ethiopia - Urban Ethiopia - Rural ESS HFPS (n=2,270) HFPS (n=1,013) ESS Rural Urban Adjusted Weights Unadjusted Weights (n=3,115) Adjusted Weights Unadjusted Weights (n=3,655) Difference† Difference† Difference† Difference† Mean Mean Mean Mean Mean Mean (2) - (1) (4) - (1) (7) - (6) (9) - (6) Characteristic (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) Consumption Mean expenditure (LCU per capita) 22,876 21,332 (1,544) 23,397 521 11,130 10,982 (148) 12,120 989 Quintiles Quintile 1 - poorest 0.059 0.068 0.009 0.044 -0.015 0.268 0.285 0.017 0.234 -0.034 Quintile 2 - poorer 0.107 0.126 0.019 0.096 -0.010 0.245 0.235 -0.009 0.227 -0.017 Quintile 3 - middle 0.158 0.170 0.012 0.157 -0.001 0.221 0.232 0.012 0.245 0.024 Quintile 4 - richer 0.274 0.282 0.008 0.294 0.019 0.164 0.158 -0.006 0.181 0.017 Quintile 5 - richest 0.402 0.354 -0.048 0.409 0.007 0.103 0.089 -0.014 0.112 0.009 Housing Improved roof 0.954 0.961 0.008 0.968 0.015 0.562 0.613 0.051 0.640 0.079* Improved floor 0.482 0.465 -0.016 0.522 0.040 0.030 0.044 0.014 0.052 0.022* Improved toilet 0.593 0.587 -0.006 0.631 0.039 0.179 0.217 0.038 0.221 0.043 Improved water source 0.976 0.979 0.003 0.983 0.007 0.637 0.673 0.036 0.695 0.058 Access to electricity 0.905 0.913 0.008 0.935 0.031 0.393 0.495 0.102** 0.511 0.118** Asset ownership Solar 0.051 0.047 -0.004 0.047 -0.004 0.202 0.237 0.035 0.238 0.035 Generator TV 0.505 0.518 0.013 0.575 0.07* 0.021 0.035 0.015 0.048 0.027** Refrigerator 0.213 0.212 -0.001 0.247 0.034 0.004 0.007 0.003 0.009 0.005 Bicycle 0.032 0.036 0.005 0.042 0.011 0.002 0.002 0.000 0.003 0.000 Motorbike 0.019 0.019 0.000 0.021 0.002 0.004 0.004 0.000 0.006 0.001 Car or other vehicle 0.028 0.026 -0.003 0.031 0.002 0.000 0.000 0.000 0.000 0.000 Mobile Phone 0.866 0.924 0.058*** 0.948 0.082*** 0.403 0.643 0.24*** 0.668 0.265*** Other activities Owns land 0.962 0.953 -0.009 0.948 -0.014 Crop farming 0.876 0.910 0.034* 0.895 0.018 Owns any livestock 0.882 0.908 0.026 0.895 0.013 Operates nonfarm enterprise 0.380 0.356 -0.024 0.372 -0.007 0.157 0.189 0.032 0.212 0.054* Any member has financial account 0.773 0.804 0.031 0.841 0.068*** 0.305 0.417 0.112*** 0.450 0.145*** Income from remittance (int'l & domestic) 0.215 0.195 -0.020 0.200 -0.015 0.111 0.120 0.009 0.126 0.015 †The results of an adjusted Wald test comparing weighted means are presented (*** p<0.01, ** p<0.05, * p<0.10). 39 Figure 4.2: Bias Reduction Results - Demographics 40 Figure 4.3: Bias Reduction Results – Head work status 41 Figure 4.4: Bias Reduction Results – Consumption expenditure quintiles* *For Malawi, the quintiles are based on a wealth index rather than consumption expenditures. 42 Figure 4.5: Bias Reduction Results – Housing 43 Figure 4.6: Bias Reduction Results – Asset ownership 44 Figure 4.7: Bias Reduction Results – Agricultural activities 45 Figure 4.8: Bias Reduction Results – Other economic activities 46 Figure 4.9: Bias Reduction Results – Consumption Expenditures 5. Conclusion Several developing countries are currently implementing phone surveys in response to immediate data needs to monitor the socioeconomic impact of COVID-19. In addition to being a safe alternative during the pandemic, phone surveys have several other logistical advantages. However, they are often subject to coverage and non-response bias that can compromise the representativeness of the sample and the external validity of the estimates obtained from the survey. These biases can be more relevant to developing countries where a considerable share of the population lacks access to a phone and connectivity problems are pervasive. Using data from high frequency phone surveys in Ethiopia, Malawi, Nigeria, and Uganda, this study investigated the magnitude and source of the biases and explored the effectiveness of the techniques applied to reduce them. The study demonstrated the advantages of sampling from representative face-to-face surveys to adjust for these biases. The study finds substantial coverage bias in Ethiopia, Malawi, and Uganda. The profile of households with contact information tended to be considerably different from the representative F2F survey. Households in the phone survey frame were more likely to be urban and richer and more likely to own key assets and to live in dwellings with improved features such as a modern roof and floor, improved water source and 47 toilet facilities, and electricity. Coverage bias was not much of a concern in Nigeria, but this can be largely attributed to relatively higher mobile phone penetration in the country compared to Ethiopia, Malawi, and Uganda. However, a more serious problem and common in all four countries is non-response bias due to unsuccessful contact with the respondent. The successfully contacted sample was biased towards wealthier households with higher living standards. This bias can be largely attributed to the difficulties contacting respondents either as a result of poor network reliability, respondent’s phone being turned off or unpowered, or the respondent not picking up the phone. This shows that even though higher phone penetration can lead to lower coverage bias, there is still substantial potential for bias due to unsuccessful contact with respondents in these countries. On the other hand, bias due to non-interview (refusals and breakoffs) did not introduce much additional bias in any of the countries. The overall bias found is substantial and widespread across different characteristics. The bias nearly always tends to favor wealthier households and thus poorer households are underrepresented. This direction in the bias, left unaddressed, would result in biased estimates from the interviewed sample that do not fully reflect the situation of poorer households in the county. This is a population of critical interest to policy makers since poorer households are likely most vulnerable to the negative impacts of the COVID- 19 crisis. Counteracting this bias is therefore essential to ensure that the results more closely reflect the reality of the poor and provide accurate information to policy makers. The study has shown that these phone survey biases can be substantially reduced by applying survey weight adjustments using information from the representative F2F survey from which the sample is drawn. This was demonstrated using a wide array of demographic and socioeconomic variables that are often included in standard household consumption and well-being measurement surveys. While these bias adjustments did not fully eradicate the bias across all dimensions, they were highly effective at reducing bias. This highlights one advantage to drawing phone survey samples from existing face-to-face, representative surveys rather than from RDD or lists from telecom providers where such adjustment methods can be more limited. Phone surveys are likely to be widely considered as alternative data collection platforms in developing countries especially in emergency situations. National statistical offices have unique opportunities to implement these techniques using their recently implemented representative F2F surveys that can serve as frames for phone surveys. 48 References Ambel AA.; Bundervoet T, Tsegay AH, Wieser C. 2020. Survey Methodology Document Vol. 1 of Monitoring COVID-19 Impacts on Households in Ethiopia. Monitoring COVID-19 Impacts on Households in Ethiopia. Washington, D.C.: World Bank Group. https://hubs.worldbank.org/docs/ImageBank/Pages/DocProfile.aspx?nodeid=32076450 Andersson, P. G., & Särndal, C. E. (2016). Calibration for nonresponse treatment using auxiliary information at different levels. In The Fifth International Conference on Establishment Surveys (ICES-V), Geneva, Switzerland, June 20–23, 2016. Ballivian, A., Azevedo, J., Durbin, W., Rios, J., Godoy, J., & Borisova, C. 2015. Using mobile phones for high-frequency data collection. Mobile Research Methods, 21. Dabalen, A, Etang, A, Hoogeveen, J, Mushi, E, Schipper, Y, and von Engelhardt, J. 2016. Mobile Phone Panel Surveys in Developing Countries: A Practical Guide for Microdata Collection. Washington, DC: World Bank. Fu, N., Glennerster,R., Himelein, K., Rosas Raffo, N., Suri, TK. 2015. The socio-economic impacts of Ebola in Sierra Leone: results from a high frequency cell phone survey- round one. Washington, D.C.: World Bank Group. Himelein K. 2014. Weight calculations for panel surveys with subsampling and split‐off tracking. Statistics and Public Policy 1(1): 40– 45. Himelein K, Eckman S, Kastelic J G, McGee K R, Wild M, Yoshida N, Hoogeveen, JG. 2020. High Frequency Mobile Phone Surveys of Households to Assess the Impacts of COVID-19 (Vol. 2): Guidelines on Sampling Design. Washington, D.C.: World Bank Group. Himelein K. and Kastelic JG. 2015. The socio-economic impacts of Ebola in Liberia: results from a high frequency cell phone survey. Washington, DC; World Bank Group. Hoogeveen J, Croke K, Dabalen A, Demombynes G, Giugale M. 2014. Collecting high frequency panel data in Africa using mobile phone interviews. Canadian Journal of Development Studies 35(1): 186– 207. Lau C Q, Cronberg A, Marks L, Amaya A. 2019. In Search of the Optimal Mode for Mobile Phone Surveys in Developing Countries. A Comparison of IVR, SMS, and CATI in Nigeria. In Survey Research Methods (Vol. 13, No. 3, pp. 305-318). Leo B, Morello R, Mellon J, Peixoto T, Davenport S. 2015. Do mobile phone surveys work in poor countries? Center for Global Development Working Paper 398. Little, R. J. 1986. Survey nonresponse adjustments for estimates of means. International Statistical Review/Revue Internationale de Statistique, 139-157. Lundström, S., & Särndal, C. E. 1999. Calibration as a standard method for treatment of nonresponse. Journal of official statistics, 15(2), 305. 49 McGee, K R, Amankwah A, Sagesaka A . 2020. Impact of COVID-19 on Nigerian Households (Vol. 3): Basic Information Document (English). Monitoring COVID-19 Impacts on Households in Nigeria Washington, D.C.: World Bank Group. http://documents.worldbank.org/curated/en/717901591889288314/Basic-Information- Document Tillé, Y. 2006. Sampling algorithms (pp. 147 - 176). Springer New York. Tomlinson M, Solomon W, Singh Y, Doherty T, Chopra M, Ijumba P, Tsai AC, Jackson D. 2009. The use of mobile phones as a data collection tool: a report from a household survey in South Africa. BMC Medical Informatics and Decision Making 9(51): 1– 8. 50 Appendix Tables Table A3.1: Response Propensity Model Results (logit) Variable Ethiopia Malawi Uganda Head age 0.045*** 0.051** (0.010) (0.017) Head age squared -0.000*** -0.000** (0.000) (0.000) Head is female 0.010 -0.227** (0.073) (0.078) Head is married -0.008 (0.077) Head is literate 0.365*** (0.089) Head education level (none omitted) Incomplete primary 0.524*** (0.074) Complete primary 0.786*** (0.123) Incomplete secondary 0.900*** (0.105) Complete secondary 0.959*** (0.143) Complete post-secondary 0.983*** (0.101) Number of school years completed by head 0.083*** (0.014) Highest years of education of any household member 0.062*** (0.013) Head labour market status Unemployed -0.105 (0.114) Inactive -0.392*** (0.076) Agriculture -0.410*** (0.085) Casual labor -0.068 (0.146) Household members working status % of Household Members that were employed for a wage 0.681** (0.211) % of Household Members that engaged in casual/ganyu labor -0.043 (0.124) Household received remittances 0.121* (0.060) Any household member has an account at a financial institution 0.354*** (0.085) 51 Table A3.1 (continued): Response Propensity Model Results (logit) Variable Ethiopia Malawi Uganda Household received cash directly from government -0.199 (0.187) Household had access to credit 0.301* (0.138) Household Owns dwelling -0.331*** (0.069) Number of persons per room 0.121* (0.055) Urban 0.474*** (0.073) Household size -0.015 0.058*** 0.116*** (0.015) (0.017) (0.027) Consumption/Wealth Quintiles (poorest quintile omitted omitted) Q2 - Poorer 0.416*** 0.706*** 0.411** (0.099) (0.104) (0.146) Q3 - Middle 0.658*** 1.336*** 0.578*** (0.097) (0.108) (0.151) Q4 - Richer 0.735*** 1.826*** 0.763*** (0.098) (0.121) (0.162) Q5 - Richest 0.852*** 1.878*** 0.952*** (0.100) (0.159) (0.193) Food consumption score 0.005** (0.002) Coping strategies index (lowest risk tercile omitted) CSI Tercile 2 - Medium Risk -0.282*** (0.081) CSI Tercile 3 - Highest Risk 0.012 (0.090) # of months head was away from the household in the past 12 months 0.106 (0.062) Head spent last night in the dwelling 0.386 (0.206) Location fixed effects None District District Language fixed effects No Yes No Main economic industry of household No No Yes R-squared 0.119 0.216 0.123 N 6767 5297 3030 Note: Coefficients presented with standard errors in parentheses. Significance denoted * p<0.05, ** p<0.01, *** p<0.001. 52