LEARNING IN THE TIME OF COVID-19: INSIGHTS FROM NEPAL 1 March 29, 2021 Karthika Radhakrishnan, Noam Angrist, Peter Bergman, Claire Cullen, Moitshepi Matsheng, Anusha Ramakrishnan, Shwetlena Sabarwal, and Uttam Sharma SUMMARY This note discusses the impact of COVID-19 and related school closures on primary students’ access to learning in Nepal. Our primary source of data is a phone-survey with 1,800 households that have children enrolled in public schools (grades 3-5) collected from November 2020 to February 2021. We describe student learning, parental perception of student levels, access to learning during school closures, and families’ emotional health during COVID-19. For a majority of students (77 percent), school textbooks have been the major form of remote learning during school closures. Only 31 percent of students had teacher interaction during this time. Nearly 25 percent of parents spent no time helping children learn. Among those that do, average time spent is about 9.3 hours per week. This number is significantly lower than the attention children would have received in schools. Households, especially disadvantaged households, were significantly more likely to access low- technology solutions like working with textbooks and interacting with teachers (instead of online or TV- learning). Overall, during school closures, disadvantaged parents (by wealth/ caste) are significantly: (i) less likely to have accessed active remote learning; (ii) less likely to engage in the child’s learning; and (iii) less likely to receive teacher support. Disadvantaged parents have children performing at relatively lower levels of proficiency and also more experience stress/worry. However, there are no gender-based gaps in student performance, access to remote learning, and access to parental engagement. These results show the need to protect (and boost) the learning levels of children from disadvantaged backgrounds, especially in foundational domains, by better harnessing low-technology mechanisms (like phone-tutoring) which are more easily accessible to these households. 1. INTRODUCTION 2.1 BACKGROUND In Nepal, COVID-19 related school closures were enforced on March 19th, 2020, and have been largely in place for approximately eight months. To protect learning in the face of school closures, the Government of Nepal rolled out learning programs using radio, television and online platforms. They have also disseminated offline learning materials to students. 1 This note uses baseline data from the ongoing study on ‘Low-Tech Intervention for Foundational Education’ 1 Despite the government’s efforts to roll-out these multi-modal distance learning programs, these school closures are likely to have long-term impacts on student well-being, education, and learning. For Nepal, the Learning Adjusted Years of Schooling (LAYS (Filmer et. al 2020)) 2, a measure that combines quantity and quality of schooling, was 6.9 years before the pandemic.3 Even conservative estimates suggest that LAYS for Nepal will likely drop to 6.4 years—a 0.5 year drop. This is a significant decrease (Sharma, Sherpa, and Radhakrishnan 2020). This is likely to decrease the future annual earnings of Nepali students by about US$ 296 a year. The present value of lifetime earnings for all students is expected to decrease by US$ 14 billion (in 2011 PPP), which is three times the current annual education expenditure in Nepal (Sharma, Sherpa and Radhakrishnan 2020). Risks of learning loss and drop-outs are significantly higher among students from poorer socio-economic backgrounds. This is because more educated and wealthy families will be better able to sustain their children’s learning at home through higher access to devices, books, and educated caregivers who can actively tutor the child (World Bank 2020). In Nepal, there are many households that do not have access to radio, TV and internet. Only 63% of the households have access to at least one technology (radio, TV, internet)4. This rate also varies a lot across rural and urban households (54 percent vs. 69 percent) and across provinces. This creates a real risk that in Nepal learning inequalities will increase as a result of the school closures and when schooling restarts, disadvantaged children will find themselves even further behind their peers. 2.2 LOW-TECH PILOT To address the challenge of increased learning inequality due to COVID-related school closures, the World Bank, in collaboration with The Ministry of Education, Science and Technology (MoEST), local governments and Teach for Nepal (TFN), are undertaking a pilot to protect and boost the learning of poorer students using low-tech solutions. This is called the Low-Tech Intervention for Foundational Education (LIFE) and is based on a successful large-scale trial in Botswana (Angrist et. al 2020). LIFE’s main goal is to help prevent learning loss among primary-school age children from poorer families. It involves SMS messages and phone calls to support foundational skills in math among primary school aged children (Grades 3-5). It is designed as a randomized control trial - in one treatment arm the intervention is being delivered through trained facilitators from a local NGO (Teach for Nepal) and in the other arm the intervention is being delivered through public school primary teachers. The intervention design promotes foundational learning among students irrespective of the timeline of school re-opening. It is beneficial for children, even if schools re-open. The pilot was rolled out in late January 2021 to around 3,700 students (including students in the control group) across 10 local governments 2 For more details see Filmer et. al 2018 3 “World Bank. 2018. The Human Capital Project. World Bank, Washington, DC. © World Bank. https://openknowledge.worldbank.org/handle/10986/30498 License: CC BY 3.0 IGO.” 4 Nepal Labor Force Survey 2017-18 2 (from all seven provinces).5 These were selected (i) to provide a good geographic representation across different provinces and (ii) where there was strong interest from local government officials. 2. DATA AND METHODOLOGY The data for this note comes from two surveys – (i) Sensitization survey and (ii) Baseline survey. Both were phone-based and focused on the ten local governments selected for participation in the study. The sample for the study was created as follows. In the 10 selected local governments, public schools were contacted and asked to provide the list of names and phone numbers for all students enrolled in grades 3, 4, and 5. From this list of 10,000 phone numbers, we randomly selected 8,678 households to approach for potential participation in the study. This process of selecting our final sample from the broader list of 8,678 potential participants was done through the phone-based sensitization survey. Sensitization survey: Between November and December 2020, we interviewed 4,085 households that had a child in Grades 3-5 in public school for a 20 minute sensitization survey. The purpose of this survey was to confirm that the household: (i) had a child in Grades 3-5 and (ii) was interested in being a part of the study. In addition, a few questions were asked about the students learning level and access to phone. Of the 8,678 households contacted, 5,166 answered after 5 attempts to call them, and 4,944 were interested and had eligible children. After ruling out the households where we were not able to speak directly to the best contact point for the child, we were left with a sample of 4,085 households6. This was the full sample for our pilot. Baseline survey: From January 4 to January 17 2021, we interviewed a randomly selected sub-sample of 1,872 students7 from sensitization for a more detailed (40 minute) baseline survey. Below we present some key findings from both the sensitization and baseline surveys. 3. A SNAPSHOT OF STUDENT LEARNING As part of the phone survey, a short Math test was administered (5 questions). This test is based on the ASER test (Banerji et. al 2013) and was adapted for phone delivery (Angrist et. al 2020). The test includes addition, subtraction, multiplication and division8 questions that Grade 2 students are expected to answer. Overall, only 1.2 percent of grade 3 students, 3.8 percent of grade 4 students and 6.2 percent of grade 5 students were able to answer all questions on the mathematics assessment correctly. Only 16 percent of 5 Suryodaya and Duhabi Municipality from Province 1, Mithila Bihari Rural Municipality and Thori Rural Municipality from Province 2, Melamchi Municipality and Siddhalek Rural Municipality from Bagmati province, Hupsekot Rural Municipality from Gandaki province, Rampur Municipality from Lumbini province, Birendranagar Municipality from Karnali province and Dhanagadhi Sub-Metropolitan City from Sudurpaschim province. 6 From the 4,085 sensitization sample households, we were left with a final sample of 3,732 households after ruling out baseline households that were no longer contactable, eligible, interested, or were not the best contact point for the student. The final sample of 3,732 students were then assigned into treatment or control groups. 7 We randomly selected 2,250 households from our full sample of 4,085 households for the baseline survey. However, we were able to reach 1,872 households for the baseline survey. 8 The division question asked students to divide a two-digit number by a one-digit number, with a remainder. 3 grade 3 students, 27 percent of grade 4 students and 38 percent of grade 5 students answered 3 or more questions correctly. Analysis by question type shows that student competencies are in fact increasing perceptibly by grade. However, even by grade 5, most students struggle with division, and to a lesser extent, 2-digit multiplication. At the other end, nearly 35 percent of Grade 5 and 49 percent of grade 4 students have not mastered foundational topics like 2-digit addition with carryover9. Children belonging to poor households10 or households from disadvantaged castes11 have lower scores on average. Children from poor households scored 24 percent on average, while those from non-poor households scored 34 percent. Children from disadvantaged castes scored 30 percent, while those from advantaged castes scored 40 percent on average. The differences are significant at the 1 percent level. There are no significant differences in children’s scores by gender. Given that these data come from very short assessments (5 questions only), delivered over the phone, they should be interpreted with caution. 4. DO PARENT’S KNOW THEIR CHILD’S LEARNING LEVEL? 9 We show data on place value but we are not sure how much salience this question had with students given the phone-based test administration 10 Poor households are defined as those who do not own a TV or a radio. 11 These include Janajati, Tharu, Dalit, Madhesi. 4 One interesting insight from this work is that there is a low correlation between the parent’s perception of their child’s ability level and the child’s demonstrated ability at the baseline test. The actual Pearson’s correlation coefficient is only 0.2. Parents' perception of child's ability as compared to other children in the same age group Actual performance on the mathematics assessment Below average Average Above average Below average 10% 3% 2% Average 27% 14% 18% Above average 9% 5% 12% - Percentage of parents who perceived their child’s ability accurately. - Percentage of parents who overestimated their child’s ability (i.e., actual performance of the child on the mathematics assessment was below parental perception - Percentage of parents who underestimated their child’s ability (i.e. actual performance of the child on the mathematics assessment was above parental perception) A majority of parents are likely to misjudge their child’s ability as compared to other children in the same age group. As can be seen in the table above, around 41 percent of parents overestimated their child’s ability; 23 percent underestimated their child’s ability; and 36 percent perceived their child’s ability accurately. 5. LEARNING DURING SCHOOL CLOSURES 5.1 REMOTE LEARNING Around 70 percent of children accessed some form of remote learning during school closures. However, the most common form of remote learning, at 77 percent of students, was simply via grade-level textbooks. This is similar to findings from Bangladesh where around 85 percent of students are relying solely on textbooks for remote learning (Baird et. al 2020). The next most common method, showing significantly lower take-up, is mobile education12 at 25 percent. TV-based learning is even lower at 9 percent. Around 12 percent of students also received homework from teachers. Note that about 23 percent of students accessed multiple forms of remote learning, which is why the total sums to more than 100 percent. While 70 percent of households surveyed reported that they don’t have a radio, over 87 percent reported that they can access radio / FM on their mobile. In terms of internet access, 58 percent households reported that at least one member from their household uses internet. However, access to internet differs by caste and poverty level. Households belonging to advantaged castes13 are significantly more likely to have internet access as compared to other households. Further, poor households are significantly less likely to have access to internet. 12 Mobile education in this context refers to support provided by teachers to students during school closure that involved them going to a certain area on a certain day and teaching students residing in that area. 13 These include Brahmin, Chhetri and Newar. 5 The likelihood of accessing remote learning does not vary systematically between poor and non-poor households. However, it does vary by caste. Among households belonging to advantaged castes, only 25 percent of children did not access remote learning; this number is 33 percent for households belonging to disadvantaged castes. This difference is significant at the 1 percent level. Access to active remote learning14, defined as instructional support provided in addition to basic inputs like textbooks, differs by poverty level. 16 percent of poor households reported using some kind of active remote learning support, as compared to 23 percent of non-poor households. This difference is significant at the 1 percent level. There is no difference in access by caste or gender. As mentioned above, around 12 percent of students received homework from teachers. In addition, 25 percent of students received direct support from teachers. However, likelihood of receiving teacher support differed by poverty level. 21 percent of poor students received support from teachers, as compared to 27 percent of non-poor students. This difference is significant at the 1 percent level. There is no difference by caste or gender. On average, households that received teacher support (468 households), reported that teachers spent 7.3 hours per week providing support to children. 5.2 LEARNING SUPPORT AT HOME 14 Active remote learning is defined as instructional support provided in addition to inputs. This includes all forms of learning support except providing textbooks and learning packages. Mobile education, teacher support, TV program, temporary learning center, online class, learning portal and phone-based teaching all constitute active remote learning. 6 Around 75 percent of households report spending some time per week supporting children’s learning. Poor households and households belonging to disadvantaged castes were less likely to spend time supporting children when compared to non-poor households / advantaged households. 69 percent of poor households reported spending some time in comparison to 77 percent of non-poor households. Further, 72 percent of households belonging to disadvantaged castes spent time with children as compared to 78 percent of households belonging to advantaged castes. These differences are significant at the 1 percent level. There is no difference in time spent by gender. On average, households that reported spending time with children, spent 9.3 hours per week supporting children’s learning. This contrasts quite negatively with what children would experience in schools, where classes take on average 5-6 hours a day. Also, since these children are in Grades 3-5, parental support for learning is relatively more important as the children are still below the age where they can engage in meaningful self-study. If we include all parents (including those that spend no time helping their child learn), there are significant differences between the time spent by household members in supporting children’s learning in poor households that spend 6 hours per week and non-poor households that spend 7.4 hours. Similarly, households belonging to disadvantaged castes spend 6.7 hours on average each week, and households belonging to advantaged castes spent 7.5 hrs. This difference is significant at the 1 percent level, implying that support provided to children at home is significantly less in households belonging to disadvantaged castes. This difference is significant at 1 the percent level. On the positive side, there is no difference in the support provided, based on the child’s gender. These findings underline the need to provide targeted support to disadvantaged families where children have less access to remote learning and to parental support for learning. 6. COVID-19 AND EMOTIONAL HEALTH There are encouraging signs in these data. Around 86 percent of households reported that the child was rarely / never stressed in the last two weeks. There was no difference in this statistic by caste, gender, or poverty level. Almost 94 percent of the households reported that the child was happy most of the time / some of the time in the last month. There were no significant differences in this metric by caste, gender, poverty level. However, to the extent that parents do not have complete knowledge of their child’s emotional well-being, these numbers should be interpreted with caution. On the other hand, parents themselves are more likely to report being stressed (overall number 12.5 percent). Not surprisingly poor households were significantly more likely to report that they had been stressed ‘often’ / ‘always’ in the last two weeks as compared to non-poor households. Around 17 percent of poor households reported being stressed ‘often’ / ‘always’ in the last two weeks as compared to 10 percent of non-poor households. The difference is significant at the 1 percent level. When parents were asked about the biggest problem their child faced during COVID, 57 percent mentioned their child falling behind in their education, while another 4 percent said not being able to take their exams. Only 7 percent talked about getting sick and 24 percent said they were not concerned. 7 7. CONCLUSION Our findings show that in the 8-9 months since school closures, children’s access to active remote learning and active support for learning remains low in Nepal. This access is significantly lower for disadvantaged households (in terms of wealth and/or caste). These gaps are even more troubling given that children from these households perform at lower levels and parents exhibit more stress and worry. In light of this, it is clear that without active support learning inequality between advantaged and disadvantaged children is likely to increase in the face of COVID-related school closures. The ongoing pilot titled ‘Low-Tech interventions for foundational education’ is helping address these challenges. REFERENCES Angrist, N., Bergman, P., Brewster, C., & Matsheng, M. (2020). Stemming learning loss during the pandemic: A rapid randomized trial of a low-tech intervention in Botswana. Available at SSRN 3663098. Baird, S., Seager, J., Sabarwal, S., Guglielmi, S., & Sultan, M. (2020). Adolescence in the Time of COVID- 19. World Bank Policy Note. Banerji, R., Bhattacharjea, S., & Wadhwa, W. (2013). The annual status of education report (ASER). Research in Comparative and International Education, 8(3), 387-396. Filmer, D., Rogers, H., Angrist, N., & Sabarwal, S. (2020). Learning-adjusted years of schooling (LAYS): Defining a new macro measure of education. Economics of Education Review, 77, 101971. Sharma, U., Sherpa, M. & Radhakrishnan, K. (2020). Covid-19 related learning losses in Nepal. Unpublished Manuscript. World Bank. World Bank. (2020). The Covid-19 Pandemic: Shocks to Education and Policy Responses. World Bank: Washington, DC. https://openknowledge.worldbank.org/handle/10986/33696 © 2020 International Bank for Reconstruction and Development / The World Bank 1818 H Street NW, Washington DC 20433 Telephone: 202-473-1000, Internet: www.worldbank.org 8 This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Nothing herein shall constitute or be considered to be a limitation upon or waiver of the privileges and immunities of The World Bank, all of which are specifically reserved. Rights and Permissions This briefing note is available under the Creative Commons Attribution 3.0 IGO license (CC BY 3.0 IGO) http://creativecommons.org/licenses/by/3.0/igo. 9