POLICY BRIEF July 2021 Remote Learning: Evidence from Nepal during COVID-19 Karthika Radhakrishnan, Shwetlena Sabarwal, Uttam Sharma, Claire Cullen, Colin Crossley, Thato Letsomo, Noam Angrist* SUMMARY: 84% of students in grades 3 to 5 lack foundational numeracy skills such as 2-digit division 30% increase in foundational numeracy via phone calls & SMS messages. Government teachers were highly effective, revealing the program is scalable by the education system. SMS messages alone were not effective. 98% Demand from households to continue to receive educational services through mobile phone BACKGROUND This note discusses early results from a distance education program on foundational numeracy for primary school students in Nepal during COVID-19 evaluated in a randomized trial. The trial included 3,700 households with children in public school (grades 3-5) across 10 local governments in seven provinces in Nepal. Households enrolled in January 2021 and follow-up data was collected in April-May 2021. We describe early results, including findings on foundational numeracy skills. This project is led by the World Bank, in collaboration with The Ministry of Education, Science and Technology (MoEST), local governments, Teach for Nepal (TFN), Street Child, and Young 1ove.† The program’s main goal was to prevent learning loss during school closures and provide instructional support for foundational numeracy using mobile phones, a high-access and low-cost approach to provide instruction at scale. The program was based on a successful trial in Botswana1 and was explicitly designed to be effective both when schools are closed as well as when they reopen to remediate learning loss. * We thank the government of Nepal for close collaboration, Nepal Development Research Institute for careful data collection, Teach for Nepal and Street Child for high-quality implementation, and Young 1ove for technical assistance. A special thank you to Amy Jung, Sunil Poudel, Prativa Yadav, and Pratigya Remi who played a critical role in data collection. We also gratefully acknowledge the contribution of Mohan Prasad Aryal and Maya Sherpa who were consulted during the course of the trial. † The Nepal trial is funded by the World Bank. This trial is a part of a multi-country effort to conduct randomized trials on common distance mobile education interventions led by Noam Angrist, Peter Bergman, and Moitshepi Matsheng. The broader multi-country trial is supported through a coalition of partners including: Douglas B. Marshall Foundation, Echidna Giving, Jacobs Foundation, Mulago Foundation, Stavros Niarchos Foundation, and UBS Optimus Foundation. Research for the multi-country trials was supported by J-PAL, Columbia University, UK Global Challenges Research Fund, and the University of Oxford. CONTEXT In Nepal, COVID-19 related school closures were enforced on March 19th, 2020, and have been largely in place since, with temporary reopening in late 2020, prior to the April 2021 resurgence of COVID-19 cases. During this time, the Government of Nepal has rolled out learning programs using radio, television, and online platforms. They have also disseminated printed learning materials to students. Despite the government’s efforts to roll-out these multi-modal distance learning programs, school closures are likely to have long-term impacts on student well-being, education, and learning.2 For Nepal, the average child was expected to receive 7.2 years of high-quality education before the pandemic. This metric of high-quality schooling combines the quantity (years of schooling) and quality (how much students learn) of schooling into a single measure: Learning-Adjusted Years of Schooling (LAYS).3 For example, the average child in Nepal is expected to receive 12.3 years of schooling. However, the average child in Nepal only scores 369 on internationally benchmarked learning assessments relative to a threshold of advanced skills of 625, thus discounting 12.3 years of schooling to 7.2 years of high- quality schooling. The LAYS measure is the education pillar of the World Bank Human Capital Index. Conservative estimates suggest that LAYS for Nepal will likely drop by 0.5 post pandemic.4 This is likely to decrease future annual earnings of Nepali students by about US $296 a year. The present value of lifetime earnings for all students is expected to decrease by US $14 billion (in 2011 PPP), which is three times the current annual education expenditure in Nepal.4 In Nepal, there are many households that do not have access to radio, TV, and internet. A baseline survey showed that over 40% of households did not have a single member with access to the internet and 70% did not have a radio.5 This creates substantial risk that in Nepal learning inequalities will increase as a result of the school closures and when schooling restarts, disadvantaged children will find themselves even further behind their peers. While mobile phone access is high (for example, 87% say they can use their mobile phone to access various types of remote education), mobile phone-based education remains underutilized at just 1.5 Moreover, a baseline found that only 31% of students in Nepal had teacher interaction during school closure.5 This reveals a need to activate mobile-based education, such as teacher phone calls to students as well as SMS messages. EVALUATION AND INTERVENTION DESIGN The evaluation was conducted as a randomized controlled trial and included multiple treatment arms: a pure control, weekly SMS content messages only, weekly SMS nudge messages to engage in education, 20-minute weekly phone calls by teachers, and 20-minute weekly phone calls by NGO facilitators. The trial took place across 10 local governments and seven provinces in Nepal. Local governments include Birendranagar, Dhangadi Sub Metropolitan City, Duhabi, Hupsekot Rural, Melamchi, Mithila Bihari Rural, Rampur, Siddhalek Gaunpalika, Suryodaya, Thori Rural. Initial phone number collection and securing consent to participate in the program was conducted from November-December 2020; baseline data was collected in January 2021; and follow-up data was collected in April-May 2021. The trial was implemented both when schools reopened post the pandemic and during the school closures due to a second wave in Nepal. Figure 1 includes a timeline. Randomization was conducted at the household level, since often a mobile phone belonged to a caregiver in the household who shared it with the primary school child, rather than the child owning a phone themselves. More details around sampling and sample characteristics noteTimeline: Figure: are discussed in the baseline here. Low-Tech Trial available Nepal Figure 1: Timeline of Intervention and Trial Pilot Baseline Launch Intervention Follow-up Data Nov 2020 Jan 2021 Feb 2021 Apr 2021 Preparation Ongoing Trial TRAINING OF FACILITATORS AND TEACHERS Facilitators and teachers were provided a five-day virtual training which included (i) assessing the learning level of the child; (ii) focus on basic numeracy and target instruction to the child’s level; and (iii) checking for understanding through regular assessment. Information sharing sessions with teachers and facilitators were also conducted 0 / 0 the May 30, 2021 during course of the trial. Regular monitoring to address or clarify any issues from teachers and facilitators was an integral part of the trial. Teachers and facilitators were provided the phone numbers of households to schedule the weekly phone calls at a convenient time to ensure that no costs are incurred by the households. The trial took place both during the lockdown when schools were closed and while the schools were open. Reaching caregivers and students was easier during the lockdown since the caregivers who carried the mobile phones were available at home. DATA AND METHODOLOGY A series of outcomes were collected, including foundational skills based on the Annual Status of Education Report (ASER), an assessment conducted in over 14 countries across the world.6-8 Data was also collected on caregiver demographics, engagement in education, and demand for the program, among a suite of other outcomes. Here we present a snapshot of early findings. 3 FINDINGS The results above reveal a few important takeaways. First, as shown in Table 1, we find that many caregivers in the household are engaged in mobile education, including mothers (35.4%), fathers (20.0%), and siblings (30.5%). Table 1: Multiple members of the household involved in providing education Mother 35.4 Father 20.0 Grandparent 2.1 Sibling 30.5 Others not mentioned above in the household 6.3 Teacher 2.1 Adults from the community who don't live in the household 3.6 Note: Answers to the question “Who typically provides educational instruction/ support with schoolwork to the student” in treatment groups Second, as shown in Figure 2, we find engagement in numeracy increases overall as a result of the intervention. Whereas in the control group only 24% of households say they always engaged in numeracy activities in a given week, this increases to 49%-55% for those who receive phone calls from teachers and NGO facilitators. Figure 2: Higher engagement with educational instruction if received the intervention 1 0.08 0.10 0.18 0.03 0.02 0.28 0.13 .8 0.07 0.17 0.09 0.18 0.19 0.19 .6 0.22 0.15 .4 0.09 0.55 Don't Know 0.49 Always 0.40 .2 0.24 Frequently Sometimes Rarely 0.09 Never 0.02 0.02 0.03 0 Control Nudge &/or SMS Teacher NGO Note: Answers to the question “Over the past week, how often did the guardian spend doing educational activities with the child?” 4 Demand for the intervention is high (see Table 2): over 97.5% percent of households want to continue receiving some version of mobile education services, even when schools are to reopen. The vast majority of households demand active phone calls (86.9%) and a substantial share of households want both the phone calls and SMS messages (35.7%). Table 2: High demand for the program (over 85 percent demand for phone calls) No they are not interested 2.5 Yes through both Phone Calls and SMS 35.7 Yes only through SMS 10.6 Yes only through Phone Calls 42.4 Yes through either Phone Calls or SMS 8.8 Note: Percent demand to continue mobile education service if in any treatment group In terms of learning, in the status quo (control group), only 54 percent of students can do 2- digit addition, 45 percent can do subtraction, 30 percent can do multiplication, and 16 percent can do division (Table 3). This reveals a striking gap of 84% students unable to do basic foundational numeracy skills such as 2-digit division. The intervention bridges this gap substantially. Table 3: Large gaps in foundational numeracy (e.g., only 16% can do division) The student was to solve: 52 + 39 0.54 The student was to solve: 42 - 29 0.45 The student was to solve: 28*3 0.30 The student was to solve: 65/8 0.16 Note: Foundational numeracy skills in the status quo – % of students who get a given proficiency correct in the control group We estimate learning gains in Figure 3 in terms of standard deviation gains. While the SMS messages / nudges on their own are only marginally effective with 0.03 gains (not statistically significant), active phone calls by teachers and NGO facilitators are highly effective (and statistically significant at the 1% level). Teachers improve learning by 0.19 standard deviations, and NGO facilitators improve learning by 0.14 standard deviations. In terms of proficiency gains these represent around 20-30 percent increases. For example, while 30 percent of students can do multiplication in the control group, after the teacher intervention, this increases to 40 percent, representing a 30 percent increase; in the NGO treatment this increases to 36 percent, representing a 20 percent increase. It is noteworthy that government teachers are highly effective, revealing the approach tested can be adopted and scaled by the education system. We further observe larger effects for students from the poorest and lowest parent literacy backgrounds, suggesting the approach tested can bridge equity gaps. 5 Figure 3: Large gains in foundational numeracy from phone calls by Teachers and NGOs 0.22 0.30 0.19 0.17 0.14 0.20 0.05 0.03 0.10 0.00 -0.10 S er O SM G ch N a or Te &/ ge ud N Main effects Effects for poorest/low literacy households Notes: The outcome is a continuous variable capturing whether a child answered multiple foundational numeracy domains correctly: addition, subtraction, multiplication, and division. Foundational numeracy gains are reported by treatment arm relative to the control group in terms of standard deviations. DISCUSSION AND POLICY IMPLICATIONS The results of this trial have tremendous policy relevance for Nepal and other countries. Schools remain closed in many parts of Nepal, and even prior to COVID-19 many students lacked foundational numeracy and literacy skills. The government of Nepal is launching a large-scale effort to recover learning losses from COVID-19 school closures, with a focus on building foundational numeracy and literacy into the long-run. The approach tested in this trial could be a promising component of a multifaceted program to support remedial education and catch-up programs to improve foundational skills. Low-cost phone-based interventions to support student learning can be expanded while school closures continue. However, even after schools re-open, this approach can be used to provide additional targeted support to struggling students. The approach tested is noteworthy not just for its effectiveness but also for its cost- effectiveness. The estimated cost per child of the intervention is US $20, which translates to 0.69-0.93 standard. deviations gained per $100. These results equate to a full Learning- Adjusted Year of School (LAYS) gained per $100.9 Thus, these results more than mitigate the predicted 0.5 LAYS loss from covid-19 school closures, revealing the approach tested could be a promising component of a response which “builds back better.” The remote learning approach tested in this study also relates closely to remediation approaches to improve foundational learning, such as Teaching at the Right Level, which 6 conduct regular assessment, target instruction to a child’s level, and focus on basic numeracy and literacy. This approach has recently been highlighted in a World Bank, FDCO, UN, and BE2 policy note outlining “Smart Buys” to cost-effectively improve education outcomes.10 TESTIMONIALS “I was worried that it would be difficult to teach a girl over the phone who could not even recognize numbers, but I am now confident that she is learning. She attends the phone call every week when she takes the cattle to the farm for feeding,” -Tulasa Kafle, a community school teacher from Illam. “I like the way the teacher is teaching over the phone. In the beginning my child did not know to solve problems, but he has improved a lot now.” -Janaki Mandal, guardian of Ashik Kumar Mandal. “Teaching using mobile phone is the best way in the current situation. I spoke with some of the teachers regarding their experience and their response was that teaching at the level of each child definitely helps. According to them, this type of phone-based learning has helped improve the learning level of children. The program is also cost-effective and is very easy to implement” -Bikal Kafle, Section Officer, Education Office, Rampur Rural Municipality. “I am able to solve math problems that I had not learned in school. I have now learned up to fractions. I found the method of teaching through phone easier than it used to be in class” -Bedika Poudel, Grade 3 student. 7 Teacher Phone Calls in Action 1. Angrist, Noam, Peter Bergman, and Moitshepi Matsheng. School’s Out: Experimental Evidence on Limiting Learning Loss Using “Low-Tech” in a Pandemic. No. w28205. National Bureau of Economic Research, 2020. 2. World Bank. "The COVID-19 pandemic: Shocks to education and policy responses." (2020). 3. Filmer, Deon, Halsey Rogers, Noam Angrist, and Shwetlena Sabarwal. "Learning-adjusted years of schooling (LAYS): Defining a new macro measure of education." Economics of Education Review 77 (2020): 101971. 4. Sharma, U., Sherpa, M. & Radhakrishnan, K. (2020). Covid-19 related learning losses in Nepal. Unpublished Manuscript. World Bank. 5. Radhakrishnan, Karthika, Noam Angrist, Peter Bergman, Claire Cullen, Moitshepi Matsheng, Anusha Ramakrishnan, Shwetlena Sabarwal, and Uttam Sharma. "Learning in the Time of COVID-19." (2021). 6. Banerji, Rukmini, Suman Bhattacharjea, and Wilima Wadhwa. "The annual status of education report (ASER)." Research in Comparative and International Education 8, no. 3 (2013): 387-396. 7. Banerjee, Abhijit, Rukmini Banerji, James Berry, Esther Duflo, Harini Kannan, Shobhini Mukerji, Marc Shotland, and Michael Walton. "From proof of concept to scalable policies: Challenges and solutions, with an application." Journal of Economic Perspectives 31, no. 4 (2017): 73-102. 8. People’s Action for Learning Network (PAL). https://palnetwork.org/ 9. Angrist, Noam, David K. Evans, Deon Filmer, Rachel Glennerster, F. Halsey Rogers, and Shwetlena Sabarwal. "How to Improve Education Outcomes Most Efficiently? A Comparison of 150 Interventions Using the New Learning-Adjusted Years of Schooling Metric." (2020). The World Bank. 10. Global Education Evidence Advisory Panel. “Cost-Effective Approaches to Improve Global Learning: What Does Recent Evidence Tell Us Are “Smart Buys” for Improving Learning in Low and Middle Income Countries?“ (2020). The World Bank. 8