Policy Research Working Paper 10033 An Analysis of COVID-19 Student Learning Loss Harry Anthony Patrinos Emiliana Vegas Rohan Carter-Rau Education Global Practice May 2022 Policy Research Working Paper 10033 Abstract COVID-19 caused significant disruption to the global consolidate all the available evidence and documents the education system. Early reviews of the first wave of lock- empirical findings. Thirty-six robust studies were identi- downs and school closures suggested significant learning fied, the majority of which find learning losses on average loss in a few countries. A more recent and thorough analysis amounting to 0.17 of a standard deviation, equivalent to of recorded learning loss evidence documented since the roughly a one-half year’s worth of learning. These find- beginning of the school closures between March 2020 and ings confirm that learning loss is real and significant, even March 2022 finds even more evidence of learning loss. Most compared to the first year of the pandemic. Further work studies observed increases in inequality where certain demo- is needed to increase the quantity of studies produced, and graphics of students experienced more significant learning to ascertain the reasons for learning loss and in a few cases losses than others. But there are also outliers, countries that mitigation of loss. managed to limit the amount of loss. This review aims to This paper is a product of the Education Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at hpatrinos@worldbank.org. The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Produced by the Research Support Team An Analysis of COVID-19 Student Learning Loss Harry Anthony Patrinos Emiliana Vegas Rohan Carter-Rau World Bank Harvard University Brookings Institution JEL codes: I14, I15, I20, I24, I25, O151 Keywords: COVID19, Education, Learning loss, Systematic review I. Introduction The COVID-19 pandemic led to school closures around the world, affecting almost 1.6 billion students (UNESCO 2020). Even short disruptions in a child’s schooling have significant negative effects on their learning and can be long lasting. The capacities of education systems to respond to the crisis by delivering remote learning and support to children and families have been diverse yet uneven. In response to this disruption, education researchers are beginning to analyze the impact of these school closures on student learning loss. The term learning loss is commonly used in the literature to describe declines in student knowledge and skills. Outside the classroom, these losses may translate to greater long-term impacts (Azevedo et al 2021; Hanushek and Woessmann 2020; Psacharopoulos et al 2021). But there are also outliers, countries that managed to limit the amount of loss. Research on the actual impact of COVID-19 on learning progress is beginning to emerge. School closures have a large, persistent, and unequal effect on learning. Online education is an imperfect substitute for in-person learning, particularly for children from low-income families. Peer effects also change; schools allow children from different socio-economic backgrounds to mix, and this effect is lost when schools are closed. Another factor is the response of parents, some of whom compensate for the changed environment through their own efforts, while others are unable to do so. All this leads to the finding that school closures have a large, persistent, and unequal effect on human capital accumulation. In the USA, high school students from low-income communities suffer a learning loss of 0.4 standard deviation after a one-year school closure, whereas children from high-income neighborhoods initially remain unscathed (Agostinelli et al. 2022). The channels operating through schools, peers, and parents all contribute to growing educational inequality during the pandemic. Early systematic reviews based on limited data from high-income countries 2 suggest large learning losses and increased learning inequality. Most recently, Donnelly and Patrinos (2021) find average learning losses of about 0.13 standard deviation (SD) in seven high- income countries. Hammerstein et al. (2021) find learning losses of 0.10 SD, Storey and Zhang (2021) estimate losses at 0.15 SD, and Zierer (2021) finds losses of 0.14 SD. Moscoviz and Evans (2022) also find that learning losses are mostly negative. In this paper, we focus on the question, to what extent did COVID-19 lead to student learning losses across the world? Our review covers learning loss evidence documented during the first two years of the pandemic. Previous reviews have documented learning loss, and we build on those efforts. We make three contributions. First, we develop a comprehensive review that consolidates the research that has been completed related to the impact of COVID-19 on student learning progress and build a public use database. Second, we respond to the research question of the extent to which there is evidence of pandemic-related learning losses. Third, based on our review, we identify significant gaps in the literature and provide relevant guidance for further research. We begin by describing the methods used in this review to identify and collect the articles analyzed. We then present an analytical review and our strategy for reporting effects, where each article was categorized by its geographical region studied, length of school closure, education level of students analyzed, sample size, subject(s) analyzed, documented learning impact, and presence of increased inequality. Next, we discuss the findings of our review of the emerging evidence on learning losses. Last, we briefly summarize the key findings and discuss areas for future research. 3 This paper aims to identify, review and analyze the existing evidence on the effect of the COVID- 19 pandemic and school closures on learning outcomes progress among school-age children and youth. The specific research questions are: 1. What is the impact of the COVID-19 pandemic and school closures on learning progress among school children and youth? 2. How does the impact differ across socioeconomic background, age, boys and girls, subjects, sub-national contexts, ethnicity, etc. (data permitting)? II. Search strategy and identification methods We searched the following scholarly sources: • ECONLIT, Google Scholar, PubMed, Education Resources Information Center, and Cochrane Library • Coronavirus Research Database, Education Database (ProQuest), Web of Science • Preprint and working paper repositories: SSRN, MPRA, IZA, NBER, OSF Preprints, PsyArXiv, SocArXiv, EdArXiv • Reach out to researchers. To conduct the search, the keywords “COVID-19”, “coronavirus”, “2019-ncov”, “sars-cov-2”, or “cov-19”, were used in combination with “learning loss”, “learning slide”, “education gap”, or “achievement gap”. We focused on learning loss among school-age children across all subjects, with special attention to the most reported subjects of math and reading. School-age children and youth are defined as being between ages 5 and 18, thus excluding both kindergarten and higher education. 4 We considered all observational research including, but not limited to, studies using quantitative data that is longitudinal or cross-sectional, individual or aggregate (e.g., at the level of schools) on the effect of the COVID-19 pandemic on learning loss among students. To be included in our sample, the outcome measure must be a valid measure of learning progress. These include measures using scores from school-based tests or from assessments administered by researchers or international assessment programs, including EGRAs, and any study that can be scaled to the Harmonized Learning Outcomes (HLO) (Angrist et al 2021). In addition, studies must have measured learning loss with pre-COVID control data, and during or post-COVID treatment data. Studies using projections and simulations were excluded. Secondary research such as systematic reviews were excluded from the analysis but used to identify further relevant studies. Given the focus on the COVID-19 pandemic, we restricted our search to papers published between March 1, 2020, and March 1, 2022. Studies were screened according to the following selection criteria: Studies that conducted student analyses and reported impacts on learning progress (either positive, negative, or insignificant) because of COVID-19 school disruptions were included. Factors for rejecting studies from our review included the absence of student analyses or recorded impacts on learning progress, analyses occurring before the onset of COVID-19, or hypothesized results. We collected study details such as author name(s), sample size, information on the age of students in sample, length of closures, measures of social background recorded, outcomes measured, follow-up period(s), and the effect sizes. We recorded the data in an MS Excel spreadsheet. We assessed the methodological rigor of each study and subsequently rated the robustness of the overall body of evidence across all outcomes. 5 For studies that contained data on individual learning loss, we aggregated these into a composite measure of learning for the most common specific learning areas examined in all relevant studies (such as math, reading). If sufficient evidence was available, we also aggregated estimates of learning loss in other specific learning areas. If there were multiple model specifications including controls, we extracted the relevant coefficients from the models that allowed for comparison of effect sizes across studies. Our initial search consisted of a google scholar query using the keywords (“COVID-19” OR “coronavirus” OR "pandemic)" AND ("learning loss" OR "learning slide"), restricted to publications between March 2020 and March 2022. This search revealed 3,840 results. Of these, 167 directly addressed learning loss during the pandemic. Of these studies, 32 attempted to directly measure learning loss caused by the COVID-19 pandemic. We then performed further searches of the other databases, and of the working paper repositories. This revealed an additional 13 studies. Finally, while evaluating these studies, we pulled a further 36 papers from reference lists and other review papers. This resulted in a total of 81 studies that examine learning loss due to the COVID- 19 pandemic. Five of these studies were duplicates published in different formats and were dropped from the analysis. Another five studies did not contain a direct measure of learning loss in the form of pre- and post-pandemic data and were also dropped from the analysis. Finally, we dropped from our analysis four studies that examined learning loss only at the higher education level. III. Analytical review and effects reporting We carefully reviewed the remaining 67 studies and rated their methods scored on a scale of 1 (lowest quality methods) to 5 (highest quality methods). We rated 1 those papers that presented learning loss through a comparison of pre- and post-COVID means, without statistical analysis for 6 variation to account for potential confounding variables, or random sampling techniques. We rated 2 those papers that performed difference of means tests but used convenience sampling or had small sample sizes. We rated 3 papers that performed difference of means tests, had random or representative sampling, and a relatively large sample size. We rated 4 papers that performed additional statistical analysis beyond a comparison of means, including strong sampling and basic individual or school level controls. Finally, we rated 5 those papers that contained sophisticated statistical analysis, with strong controls and sampling. This systemic review focuses on the 35 studies that are scored 3, 4, and 5. The findings of studies scored 1 and 2 are included in Appendix 1. When reporting learning loss effect size, we present findings in terms of standard deviation change of the treatment (post-COVID) group from the pre-COVID control group. As each study uses a different measure of learning loss, losses reported in percentiles and points are not easily comparable. Following Hanushek and Woessmann (2020), we convert results presented in months of loss to standard deviations, with 1 school year of learning equal to 0.33 standard deviation. Unfortunately, many papers do not present results in standard deviation or months of learning losses, and do not have sufficient information for us to calculate these. These results are excluded from Table 1 and Figure 1, but they are included in the overall analysis. For these papers, we primarily discuss the direction of their findings, as the quantitative findings are not comparable across papers. Where possible, we also present heterogeneous effects in standard deviations. Many studies include only some of these effects, meaning the number of study results reported varies substantially by effect. 7 IV. Learning losses due to COVID-19 Our final database consists of 35 robust studies and reports documenting learning loss, representing data from 20 countries (see Table 1). Most studies (32) find evidence of learning loss. Of the 35 studies reporting learning loss, 27 reported findings in a comparable effect size format. As shown in Figure 1, most studies found learning losses ranging from 0.25 to 0.12 SD. In five studies, learning losses were even greater. The average learning loss across these studies is 0.17 standard deviation – which equates to over half a school year of learning loss. 8 Table 1: Final database of robust studies of learning loss due to COVID-19 Closure Average length learning Equity Source Country weeks 1 losses (SD) impact Sample size 1 Ardington et al 2021 South Africa 22 -0.22 Yes 5,810 2 Arenas and Gortazar 2022 Spain -0.05 Yes 41,000 3 Asakawa and Ohtake 2021 Japan 11 Yes 5,400 4 Bielinski et al 2021 United States -0.14 N not reported 5 Birkelund and Karlson 2021 Denmark 8 0 Yes 200,000 6 Borgonovi and Ferrara 2022 Italy Yes 850,000 7 Chen et al 2021 United Arab Emirates N/A 1,920 8 Clark et al. 2021 China 7 -0.22 N/A 1,835 9 Contini et al 2021 Italy 15 -0.19 Yes 1,539 10 Domingue et al 2021 United States -0.05 N 100,000 11 Engzell et al 2021 Netherlands 8 -0.08 Yes 350,000 12 EPI 2021 England -0.09 Yes 180,000 13 Gambi and De Witte 2021 Belgium -0.23 Yes 213000 14 Gore et al. 2020 Australia 8 0 N 4,800 15 Haelermans et al 2021 Netherlands 10 -0.17 Yes 201,819 16 Halloran et al 2021 United States Yes 11,700 17 Hevia et al 2021 Mexico 48 -0.55 Yes 3,161 18 Hicks and Faulk 2022 United States Yes not reported 19 Jakubowski 2022 Poland 29 -0.30 N/A 4,581 20 Kogan and Lavertu 2021 United States -0.23 Yes 124,700 21 Korbel and Prokop 2021 Czech Republic -0.11 N/A 2,234 22 Kuhfeld et al 2022 United States -0.19 Yes 5,400,000 23 Lichand et al 2021 Brazil -0.32 Yes 7,000,000 24 Locke et al 2021 United States -0.15 No 55,000 25 Ludewig et al 2022 Germany -0.14 N/A 4,290 26 Maldonado and Witte 2022 Belgium 9 -0.18 Yes 1,300 27 Philadelphia 2021 United States N/A 65,000 28 Pier et al 2021 United States -0.08 N/A 100,000 29 Schult et al 2021 Germany 10 -0.08 N/A 80,000 30 Schuurman et al 2021 Netherlands 8 -0.09 Yes 886 31 Skar et al 2021 Norway 7 -0.24 N/A 2,453 32 Spitzer, Musslick Germany 8 N/A 2,500 33 Tomasik et al 2020 Switzerland 8 -0.20 N/A 28,685 34 van der Velde Netherlands 8 N/A 133,450 35 Chaban et al 2022 Russian Federation 8 - 20 -0.27 No 165,740 1 We report the number of weeks of school closures as reported in each study. When the study did not report the length of school closures, we also do not report it in Table 1 or use it in our analyses. 9 Figure 1: Range of learning loss (in SDs) in the 24 studies reporting comparable effect sizes The studies consistently find heterogeneous effects of learning loss by socio-economic status, past academic learning, and subject of learning. In our review, 20 studies examined learning loss by socio-economic status. Of these, 15 find greater learning loss among students or schools with lower socio-economic status, while 5 fail to find a statistically significant difference. Many studies have also found learning loss to be worse for students who had struggled academically prior to the pandemic. Of the studies that included this measure, 11 documented greater learning losses for students at lower levels of academic achievement, while 3 documented greater learning losses for students with greater past academic achievement. The COVID-19-related school closures varied in duration across the world. In some countries, school systems were completely shut down for many months – or even years in some cases – while in a few countries schools were closed for very short periods of time. As the pandemic continued, some school systems partially opened some schools for some months. Also, the experience varied by grade level. Some countries kept primary schools open but closed 10 secondary schools. Some countries had good internet connectivity and students had access to devices; these systems were well positioned to provide quality distance education online. Other countries, with limited internet connectivity and access to devices, broadcast lessons and used radio or other mass media to provide remote education. As Table 2 shows, the longer the schools remained closed, the greater were the learning losses. For the 19 countries for which we have robust learning loss data, average school closures were 15 weeks, leading to average learning losses of 0.18 standard deviation. Put another way, for every week that schools were closed, learning declined by 1.2 point, or 0.01 standard deviation, on average. Most of our robust data come from 12 European countries. Given similar starting conditions overall (such as GDP per capita, access to connectivity and devices), these countries provide compelling evidence that the pandemic-related school closures led to learning losses. In these European countries, average school closures were 11 weeks in duration, and average learning losses were 0.16 standard deviation. Therefore, for every week of school closure, learning losses were about 1.5 points, or 0.015 standard deviation, per month. 11 Table 2: Length of school closures and average learning losses, by country Average learning losses Country Closure length weeks Source (SD) Mexico 48 -0.55 Hevia et al 2021 Brazil 26 -0.32 Lichand et al 2021 Poland 20 -0.3 Jakubowski 2022 Russian 14 -0.27 Chaban et al 2022 Federation Norway 7 -0.24 Skar et al 2021 China 7 -0.22 Clark et al. 2021 South Africa 22 -0.22 Ardington et al 2021 Switzerland 8 -0.2 Tomasik et al 2020 Italy 15 -0.19 Contini et al 2021 Belgium 9 -0.18 Maldonado and Witte 2022 Netherlands 10 -0.17 Haelermans et al 2021 Germany 10 -0.14 Ludewig et al 2022 Average of: Bielinski et al 2021; Domingue et al 2021; United 23 -0.14 Kogan and Lavertu 2021; States Kuhfeld et al 2022; Locke et al 2021; Pier et al 2021 Czech -0.11 Korbel and Prokop 2021 Republic 9 England 10 -0.09 EPI 2021 Spain 12 -0.05 Arenas and Gortazar 2022 Australia 8 0.00 Gore et al. 2020 Denmark 8 0.00 Birkelund and Karlson 2021 Japan 11 0.0 Asakawa and Ohtake 2021 We also observed notable variation in learning loss by subject. The subjects most frequently analyzed in the reviewed studies are reading and mathematics. Of the 11 studies to include both, in 9 studies learning loss in mathematics exceeded that of reading, while reading losses exceeded mathematics in 2 studies. Many studies have examined learning loss by gender, although the findings are not consistent. While some studies report greater learning loss among girls (Lichand et al 2021), others find 12 greater learning loss among boys (Birkelund and Karlson 2021; Schuurman et al. 2021). Seven other studies find no statistical difference in learning loss by gender. Learning losses are significant in countries reporting losses through descriptive statistics. For example, adolescent girls’ literacy and numeracy scores declined by more than 6 percent in Bangladesh (Amin et al 2021). In India, the share of grade 3 children in government schools who can perform simple subtraction decreased from 24 percent in 2018 to only 16 percent in 2020 and the share who can read a grade 2 level text decreased from 19 percent in 2018 to 10 percent in 2020 (ASER 2021a). In Pakistan, the proportion of children in classes 1-5 who can read a story declined from 24 percent in 2019 to 22 percent in 2021 (ASER 2021b). There are reported learning losses in one college in Sri Lanka (Sayeejan and Nithlavarnan 2018). In Uganda, the percentage of learners rated proficient in literacy in English and numeracy in 2021 dropped by 5 percent and 13 percent from that of 2018 (NAPE 2021). In Canada, grade 2 and 3 students reading assessments declined by 4 to 5 points (Georgiou 2021). In the Republic of Korea, there was a significant decrease in scores in for medical school students (Kim et al. 2021). V. Discussion As a result of the first wave of COVID-19-induced lockdowns and school closures beginning in March 2020, a student has lost on average about one-third to one-half year’s worth of learning. These learning losses may impact a student’s education trajectory, as the lost learning is likely to limit opportunities to advance to higher levels of schooling. There are also long-term future earnings losses associated with lost human capital, with students potentially losing trillions of dollars in future income (Psacharopoulos et al 2021). Nevertheless, there are some causes for optimism from the existing research. In Nara City, Japan, math achievement scores had increased six months after COVID-related school closures. While 13 the initial closures had brought down scores, responsive policies largely overcame this decline (Asakawa and Ohtake 2021). During the school closures, students watched virtual lessons streamed by their teachers, carried out independent work, and submitted paperwork for revision by their teachers. Nara City also decreased its summer vacation in response to the school closures, from 34 days to 16 days. Although a broadly successful response, the school closures impacted lower income students significantly more, and these students also showed slower growth than their peers upon returning to school. Another positive finding comes from England, where the Educational Policy Institute found learning loss to decline substantially after schools had reopened. In autumn 2020, students were found to be behind by an average of 3.6 months in mathematics, but by the summer of 2021 the loss had decreased to an average of 2.2 months. Learning loss evidence is mixed in France, with on average very low levels of learning loss experienced thus far, though there are still equity effects with more disadvantaged students suffering more (Thorn and Vincent-Lancrin 2021). However, no learning loss is evident in Denmark and no equity effect (Birkelund and Karlson 2021). It seems that Danish children received good home support and their reading behavior increased significantly because of the lockdown of schools (Reimer et al 2021). In Sweden, where primary schools did not close during the pandemic, there are no reported learning losses (Falth et al 2021). In the state of New South Wales, Australia, there are no significant differences between 2019 and 2020 in student achievement growth; however, learning did decrease in the least advantaged schools (Gore et al 2021). As this emerging evidence suggests, the impact of COVID-19 on student learning worldwide has been substantial, and inequality in learning across more advantaged and disadvantaged groups is likely going to grow over time. We plan to continue to assess these trends over time and to 14 evaluate longer-term impacts on individuals’ education and labor market trajectories as well as productivity and economic development globally. 15 References Agostinelli, F., Doepke, M., Sorrenti, G., Zilibotti, F. (2022). When the great equalizer shuts down: Schools, peers, and parents in pandemic times. Journal of Public Economics 206 104574. https://doi.org/10.1016/j.jpubeco.2021.104574 Amelia, R., G. Kadarisma, N. Fitriani and Y. Ahmadi. (2020). The effect of online mathematics learning on junior high school mathematic resilience during covid-19 pandemic. Journal of Physics. Conf. Ser. 1657 012011 https://iopscience.iop.org/article/10.1088/1742- 6596/1657/1/012011 Amin, S, Hossain, M, Ainul, S. (2021). Learning loss among adolescent girls during the COVID- 19 pandemic in rural Bangladesh. New York: Population Council. https://knowledgecommons.popcouncil.org/departments_sbsr-pgy/1501/ Amplify. (2021). Research brief: COVID-19 means more students not learning to read. https://go.info.amplify.com/download-whitepaper- fy21_general_moydata_national_readingresearch#:~:text=Amplify's%20latest%20middle%2Dof %2Dyear,of%20not%20learning%20to%20read.&text=Research%20suggests%20a%20near%20 90,to%20close%20this%20opportunity%20gap Ardington, C, Willis, G., Kotze, J. (2021). COVID-19 Learning Losses: Early Grade Reading in South Africa. International Journal of Educational Development. https://www.sciencedirect.com/science/article/pii/S0738059321001334 Arenas, Andreu and Lucas Gortazar. (2022). Learning Loss One Year After School Closures: Evidence from the Basque Country. IEB Working Paper 2022/3. Institut d’Economia de Barcelona. https://ieb.ub.edu/wp-content/uploads/2022/03/Doc2022-03.pdf 16 Asakawa, S., Ohtake, F. (2021). Impact of Temporary School Closure Due to COVID-19 on the Academic Achievement of Elementary School Students. Discussion Papers in Economics and Business 21-14, Osaka University, Graduate School of Economics. https://ideas.repec.org/p/osk/wpaper/2114.html ASER (Annual Status of Education Report). (2021a). Karnataka Rural, analysis based on data from households. 24 out of 30 districts. Facilitated by Pratham. https://img.asercentre.org/docs/ASER%202021/ASER%202020%20wave%201%20- %20v2/aser2020wave1report_feb1.pdf ASER (Annual Status of Education Report). (2021b). Measuring the Impact of COVID 19 on Education in Pakistan. FINAL REPORT. ASER Pakistan Secretariat, Idara-e-Taleem-o-Aagahi (ITA), Lahore, Pakistan. Available at: http://aserpakistan.org/Learning-Losses-2021 ASER. (2021). Annual Status of Education Report (Rural) 2020, Wave 1. ASER Centre, New Delhi. Available at: https://img.asercentre.org/docs/ASER%202021/ASER%202020%20wave%201%20- %20v2/aser2020wave1report_feb1.pdf Azevedo, J.P., Hasan, A., Goldemberg, D., Geven, K. and Iqbal, S.A., 2021. Simulating the potential impacts of COVID-19 school closures on schooling and learning outcomes: A set of global estimates. The World Bank Research Observer, 36(1), pp.1-40. Belot, M., Webbink, D. (2010). Do Teacher Strikes Harm Educational Attainment of Students? Labour 24(4): 391–406. https://onlinelibrary.wiley.com/doi/full/10.1111/j.1467- 9914.2010.00494.x 17 Bielinski et al. (2021). No Longer a Prediction: What New Data Tell Us About the Effects of 2020 Learning Disruptions. https://www.illuminateed.com/download/no-longer-a-prediction- what-new-data-tell-us-about-the-effects-of-2020-learning-disruptions/ Birkelund, J.F. and Karlson, K.B. (2021). No Evidence of a Major Learning Slide 14 Months into the COVID-19 Pandemic in Denmark. https://osf.io/md5zn/download Blainey, K., Hannay, T. (2020). The impact of lockdown on children’s education: a nationwide analysis. RS Assessment, School Dash. https://www.risingstars-uk.com/media/Rising- Stars/Assessment/Whitepapers/RS_Assessment_white_paper_1.pdf Borgonovi, F. and A. Ferrara. (2022). A Longitudinal Perspective on the Effects of COVID-19 on Students’ Resilience. The Effect of the Pandemic on the Reading and Mathematics Achievement of 8th and 5th Graders in Italy. Available at: http://dx.doi.org/10.2139/ssrn.4025865 Casalone, G., A. Michelangeli and U. Turk. (2021). The Effect of Lockdown on Students’ Performance: A comparative study between Sweden, Italy and Turkey. Center for European Studies. Paper Series 473. https://www.researchgate.net/publication/353582241_The_Effect_of_Lockdown_on_Students% 27_Performance_A_comparative_study_between_Sweden_Italy_and_Turkey Chaban T.Yu., Rameeva R.S., Denisov I.S., Kersha Yu.D., Zvyagintsev R.S. (2022) Rossiyskaya shkola v period pandemii COVID-19: effekty pervykh dvukh voln i kachestvo obrazovaniya [Russian Schools during the COVID-19 Pandemic: Impact of the First Two Waves on the Quality of Education]. Voprosy obrazovaniya/Educational Studies Moscow, no 1, pp. 160–188. https://doi.org/10.17323/1814-9545-2022-1-160-188 18 Chen, D.L. and S. Ertac, T. Evgeniou, X. Miao, A. Nadaf and E. Yilmaz. (2022). Grit and Academic Resilience During the Covid-19 Pandemic. INSEAD Working Paper No. 2022/02/DSC, Available at SSRN: https://ssrn.com/abstract=4001431 or http://dx.doi.org/10.2139/ssrn.4001431 Chetty, R., Friedman, J.N., Rockoff, J.E. (2014). Measuring the impacts of teachers II: Teacher value-added and student outcomes in adulthood. American Economic Review 104: 2633–2679. https://www.aeaweb.org/articles?id=10.1257/aer.104.9.2633 Clark, A.E., Nong, H., Zhu, H. and Zhu, R. (2021). Compensating for academic loss: Online learning and student performance during the COVID-19 pandemic. China Economic Review 68, p.101629. https://halshs.archives-ouvertes.fr/halshs-02901505/document Contini, D., Di Tommaso, M.L., Muratori, C., Piazzalunga, D. and Schiavon, L. (2021). The COVID-19 Pandemic and School Closure: Learning Loss in Mathematics in Primary Education (No. 14785). Institute of Labor Economics (IZA). https://ftp.iza.org/dp14785.pdf Curriculum Associates. (2020). Understanding Student Needs: Early Results from Fall Assessments https://www.curriculumassociates.com/-/media/mainsite/files/i-ready/iready- diagnostic-results-understanding-student-needs-paper-2020.pdf Currie, J., Thomas, D. (2001). Early test scores, school quality and SES: Longrun effects on wage and employment outcomes. Research in Labor Economics 20: 103–132. https://www.emerald.com/insight/content/doi/10.1016/S0147-9121(01)20039-9/full/html De Paola, M, Gioi, F, Scoppa, V. (2021). The Impact of Online Teaching on Students’ Achievement: Evidence from COVID-19 Induced Remote Learning. 19 Depping, D., Lücken, M., Musekamp, F., & Thonke, F. (2021). Kompetenzstände Hamburger Schüler* innen vor und während der Corona-Pandemie [Alternative pupils' competence measurement in Hamburg during the Corona pandemic]. In D. Fickermann & B. Edelstein (Eds.), Schule während der Corona-Pandemie. Neue Ergebnisse und Überblick über ein dynamisches Forschungsfeld (pp. 51–79). Waxmann. https://doi.org/10.31244/9783830993315.03 Domingue, B. W., Hough, H. J., Lang, D., Yeatman, J. (2021). Changing Patterns of Growth in Oral Reading Fluency During the COVID-19 Pandemic—Policy Brief. Policy Analysis for California Education (PACE). https://edpolicyinca.org/publications/changing-patterns-growth- oral-reading-fluency-during-covid-19-pandemic Donnelly, R., Patrinos, H.A. (2021). Learning loss during Covid-19: An early systematic review. Prospects https://doi.org/10.1007/s11125-021-09582-6 Dorn, E, Hancock, B, Sarakatsannis, J, Viruleg, E. (2021). COVID-19 and education: The lingering effects of unfinished learning. McKinsey & Company. https://www.mckinsey.com/industries/education/our-insights/covid-19-and-education-the- lingering-effects-of-unfinished-learning Education Policy Institute. (2021). Understanding progress in the 2020/21 academic year: Interim findings. Department for Education UK. Renaissance Learning. https://epi.org.uk/publications-and-research/learning-loss-report-understanding-progress-in-the- 2020-to-2021-academic-year/ 20 Empower. (2021). Update on Unfinished Learning in DC. https://static1.squarespace.com/static/5f9857f027d55d2170cd92ac/t/61934ad6d24c803afec13834 /1637042905631/EK12+Unfinished+Learning+Update+Brief+from+Spring+2020-21.pdf Engzell, P., A. Frey and M. Verhagen. (2021). Learning loss due to school closures during the COVID-19 pandemic. Proceedings of the National Academy of Sciences 118(17). https://www.pnas.org/content/118/17/e2022376118 Fälth, L., A. Hallin and T. Nordström. (2021). Conrona-Pandemins Paverkan pa Lagstadie- Elevers Lasinlarning. LegiLexi. Stockholm. https://legilexi.org/ Gambi, L. and K. de Witte. (2021). The resiliency of school outcomes after the COVID-19 pandemic. Standardised test scores and inequality one year after long term school closures. FEB Research Report Department of Economics. https://lirias.kuleuven.be/3611628?limo=0 Georgia Policy Labs. (2021). Tim Sass and Thomas Goldring. Student Achievement Growth during the COVID-19 Pandemic: Insights from Metro-Atlanta School Districts. https://gpl.gsu.edu/download/student-achievement-growth-during-the-covid-19-pandemic- report/?ind=1620902643226&filename=Achievement%20Growth%20During%20COVID- 19_Report_20210512_FINAL.pdf&wpdmdl=2105&refresh=609e9390ce82b1621005200 Georgiou, G. (2021). Covid-19’s Impact on Children’s Reading Scores: Data Trends and Complementary Interview. The Reading League Journal 2(2): 3-33. https://www.researchgate.net/publication/352546252_Covid- 19%27s_Impact_on_Children%27s_Reading_Scores_Data_Trends_and_Complementary_Intervi ew 21 Gonzalez, T., M.A. de la Rubia, K.P. Hincz, M. Comas-Lopez, L. Subirats, S. Fort an G.M. Sacha. (2020). Influence of COVID-19 confinement on students’ performance in higher education. PloS One. https://doi.org/10.1371/journal.pone.0239490 Gore, J., L. Fray, A. Miller, J. Harris and W. Taggart. (2021). The impact of COVID-19 on student learning in New South Wales primary schools: an empirical study. Aust. Educ. Res. 48, 605–637. https://doi.org/10.1007/s13384-021-00436-w Haelermans, C., R. Korthals, M. Jacobs, S. de Leeuw, S. Vermeulen, L. van Vugt, B. Aarts, T. Breuer, R. van der Velden, S. van Wetten and I. de Wolf. (2021). Sharp increase in inequality in education in times of the COVID-19-pandemic. ROA. ROA Research Memoranda No. 010 https://doi.org/10.26481/umaror.2021010 Halloran, C., R. Jack, J. Okun and E. Oster. (2021). Pandemic Schooling Mode and Student Test Scores: Evidence from US States. National Bureau of Economic Research Working Paper Series No. 29497. http://www.nber.org/papers/w29497.pdf Hammerstein, S., C. König, T. Dreisörner and A. Frey. (2021). Effects of COVID-19-Related School Closures on Student Achievement-A Systematic Review. Frontiers in Psychology. https://www.frontiersin.org/articles/10.3389/fpsyg.2021.746289/full Hanushek, E.A. and Woessmann, L., 2020. The economic impacts of learning losses. Hevia, F.J., S. Vergara-Lope Tristan, A. Velásquez-Durán and D. Calderón Martín del Campo. (2021). Estimation of the fundamental learning loss and learning poverty related to COVID-19 pandemic in Mexico, International Journal of Educational Development. https://www.sciencedirect.com/science/article/pii/S0738059321001681 22 Hicks, M.J., Faulk, D. (2022). What Contributed to COVID Learning Loss in Indiana’s Schools? https://projects.cberdata.org/reports/CovidLearningLoss-20220118.pdf Jakubowski, M., Gajderowicz, T., Wrona, S. (2022). Achievement of Secondary School Students After Pandemic Lockdown and Structural Reforms of Education System Results from TICKS 2021 Assessment in Warsaw. Evidence Institute Policy Note 1, Warsaw. https://www.evidin.pl/wp-content/uploads/2022/01/POLICY-NOTE-1-2022-EN.pdf Juniper Education. (2021). The impact of the Covid-19 pandemic on primary school children’s learning https://21e8jl3324au2z28ej2uho3t-wpengine.netdna-ssl.com/wp- content/uploads/juniper_folder/Juniper-Education-National-Benchmark-Dataset-Report.pdf Kogan, V. and Lavertu, S. 2021. The COVID-19 pandemic and student achievement on Ohio’s third-grade English language arts assessment. Ohio State University. https://glenn.osu.edu/covid- 19-pandemic-and-student-achievement-ohios-third-grade-english-language-arts-assessment Kim, Dl, H.J. Lee, Y. Lin, Y.J. Kang. (2021). Changes in academic performance in the online, integrated system-based curriculum implemented due to the COVID-19 pandemic in a medical school in Korea J Educ Eval Health Prof. 18:24. https://doi.org/10.3352/jeehp.2021.18.24 Kim, J, Rose, P, Tirunch, D.T, Sabates, R, Woldehanna, T. (2021). Learning Inequalities Widen Following COVID-19 School Closures in Ethiopia. https://riseprogramme.org/blog/learning- inequalities-widen-COVID-19-Ethiopia Kofoed, M., Gebhart, L., Gilmore, D. and Moschitto, R. (2021). Zooming to Class? Experimental Evidence on College Students Online Learning During Covid-19. IZA Discussion Paper 14356. https://www.iza.org/publications/dp/14356/zooming-to-class-experimental- evidence-on-college-students-online-learning-during-covid-19 23 Kogan, V., Lavertu, S. (2021). The COVID-19 pandemic and student achievement on Ohio’s third-grade English language arts assessment. Ohio State University. https://glenn.osu.edu/covid- 19-pandemic-and-student-achievement-ohios-third-grade-english-language-arts-assessment Korbel, V., Prokop, D. (2021). Czech students lost 3 months of learning after a year of the COVID-19 pandemic. Prague: PAQ Research. https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2F88760faa-4149-467c- 8d6a- 46e154cd4c14.usrfiles.com%2Fugd%2F88760f_c8f600c6cede4137a5bcb559319a8eb3.pdf&data =04%7C01%7Chpatrinos%40worldbank.org%7Cf17ef0450e6840929f9808d9883d3218%7C31a 2fec0266b4c67b56e2796d8f59c36%7C0%7C0%7C637690619146057218%7CUnknown%7CT WFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6 Mn0%3D%7C0&sdata=k6EXxcMJPY4gycZzFAx5EfDGys30ozKURdCCP0j2Ygo%3D&reserv ed=0 Kuhfeld, Lewis K, M., Ruzek, E., McEachin, A. (2021). Learning during COVID-19: Reading and math achievement in the 2020-21 school year. NWEA https://www.nwea.org/content/uploads/2021/07/Learning-during-COVID-19-Reading-and-math- achievement-in-the-2020-2021-school-year.research-brief.pdf Kuhfeld, M., Ruzek, E., Johnson, A., Tarasawa, B., Lewis, K. (2020). Technical appendix for: Learning during COVID-19. Initial Findings on Students’ Reading and Math Achievement and Growth. NWEA. 2020. Available online: https://www.nwea.org/content/uploads/2020/04/COVID-technical-appendix_2020.04.08.pdf 24 Lichand, G, Dória, C., Onício Leal Neto et al. (2021). The Impacts of Remote Learning in Secondary Education: Evidence from Brazil during the Pandemic, 03 June 2021, PREPRINT (Version 1) available at Research Square. https://doi.org/10.21203/rs.3.rs-568605/v1 Locke, VN., Patarapichayatham, C., Lewis, S. (2021). Learning Loss in Reading and Math in U.S. Schools Due to the COVID-19 Pandemic. Istation. https://www.curriculumassociates.com/research-and-efficacy/unfinished-learning- research?gclid=Cj0KCQiAraSPBhDuARIsAM3Js4r_Crbix5KxlKOnlbtuhQ8gSKHWju2AYxqvj jYe7ZR8wJf1rpus29kaAvzmEALw_wcB Ludewig, Ulrich, Ruben Kleinkorres, Rahim Schaufelberger, Theresa Schlitter, Ramona Lorenz, Christoph Koenig, Andreas Frey, Nele McElvany. (2022). COVID-19 Pandemic and Student Reading Achievement – Findings from a School Panel Study. PIRLS. 10.31234/osf.io/hrzae Major, LE, Eyles, A., Machin, S. (2021). Learning Loss since Lockdown: Variation across the Home Nations. COVID-19 Analysis Series. No. 023. Centre for Economic Performance. https://files.eric.ed.gov/fulltext/ED614031.pdf Maldonado, J.E. and De Witte, K., 2022. The effect of school closures on standardised student test outcomes. British Educational Research Journal, 48(1), pp.49-94. Meeter M. (2021). Primary school mathematics during the COVID-19 pandemic: No evidence of learning gaps in adaptive practicing results. Trends Neurosci Educ. https://pubmed.ncbi.nlm.nih.gov/34844699/ Moscoviz, L. and D.K. Evans. 2022. Learning Loss and Student Dropouts during the COVID-19 Pandemic: A Review of the Evidence Two Years after Schools Shut Down. CGD Working Paper 609. Washington, DC: Center for Global Development. 25 NAPE (National Assessment of Progress In Education). (2021). The Effect of COVID-19 Pandemic on Teaching and Learning at Primary and Secondary Education levels in Uganda. https://www.mediacentre.go.ug/sites/default/files/media/REPORT%202ND%20REVIEWED%2 0FOR%20DISSEMINATION%202021.pdf Oktariani, F., Fionasari, R., Ramdha, T. (2022). The Impact of Learning Loss due to Family Income Factors on Students' Achievement during the COVID-19 Pandemic. Edunesia: Jurnal Ilmiah Pendidikan 3(1): 12-20. https://www.edunesia.org/index.php/edu/article/view/203 Orlov, G., McKee, D., Berry, J., Boyle, A., DiCiccio, T., Ransom, T. Rees-Jones, A, Stoye, J. (2020). Learning During the COVID-19 Pandemic: It Is Not Who You Teach, but How You Teach. NBER Working Paper No. 28022. https://www.nber.org/system/files/working_papers/w28022/w28022.pdf Psacharopoulos, G., V. Collis, H.A. Patrinos and E. Vegas. (2021). The COVID-19 Cost of School Closures in Earnings and Income across the World. Comparative Education Review 65(2): 271-287. Rahman, A. (2021). The Impact of Covid-19 Pandemic on Students' Learning Outcome in Higher Education. Al-Ishlah: Jurnal Pendidikan 13(2): 1425-1431. http://journal.staihubbulwathan.id/index.php/alishlah/article/viewFile/974/435 Reimer, D., E. Smith, I.G. Andersen, B. Sortkær. (2021). What happens when schools shut down? Investigating inequality in students’ reading behavior during Covid-19 in Denmark. Research in Social Stratification and Mobility 71. https://doi.org/10.1016/j.rssm.2020.100568. Renaissance. (2020). How Kids Are Performing: Tracking the Impact of COVID-19 on Reading and Mathematics Achievement https://renaissance.widen.net/s/wmjtlxkhbm 26 Rose, S., Twist, L., Lord, P., Rutt, S., Badr, K., Hope, C., Styles, B. (2021). Impact of school closures and subsequent support strategies on attainment and socio-emotional wellbeing in Key Stage 1: Interim Paper 1 (January 2021). National Foundation for Educational Research. https://www.nfer.ac.uk/impact-of-school-closures-and-subsequent-support-strategies-on- attainment-and-socio-emotional-wellbeing/ Sabates, R, Carter, E., Stern, J. (2021). Using educational transitions to estimate learning loss due to COVID-19 school closures: The case of Complementary Basic Education in Ghana, International Journal of Educational Development, 82, 102377. https://doi.org/10.1016/j.ijedudev.2021.102377 Sayeejan, P., Nithlavarnan A. (2018). Learning Loss and Learning Difficulties of Junior Secondary Students in Science Subject Due to Covid-19 Outbreak: Study Based on J/Attiar Hindu College, Neervely. The Sri Lanka Journal of South Asian Studies 4(1): 21-38. https://www.researchgate.net/publication/357793543_Learning_loss_and_learning_difficulties_o f_junior_secondary_students_in_science_subject_due_COVID_19_OUTBREAK School District of Philadelphia. (2021). Assessing Student Performance Before and During Virtual Learning: A Cohort Comparison of Student Performance on 2019-20 Winter and 2020-21 Fall Plus and Star Assessments. https://www.philasd.org/research/wp- content/uploads/sites/90/2021/02/AimswebPlus-and-Star-Cohort-Study-Report-January-2021.pdf Schult, J., Mahler, N., Fauth, B., Lindner, M. (2021). Did students learn less during the COVID- 19 pandemic? Reading and math competencies before and after the first pandemic wave. https://doi.org/10.31234/osf.io/pqtgf 27 Schuurman, TM., Henrichs, LF, Schuurman, NK, Polderdijk, S., Hornstra, L. (2021). Learning Loss in Vulnerable Student Populations After the First Covid-19 School Closure in the Netherlands, Scandinavian Journal of Educational Research, DOI: 10.1080/00313831.2021.2006307 https://doi.org/10.1080/00313831.2021.2006307 Skar, G.B.U., Graham, S., Huebner, A. (2021). Learning Loss During the COVID-19 Pandemic and the Impact of Emergency Remote Instruction on First Grade Students’ Writing: A Natural Experiment. Journal of Educational Psychology. Advance online publication. http://dx.doi.org/10.1037/edu0000701 Spitzer, M., Musslick, S. (2020). Academic Performance of K-12 Students in an Online-learning Environment for Mathematics Increased During the Shutdown of Schools in Wake of the Covid- 19 Pandemic. EdArXiv. https://edarxiv.org/jncwt/ Storey, N., Zhang, Q. (2021). A Meta-analysis of COVID Learning Loss. EdArXiv. https://doi.org/10.35542/osf.io/qekw2 Suroyo, B., Ibrahim, B., Yanuar. (2021). The Effect of Learning Loss on Student’s Cognitive Development During Pandemic of COVID-19. https://www.researchgate.net/publication/357577206_THE_EFFECT_OF_LEARNING_LOSS_ ON_STUDENT'S_COGNITIVE_DEVELOPMENT_DURING_PANDEMIC_OF_COVID-19 Thorn, W., Vincent-Lancrin, S. (2021), Schooling During a Pandemic: The Experience and Outcomes of Schoolchildren During the First Round of COVID-19 Lockdowns, OECD Publishing, Paris, https://doi.org/10.1787/1c78681e-en Tomasik, M., Helbling, L., Moser, U. (2020). Educational gains of in-person vs. distance learning in primary and secondary schools: a natural experiment during the COVID-19 pandemic 28 school closures in Switzerland. International Journal of Psychology. https://doi.org/10.1002/ijop.12728 UNESCO (2020). UN Secretary-General warns of education catastrophe, pointing to UNESCO estimate of 24 million learners at risk of dropping out. https://en.unesco.org/news/secretary- general-warns-education-catastrophe-pointing-unesco-estimate-24-million-learners-0 Uwezo. (2021). Are Our Children Learning? Illuminating the COVID-19 Learning Losses and Gains in Uganda https://uwezouganda.org/wp-content/uploads/2022/01/Uwezo-2021Report- FINAL_Version.pdf van der Velde, M, Florian, S., Rinske, S., Meeter, M, van Rijn Hedderik. (2021). Lockdown Learning: Changes in Online Foreign-Language Study Activity and Performance of Dutch Secondary School Students During the COVID-19 Pandemic. Frontiers in Education 6. https://www.frontiersin.org/article/10.3389/feduc.2021.712987 Whizz Education. (2021). Measuring Impact of COVID-19 on Learning in Rural Kenya. Whizz Education. https://www.whizzeducation.com/wp-content/uploads/Kenya-Covid-Impact- SCREEN.pdf Widodo, A., Umar, U. (2021). Literacy in the Midst of the Pandemic: How are Students' Literacy Skills? Primary: Jurnal Pendidikan Guru Sekolah Dasar 10 (5): 1199-1206. http://dx.doi.org/10.33578/jpfkip.v10i5.8506 Wolf, S, Aurino, E, Suntheimer, N, Avornyo, E, Tsinigo, E, Jordan, J, Samanhyia, S, Aber, JL, Behrman, JR. (2021). Learning in the Time of a Pandemic and Implications for Returning to School: Effects of COVID-19 in Ghana. https://repository.upenn.edu/cgi/viewcontent.cgi?article=1027&context=cpre_workingpapers 29 Zierer, K. (2021). Effects of Pandemic-Related School Closures on Pupils’ Performance and Learning in Selected Countries: A Rapid Review. Education Sciences 11(6): 252. https://doi.org/10.3390/educsci11060252 30 Appendix 1: Summary of findings of studies excluded from our analytic sample Length of Evidence school of closure learning Average learning loss Source Country (weeks) loss (SD) Amin et al. 2021 Bangladesh YES 26 - ASER 2021 Pakistan summer YES Depping et al 2021 Germany 8 YES -0.18 Empower 2021 United States YES Feng et al 2021 China YES -0.68 Kim et al. 2021 Korea, Rep. YES Kozakowski et al 2021 United States YES Moliner and Alegre 2022 Spain YES Rose et al 2021 England YES -0.17 Sayeejan and Nithlavarnan 2018 Sri Lanka YES Thomas 2021 Austria NO 0 Amelia et al 2020 Indonesia YES -0.11 Amplify 2021 United States YES ASER 2021 India YES Blainey, Hannay 2021 England YES -0.158 Curriculum Associates 2020 United States YES Dorn et al 2021 United States YES -0.16 Falth et al. 2021 Sweden 0 NO 0 Georgiou 2021 Canada YES Juniper Education 2021 United States YES Karimova 2020 Kazakhstan YES Kim et al. 2021 Ethiopia YES -0.11 NAPE 2021 Uganda YES Oktariani et al 2022 Indonesia YES Rahman 2021 Indonesia 32 YES Uwezo 2021 Uganda YES Whizz 2021 Kenya YES -0.36 Czech Korbel and Prokop 2021 Republic YES -0.11 31