NEW SKILLS FOR A NEW CENTURY: Informing Regional Policy T. Shmis, M. Ustinova, D. Chugunov, E. Melianova, S. Parandekar, and L. Kruske © 2021 The World Bank This work is a product of the staff of The World Bank. The findings, interpretations, and conclusions expressed in this work do not neces- 1818 H Street NW, sarily reflect the views of the Executive Directors of The World Bank or Washington DC 20433 the governments they represent. The World Bank does not guarantee Telephone: 202-473-1000; the accuracy of the data included in this work. The boundaries, colors, Internet: www.worldbank.org denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning Some rights reserved the legal status of any territory or the endorsement or acceptance of Rights and Permissions such boundaries. The material in this work is subject to copyright. Because The World Bank encourages dissemination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. Attribution – Please cite the work as follows: “T. Shmis, M. Ustinova, D. Chugunov, E. Melianova., S. Parandekar, L. Kruske. 2021. Russian Federation. New Skills for New Century: Regional policy. © World Bank.” Cover photo: Classroom action in a school of the Moscow City University. © Tigran Shmis Cover page and document design: © Elizaveta Tarasova / ET-digital All queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@worldbank.org. Report No: AUS0002233 6/25/2021 NEW SKILLS FOR A NEW CENTURY: Informing Regional Policy T. Shmis M. Ustinova D. Chugunov E. Melianova S. Parandekar L. Kruske Contents: ACKNOWLEDGMENTS 4 EXECUTIVE SUMMARY 5 1. Preface 9 2. Background 10 3. Definition of and Frameworks for 21st Century Skills 13 4. Research Questions and Methodology 16 5. Results and Discussion 22 5.1 School Physical Environment 22 5.2 School Atmosphere 31 5.3 Teaching and Learning and Student Performance 39 5.4 Nurturing 21st-century Skills in Teaching and How This Impacts TIMSS Results 44 5.5 ICT in Schools and Students’ Information and Communication Literacy 48 5.6 Qualitative Analysis of Modern Teaching in Russian Schools 55 6. Recommendations for Policymakers 59 7. Conclusion 62 REFERENCES 64 ABOUT AUTHORS 68 Annex 1: Tables and Figures 70 Annex 2: A guide for interviewing teachers 74 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice Acknowledgments This study was prepared by a multidisciplinary team led by Tigran Shmis (Senior Education Specialist) under the guidance of Harry Patrinos (Practice Manager, HECED) and Renaud Seligmann (Country Director for Russian Federation, ECCRU). Contributing authors are Maria Ustinova (Education Consultant), Dmitry Chugunov (Education Economist), Ekaterina Melianova (Consultant), Suhas Parandekar (Senior Economist) and Lucy Kruske (Consultant). The team is also grateful to invaluable review and comments provided by Cristian Aedo (Practice Manager, HSAED), Enrique Alasino (Senior Education Specialist, LCRED), and Diego Ambasz (Senior Education Specialist, HECED). The team is grateful to Galina Kovaleva and Olga Sadovschikova who provided help with TIMSS data collection, focus-group interviews, and methodological advice on the study. The team thanks Svetlana Avdeeva for the assistance with ICT Literacy study data collection and analysis. Special thanks to all Russian teachers and school principals who partici- pated in this study and continue to do a great, but at the same time very demanding work of educating Russian children. The report was edited by Fiona Mackintosh. back to table of contents 4 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice Executive summary Countries and development organizations around that students must master to succeed in life and the world are focusing their efforts on adapting work, given the rapidly changing demands of the existing education systems to meet evolving labor 21st century. The report explores how these skills market needs. There is a common understanding are defined in different international institutional that traditional schools are able to equip students frameworks and identifies the nine core skills that with foundational academic skills but need to are mentioned in all of the frameworks, which we do more to help them excel in life. In the World attempt to investigate in depth in this research. Development Report 2018: Learning to Realize Education’s Promise, the World Bank suggested According to the literature, learning 21st- century a framework where a combination of cognitive, skills is critical to ensure students’ future employ- socio-emotional, and technical skills were the ment and success in their careers.1/2 Studies in Russia most relevant skills that education system should have confirmed that having access to workers with develop in students. In this report, we suggest these skills is also critical for the success, growth, using the term 21st-century skills to represent the and productivity of businesses. combination of skills, knowledge, and expertise Twenty-first Century Skills as Defined in International Frameworks World Economic Forum European Commission Scientific Literacy OECD Curiosity Practical / Physical Skills Leadership Literacy Numeracy Financial Flexibility Entrepreneurship Literacy ATC21S Intrapersonal Skills Health and Responsibility Well-Being Initiative Ability & Willingness to Learn Collaboration Interpersonal Skills Communication Creativity & Innovation Productivity Critical Thinking Data / Information Literacy Digital / ICT Literacy Ethics Problem Solving Cultural Awareness Career Civic Literacy Skills Media Literacy Multilingual Competence ISTE Environmental Literacy P21 Religious Values UNESCO Bangkok * Intrapersonal skills include self-awareness, self-management, self-direction, self-efficacy, self-regulation, self-discipline, self-motivation, and persistence. back to table of contents 5 1 https://doi.org/10.3102/0013189X19890600 2 https://link.springer.com/chapter/10.1007/978-981-15-7018-6_7 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice Although Russia has a well-performing and The report confirms that the way in which the efficient mass education system, Russian teaching is delivered, the physical characteris- education policy is not yet focused on developing tics of the learning environment, and the school’s 21st-century skills. Although the international data psychological climate all affect how well students suggest that the performance of Russian students learn. The study identified new aspects of how in terms of collaborative problem-solving skills is teaching style impacts the development of below the OECD average, existing policies do not 21st-century and digital skills as follows: pay enough attention to enabling them to develop this kind of skill. However, the findings of World • Teaching, learning, and student performance. Bank team studies on education equity3, extra- The study shows that most Russian teachers curricular education4, and learning environments5 use the direct instruction style. Modern teaching showed that the existing system might improve practices, such as team teaching and group teaching and learning if it encompassed the work, are also used in Russian schools too but building of 21st-century skills. When we examined have not been widely adopted. The results of the the definition of these skills in the various existing OECD’s School User Survey (SUS) suggest that international institutional frameworks, we found there is a potential to increase the application nine aspects that were common to them all: the of modern teaching methods in Russian schools ability and willingness to learn, collaboration, by providing the necessary resources and communication, creativity and innovation, critical creating the appropriate learning environments thinking, data/information literacy, digital/ICT for conducting such lessons. To find out more, literacy, ethics, and problem-solving. the team created a learning environment index to look at the connection between teaching This study used the data of Trends in Mathe- styles and test scores of Russian students on matics and Science Study (TIMSS), Information the cognitive domains in the Trends in Interna- and Communication literacy (IC literacy), and tional Mathematics and Science Study (TIMSS). School User Survey (SUS) studies combined with The students in schools where teachers practice the qualitative data derived from teacher inter- modern learning styles scored up to 30 test views. The team analyzed the collected data using points higher than their peers in schools where comprehensive statistical methodologies to seek teachers use traditional teaching methods most to find answers to the following questions: of the time. We concluded that both modern and traditional educational models are conducive to • How does the teaching and learning in Russian improving students’ grades not only in math schools relate to the learning outcomes of students and science but also in applying and reasoning in both content and cognitive TIMSS domains as in math and science. The findings also indicated well as in terms of information and communication that aspects of the built environment, including literacy? both the quality of the furniture and the use of technology in the classroom, also improve the • If teaching and learning practices are adapted academic achievement scores of students who to stimulate problem-solving, decision-making, are taught using a modern teaching style. critical thinking, communication and collaboration, and project work, will there be any spillover effects • Twenty-first century skills and cognitive on the subject and cognitive domains of TIMSS? achievement. The team hypothesized that teaching and learning that stimulates • What physical and non-physical characteris- socio-emotional and higher-order cognitive tics of the learning environment in Russian schools skills would produce better scores in the subject affect student learning outcomes? domains of TIMSS 2019 and in the cognitive domains of “reasoning” and “applying.” The • What support do Russian teachers need to team found that, in the case of all skills except practice diverse teaching and learning methods in communication and collaboration, if teachers their work? back to table of contents 6 3 Shmis, T; Parandekar, S. 2018. Education Equity in the Russian Federation: Summary Report (English). Washington, D.C.: World Bank Group. http://documents.worldbank.org/ curated/en/139291530189329351/Education-Equity-in-the-Russian-Federation-Summary-Report 4 Russian Federation - Doing Extra-Curricular Education: Blending Traditional and Digital Activities for Equitable Learning (English). Washington, D.C.: World Bank Group. http://documents.worldbank.org/curated/en/341991561976813788/Russian-Federation-Doing-Extra-Curricular-Education-Blending-Traditional-and-Digital-Activities-for- Equitable-Learning 5 Shmis, T; Ustinova, M; Chugunov, D. 2020. Learning Environments and Learning Achievement in the Russian Federation: How School Infrastructure and Climate Affect Student Success. International Development in Focus; Washington, DC: World Bank. https://openknowledge.worldbank.org/handle/10986/32598 License: CC BY 3.0 IGO Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice fostered them often enough, this had a strong index and a bullying index based on the TIMSS and positive impact on students' subject and student's questionnaire. The analysis showed cognitive scores. The effect was more substan- that students’ perceptions of their safety in tial when we focused on students in the bottom school and their scores in math, applying, and 40 of the socioeconomic distribution. reasoning are related. The less students are bullied, the better their learning outcomes are. • The use of the school environment by teachers Bullying was experienced more frequently by and students and its impact on learning. students from lower socioeconomic house- According to the TIMSS/SUS data, most holds. The difference in math and science scores Russian teachers use only the classroom for between students who are bullied and those teaching. In contrast, the students use class- who are not can be as great as the equivalent rooms in combination with the canteen and the of one year of schooling. Preventing bullying school’s corridors for their learning activities. can solve many problems faced by education In modern schools where the teachers practice systems, and it is important to target anti-bul- group work and team teaching, students and lying efforts to students from the bottom 20 teachers use the common space for group work percent of the socioeconomic distribution. and classroom with direct access to other rooms slightly more often than in traditional schools. • ICT in schools and information and communica- While the teachers confirmed that they were tion literacy. The application of ICT by teachers able to re-arrange the furniture to accommo- in their day-to-day teaching practice is one of date different learning parameters, they did not the predictors of their students' information find it easy to move the technology equipment. and communication literacy. The information The data suggest that the existence of this provided by teachers in the School User Survey equipment varies among schools and that its study showed that students’ most common use still lags behind its availability. ICT-related activity is homework, which is mainly unrelated to teachers. Activities such as collab- • The importance of physical characteristics orations with other students or online research such as air quality, temperature, visual clarity, were rarely reported. The most commonly and auditory comfort. The perceptions of mentioned activities were watching the videos, students and teachers of air, temperature, preparing presentations, creative expression, visual, and auditory comfort remain important and assessing and practicing skills. According and reconfirm the pilot TIMSS/SUS 2019 study, to the students’ self-assessments of their IC while students reported dissatisfaction more literacy skills, most are proficient in all activities, often than teachers. The team created a physical from finding the information that they need, to environment index based on 21 variables of creating texts and presentations, to keeping the physical infrastructure from the OECD’s School computer safe from viruses. It is not confirmed User Survey (SUS), including temperature inside by the actual test though, as most of the the classroom when it is cold/hot outside, visual students in the sample do not reach high level and acoustic comfort, noisiness, smell, and proficiency in IC literacy. More students with a the comfort of desks and chairs. The results of low level of IC literacy never use the computer or the regression analysis confirmed the strong other devices during their leisure time. However, positive relationship between a comfortable there is no correlation between the socioeco- physical environment and student test scores. nomic status of students and their ICT literacy level. These findings show that there is scope for • School environment and climate. A positive increasing students’ ICT literacy. and supportive atmosphere in schools is important for children’s learning. To assess • Qualitative perspective of modern teaching the quality of schools’ psychological environ- and learning in Russia and the impact of ment, the team created a school environment the COVID-19 pandemic. To understand the back to table of contents 7 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice practices that teachers use in Russian schools, • Technology should be made available in schools the team conducted interviews with selected on an equitable basis to improve student learning teachers who had been identified as practicing and enhance teachers’ professional development. modern methods in the OECD’s SUS study. According to these interviews, it became clear • Teachers should be encouraged and supported that team teaching happens in some schools to adopt innovative teaching methods and to share but only once or twice a year and mostly during their experiences among themselves. extracurricular activities. Usually, the team is a combination of teachers of science, and/ • Education policymakers should prioritize the or math, and/or foreign languages (English or prevention of bullying and the development of French). Making team teaching a more common supporting measures to ensure a positive school school practice requires advanced planning climate. and cooperation between teachers and the principal. Group work is often used in daily • COVID-19 is changing how teachers can teach teaching but requires light and flexible furniture and reducing the extent to which they can adopt to easily adjust the classroom or other learning innovative approaches given safety restrictions environments to the group format. The current on circulating throughout the school space and COVID-19 safety measures in Russian schools working in groups. This means that learning losses are decreasing the possibility of circulating are continuing despite the return of students back throughout the school space and of working in to schools. Therefore, new equipment, ICT, and new groups, which means that learning losses are ways of teaching are needed to enable teachers to continuing despite the return of students back improve their practices and compensate for these to schools. learning losses. The analysis in this report shows that schools in Russia could do more to help students excel. Most of our findings point to actions that are imple- mentable and can improve teaching and learning practices and outcomes in Russian schools, in such areas as the curriculum framework, teacher training, and the promotion of a culture of collabo- ration between teachers. . The recommendations discussed in the report are as follows: • High-quality teaching can be delivered using both traditional and modern styles. The effective- ness of each will depend on the specific school context. • Modern teaching practices can be supported by expanding the use of technology and enhancing the quality of furniture in classrooms. • As the Russian education system has sufficient capacity to expand modern teaching, policymakers may utilize existing domestic and international innovative methodologies to promote modern teaching and learning. back to table of contents 8 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice 1 1. Preface As part of the World Bank’s analytical support turn, thought that teaching most socio-emotional program in Russia, the team has been working skills were the responsibility of schools. with their Russian counterparts on collecting and analyzing data in order to gather evidence to This new report presents a rigorous analysis of inform coherent policies and strategies for creating the data collected as part of the Trends in Mathe- modern schools and for promoting 21st-century matics and Science Study (TIMSS) in 2019 comple- teaching and learning in Russia. mented by data on Russia from the OECD’s School User Survey (SUS), which was also fielded in 2019. Both goals are being pursued through a series The team added the SUS to the study as part of of national projects aimed at: (i) modernizing its collaboration with its Russian counterparts. teaching and learning; (ii) putting Russia in the top The total sample represents all general education 10 countries in terms of scores on international systems in the Russian Federation. This report also assessments; and (iii) expanding school infrastruc- includes data from Information and Communica- ture to ensure full-day schooling throughout the tion (IC) Literacy Test7 conducted in 2020 combined country. with the SUS. The two datasets are not related but both cover 21st-century skills and Russian students. Since 2015 when the OECD published a report on collaborative problem-solving skills (CLPS), This report analyzes the various 21st-century skills the topic of these 21st-century skills has gained that it was possible to analyze using data from the traction in Russia. The OECD report devised and TIMSS and IC Literacy Test. The analysis provides presented a comprehensive measure of the ability descriptive statistics related to the learning envi- of students to approach a task as a team and solve ronments and teaching practices in Russia and problems as a group rather than as separate indi- draws some correlations between learning envi- viduals. Since then, collaborative problem-solving ronment variables and learning outcomes. Further- has become the latest international large-scale more, the study shows how different teaching measure of socio-emotional skills in Russia. Shmis styles affect student learning in the mathematics, and Parandekar (2018) found that the level of science, and cognitive (reasoning and applying) CLPS among Russian students was low on PISA domains of TIMSS. The team also analyzed the 2015 compared to OECD average and fell behind wellbeing construct included in TIMSS 2019. They their performance in core PISA subjects. Other found that the cognitive domains of reasoning studies attempted to shed some light on the issue and applying were well aligned with the 21st-cen- of the 21st century skills and the role of schools tury skills that can be measured directly by TIMSS. in their development.6 Dobryakova et al (2018) This study also attempts to use self-reported data found that teachers' goals were oriented towards from TIMSS student and teacher questionnaires fostering students’ subject knowledge and ability and construct measures to stimulate teaching to think critically, analyze, and learn for themselves, and learning activities related to 21st-century skills whereas they viewed collaboration and communi- development. The study also assesses the impact cation, respect for others, and other socio-emo- of such measures on student performance across tional skills as the family’s responsibility. Families, in subject and cognitive domains. In exploring IC back to table of contents 9 6 XXI century skills in Russian school: view from teacher’s and parent’s angle / M. Dobryakova, O. Yurchenko, E. Novikova; Higher School of Economics, Institute of Education. — М.: НИУ ВШЭ, 2018. https://ioe.hse.ru/pubs/share/direct/408113173.pdf 7 The Information and Communication (IC) Literacy Test is designed to measure the level of information and communication (IC) competence of a primary school graduate (Ictlit.com). Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice literacy and ICT technology in schools, the report The qualitative data that the team has collected as analyzes key characteristics of the school environ- part of the study come from interviews conducted ment, the use of ICT in schools, the use of tech- with several teachers from Russian regions who nology by teachers with different mindsets towards use a range of different teaching approaches. modern education, and students' performance on the IC Literacy Test with an analysis of the context Based on the data collected and analyzed in this of the questions. report, the team provides recommendations for policymakers in Russia on how best to develop 21st-century skills among Russia’s 21st-century students. 2 2. Background Technological progress is drastically changing as complex problem-solving, socio-behavioral the economic environment around the world and skills such as teamwork, and skill combinations that having a particular impact on labor markets and are predictive of adaptabilities such as reasoning productivity (World Bank, 2019). The develop- and self-efficacy" are becoming increasingly vital ment of robotics and automation technologies for labor market participation and employability are changing how workplaces are organized and (World Bank, 2019). are even put the existence of certain professions at risk. As a result, there is a demand for new skills In an effort to ensure that this demand can be that workers must acquire and constantly develop met, many countries have started improving their during their working lives to remain productive. national education systems and adapting them to According to the World Bank’s World Development teach these skills. All levels of education need to be Report (WDR) 2019, "advanced cognitive skills such adapted, including preschool, because laying the back to table of contents 10 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice foundations for developing new socio-emotional on outcomes related to critical areas of these and collaborative problem-solving skills has to education reforms throughout all of the regions of begin at a very early age. Russia. Education is a significant contributor to Russia's In 2013, the World Bank conducted an analysis human capital. While the scores of Russian students that demonstrated the importance of 21st-century on various international assessments of math- skills for productivity and employability in Russia ematics, reading, and science ranks the country (Vasiliev et al, 2013). The study found that inno- among the top 25 economies around the world vative employers in Russia are seeking such skills (according to the Harmonized Learning Outcome as problem-solving, communication and collabo- index in 2018 and 2020), they are underperforming ration, and the ability to work independently, but in the area of 21st-century skills. This needs to be the Russian curriculum and teaching approaches remedied because the development of these skills still need to be updated to include those skills. will fuel the competitiveness of Russia's future Unfortunately, as currently designed, the national labor force. projects do not pay sufficient attention to the need to nurture 21st-century skills. However, recent Figure 1: Russia’s Comparative Performance in studies by the World Bank team on education Collaborative Problem-solving Compared to equity, extracurricular education, and learning its Performance in PISA Science, Reading, and environments suggest that teaching and learning Mathematics. may be improved by adapting the school system to Chinese Taipei include 21st-century skills-building. Efforts towards Singapore this end would involve: (i) creating more open and flexible learning environments to enable collabo- Russia -22 ration and communication; and (ii) understanding Macao (China) the extent of students' information and commu- OECD average nication (IC) literacy to tailor their learning experi- United States 22 ences. United Kingdom Turkey Countries need instruments to evaluate the effec- New Zealand tiveness of their learning environments. Learning environment development is an emerging disci- Korea pline, and there is still little empirical evidence Japan 23 on how spaces are currently used and how this Germany impacts student outcomes and teacher produc- France tivity (Blackmore et al, 2011). In 2019, the World Australia 23 Bank team published a comprehensive review -30 -20 -10 0 10 20 30 of research on learning environments from an Source: OECD (2017). evidence-based perspective and found out that the number of empirical studies is still scarce and The Russian Federation is implementing an mostly focused on developed countries (Barrett ambitious set of national projects in the education et al, 2019). This lack of empirical studies on what sector that aim to: (i) put Russia among the very works in learning environments is worrisome given top performers on international student assess- the amount of funding spent on school infrastruc- ments; (ii) introduce the universal full-time school ture globally. model across the country; (iii) increase enrollment in early childhood development; and (iv) build new The Russian government has begun to implement teaching and learning approaches. While these a national project entitled "Contemporary digital are all important goals, it will also be necessary to learning environment in the Russian Federa- systematize good practices and collect evidence tion" to promote digital technologies and online back to table of contents 11 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice learning in primary and secondary education. teachers, the school’s psychological climate, and Focusing on teaching methods, distance learning, the teaching styles applied in these conditions and the promotion of massive online open courses have a notable impact on learning outcomes. The (MOOCs), the project may lead to adaptations to School User Survey had previously been piloted the conventional spatial arrangements of class- in Norway and is currently being used for school rooms and the design of schools in the future. infrastructure assessments in the Belarus Republic These changes will need to be based on a thorough and Chile. However, the Bank’s study was the first evaluation of students' and teachers' needs as well attempt to analyze school infrastructure char- as of the effectiveness and sufficiency of existing acteristics in relation to learning outcomes and learning environments. teaching practices. Russian experts estimate that the quality of the In 2019, the same team had an opportunity to spatial organization and usability of classrooms extend the sample to cover all of the country and in Russian schools are only average as measured thus to produce more reliable results. The new by the internationally recognized tools. For findings further strengthened existing research example, a study of schools in Moscow based on on learning environment quality both nationally the School-Age Care Environment Rating Scale and internationally and contributed to the policy (SACERS) found that Russian schools had the discussion on the most enabling environments for lowest scores on the following parameters: "Space the development of 21st-century skills. and Furnishing," "Physical Activity and Time of Conduct," and "Special Needs," in other words, the The question of 21st-century skills has long been creation of appropriate learning conditions for discussed both in Russia and globally. Therefore, students with special education needs (Ivanova they have been given different names – including and Vinogradova, 2018). The study noted a lack of skills of the future, socio-emotional skills, and alignment between social and pedagogical needs higher-order cognitive skills. In this research, we and existing school designs. The design of new propose to analyze existing definitions or frame- buildings often does not take into account the works and drill down to those skills that are most needs of the pedagogical community, students, essential for Russian national policy. parents, or the municipality, which hinders the development of contemporary learning environ- ments and has a seriously negative effect on the quality and comfort of modern schools. International best practices and a growing pool of scientific evidence suggest that only a holistic approach to school design and planning can guarantee the high quality of education facilities and of the education provided or the well-being of students and teachers (Barrett et al, 2019). In 2018, the World Bank conducted a pilot study on school learning environments in three Russian regions using two international instruments: the OECD’s School User Survey (SUS) and the pilot of the Trends in Mathematics and Science Study (TIMSS) (Shmis et al, 2019). The authors’ findings confirmed that the physical characteristics of a school’s infrastructure, the way that the school is organized and used daily by the students and back to table of contents 12 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice 3 3. Definition of and Frameworks for 21st Century Skills Twenty-first century skills enable individuals to The following frameworks have been included in navigate and thrive in today's rapidly evolving, our analysis: technology-mediated, and interconnected world. The concept of 21st-century skills generally encom- • The Partnership for 21st Century Learning’s passes a combination of: Frameworks for 21st Century Learning • Cognitive skills: Use of logical, intuitive, • The OECD’s The Future of Education and Skills and creative thinking, such as verbal ability, 2030 Learning Framework numeracy, problem-solving, memory, and mental speed. • International Society for Technology in Education (ISTE) Standards • Socio-emotional skills: Beliefs, personality traits, and behavioral skills. • The European Reference Framework of Key Competences for Lifelong Learning • Technical skills: Manual dexterity and use of methods, materials, tools, and instruments, • The World Economic Forum’s New Vision for including those related to specific occupations Education or trades. • Assessment and Teaching of 21st Century Skills Several policy and advocacy organizations (ATC21s) developed frameworks to promote a common understanding and definition of for 21st-century • UNESCO’s Bangkok Asia and Pacific Regional skills and to advocate for them to be prioritized in Bureau for Education's Framework on Trans- school curricula. These research-informed models versal Competencies. usually describe the skills, knowledge, and expertise that students must master to succeed in life and Figure 2 below illustrates the nine skills that are work given the rapidly changing demands of the common to all of the frameworks that we studied.9 21st century. Realizing the importance of these skills As captured in the red square, they are: the ability for the development of a healthy society and skilled and willingness to learn, collaboration, communi- future workforce, a growing number of countries cation, creativity and innovation, critical thinking, have incorporated them into their education data/information literacy, digital / ICT literacy, systems. While there are considerable differences ethics, and problem-solving. in the breadth of competencies included in each framework, there are also significant overlaps. The four skills most commonly mentioned in national policy documents are communication, creativity, critical thinking, and problem-solving.8 back to table of contents 13 8 https://www.brookings.edu/wp-content/uploads/2018/11/Education-system-alignment-for-21st-century-skills-012819.pdf 9 In some cases, the exact terminology used in different frameworks differed but were merged to one common name based on shared characteristics in the definition of the competency. Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice Figure 2: Skills Common to All 21st-century Frameworks Studied World Economic Forum European Commission Scientific Literacy OECD Curiosity Practical / Physical Skills Leadership Literacy Numeracy Financial Flexibility Entrepreneurship Literacy ATC21S Intrapersonal Skills Health and Responsibility Well-Being Initiative Ability & Willingness to Learn Collaboration Interpersonal Skills Communication Creativity & Innovation Productivity Critical Thinking Data / Information Literacy Digital / ICT Literacy Ethics Problem Solving Cultural Awareness Career Civic Literacy Skills Media Literacy Multilingual Competence ISTE Environmental Literacy P21 Religious Values UNESCO Bangkok * Intrapersonal skills include self-awareness, self-management, self-direction, self-efficacy, self-regulation, self-discipline, self-motivation, and persistence. Although some countries and organizations have science to include 21st-century skills such as prob- included the development of 21st-century skills in lem-solving, financial literacy, computer and infor- their strategic education policy objectives, there mation literacy, global competence, and civic and is still no clear answer to the question of how to citizenship education. The dearth of measurement assess these skills as many countries do not have tools might be related to the lack of a consensus on any tools to specifically measure 21st-century what constitutes "21st-century skills," without which skills.10 In some of the frameworks, assessment it is difficult to design a single global instrument to has not been included. For example, the OECD measure the possession of these skills. emphasizes that its Future of Education and Skills 2030 Learning Framework is a learning framework, One way to proceed might be to extend and adapt not an assessment framework and offers a broad existing assessment tools instead of creating new vision of the types of skills that students will assessment tools from the ground up.11 Existing need to thrive in the workplace in 2030 instead assessment instruments –including large-scale of what kind of skills should or can be measured. assessment programs such as PISA, TIMSS, PIRLS, Other organizations, such as ATC21s, acknowl- PIAAC, and ICILS – can capture students’ progress edge that traditional forms of assessment may in learning critical cognitive 21st-century skills and not be suitable for measuring many 21st-century measure the impact of teaching methods designed skills, especially those that might be considered specifically for these skills. After all, the transfer- non-cognitive. However, the International Asso- able nature of 21st-century skills implies that these ciation for the Evaluation of Educational Achieve- skills can be taught through and used in existing ment (IEA) and the OECD have started to expand academic programs.12 their assessments beyond literacy, numeracy, and back to table of contents 14 10 https://neqmap.bangkok.unesco.org/wp-content/uploads/2020/09/Assessment-of-TVC-Current-Tools-in-Asian-Region.pdf 11 https://neqmap.bangkok.unesco.org/wp-content/uploads/2020/09/Assessment-of-TVC-Current-Tools-in-Asian-Region.pdf 12 https://www.brookings.edu/wp-content/uploads/2020/07/OAA-Part-5_Final.pdf Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice Figure 3: Prioritization of the Skills Common to All Frameworks Studied World Economic Forum European Commission Scientific Literacy OECD Curiosity Practical / Physical Skills Leadership Literacy Numeracy Financial Flexibility Entrepreneurship Literacy ATC21S Intrapersonal Skills Health and Responsibility Well-Being Initiative Ability & Willingness to Learn Collaboration Interpersonal Skills Communication Creativity & Innovation Productivity Critical Thinking Data / Information Literacy Digital / ICT Literacy Ethics Problem Solving Cultural Awareness Career Civic Literacy Skills Media Literacy Multilingual Competence ISTE Environmental Literacy P21 Religious Values UNESCO Bangkok * Intrapersonal skills include self-awareness, self-management, self-direction, self-efficacy, self-regulation, self-discipline, self-motivation, and persistence. From among the nine skills that all of the frame- health competency.13 It further emphasizes the works share, we have identified six that can importance of digital literacy and the use of tech- be captured or partly captured by the assess- nology to support learning. Under this project, soft ment tools used in this study: (i) collaboration; (ii) skills are defined as a set of non-subject-specific communication; (iii) critical thinking; (iv) data/infor- knowledge and skills necessary to manage life mation literacy for decision-making; (v) digital/ICT and work successfully. Soft skills, in contrast to literacy; and (vi) problem-solving. professional skills in the traditional sense, are inde- pendent of a particular role or profession and are While Russia does not have a formal framework for closely related to personal qualities and attitudes. 21st-century skills, the National Education Project, Among them are cognitive problem-solving skills launched in 2019, aims to incorporate 21st-cen- (analytical and creative thinking), collaboration tury skills (referred to as soft skills) into its federal skills (teamwork, communication), and self-man- education standards. The project is also developing agement skills (self-regulation and self-organiza- a new methodology for measuring and assessing tion). Many of these soft skills overlap with those student learning of 21st-century skills and basic included in international frameworks that define literacy in areas such as financial, data, legal, and 21st-century skills. back to table of contents 15 13 Meeting Minutes, Project Committee on the Main Direction of Strategic Development of the Russian Federation. Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice There are also some positive ongoing efforts in its educational model is currently being repli- Russia to revise the curriculum for and assessment cated across Russia. of some critical 21st-century skills: • Technological parks Kvantoriums.The main aim • Sberbank: 4K of the contemporary world. of this project is to facilitate the extracurricular Led by the biggest private bank of Russia – education of Russian children in STEM subjects Sberbank – and its charity foundation, "Invest- (science, technology, engineering, and math- ment in the Future," this project focuses on ematics) by implementing effective learning the development of creativity, critical thinking, methods and technologies that can be repli- communication, and cooperation. The aims of cated in all regions of the country.16 the project are to: (i) develop classroom tools and methods to improve the learning process Most of these activities are on a small scale and by including modules on "joint problem solving" have not been evaluated for impact. Further and (ii) develop an instrument to monitor inter- investments will need to be made in conducting disciplinary and indivilual outcomes in these impact evaluations of these initiatives before it will domains.14 be feasible to define effective skill development policies on a national scale. • Sirius school. This is an experimental school for gifted children focused on 21st-century skills As part of this study, our goal has been to under- development.15 It includes a 24-day educational stand what effect the development of these program in the fields of science, technology, selected 21st century skills can have on students’ engineering, arts, and mathematics (STEAM), as performance in cognitive domains using the data well as sports. The school is based in Sochi, but from international student assessments. 4 4. Research Questions and Methodology The current study, building on the previous work, 3. What physical and non-physical characteristics seeks to answer the following questions: of the learning environment in Russian schools affect student learning outcomes? 1. How does teaching and learning in Russian schools relate to the learning outcomes of students 4. What support do Russian teachers need to in both content and cognitive TIMSS domains as practice diverse teaching and learning methods in well as in terms of information and communication their work? literacy? Figure 4 shows the structure of the analysis in 2. If teaching and learning practices are adapted this report. It emphasizes the importance of to stimulate problem-solving, decision-making, improving the learning outcomes and building the critical thinking, communication and collaboration, 21s-century skills of Russian students. The study and project work, will there be any spillover effects shows that this long-term goal can be reached if on the subject and cognitive domains of TIMSS? two necessary pre-conditions are in place, namely: back to table of contents 16 14 https://vbudushee.ru/education/arkhiv-programm-i-proektov/proekt-4k-sovremennogo-mira/ 15 https://sochisirius.ru/o-siriuse/obschaja-informatsija 16 https://www.roskvantorium.ru/en/ Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice (i) modern teacher practices and (ii) supportive and high-quality physical learning environments (see columns 2 and 3 on Figure 4). Based on the evidence that we collected, we identified several important practical steps that can be taken to ensure that these preconditions are put in place, including enabling more teacher training and knowledge exchanges, making efforts to reduce bullying in schools, providing more comfortable and flexible furniture and learning materials, ensuring more equitable access to ICT, and increasing the digital literacy of both teachers and students (Figure 4). Figure 4: Structure of the Analysis Developing and implementing practice-based teaching models to build students’ 21st-century skills. Students’ engagement and performance Supporting the provision of Modern teaching learning materials for modern practices (group work teaching and team teaching) that support the Developing training for development of teachers on bullying students’ 21st-century prevention and providing skills psychological support Teacher productivity services for pupils and motivation Supporting teacher’s knowledge exchanges Improved learning outcomes in reading, math, and science and Ensuring minimum standards better 21st-skills of light, air quality, acoustic comfort, and temperature comfort Positive school climate Promoting flexible learning Physical learning spaces designed to allow environments that are different teaching modalities safe and are conducive Enhancing the quality and to using modern comfort of furniture in teaching styles schools Collaborative behavior Expanding equitable access to technology and increasing its use. back to table of contents 17 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice RESEARCH QUESTIONS AND METHODOLOGY 4.1 Student Performance and Datasets To measure student performance, the World Bank to ensure comparability with other international used the TIMSS 2019 study and the IC literacy results. The TIMSS score ranged between 0 and study carried out in 12 Russian regions. 1,000 points, with 500 points being the average across all countries. Every student did the full set The TIMSS 2019 study was conducted in April-May of tasks included in the test. 2019 and included 4th and 8th grade students in 590 schools in 49 Russian regions. For our study, The IC literacy study was conducted by the we used data for 8th graders only. The 8th grade National Training Foundation (NTF) in 12 regions sample for TIMSS 2019 constituted 3,901 students of Russia and covered 792 schools. The sample for from 204 schools in 43 regions. The overlap in all 12 regions consisted of 14,990 students and samples between the School User Survey (SUS) and 4,962 teachers. The survey and testing took place TIMSS is 84 percent, with 3,725 tested students, between October 1 and November 15, 2020. In 833 associated teachers, and 169 school principals two of the sample regions, some results had been replying to the SUS questionnaire. The outcome collected in the early spring of 2020. results for TIMSS 2019 were weighed and scaled RESEARCH QUESTIONS AND METHODOLOGY 4.2 Data Collection and Methodologies Every student who took the TIMSS in the sampled same day. Their teachers also filled out the teacher schools filled out the SUS student questionnaire. questionnaire. The school-level questionnaire was Teachers of the students who took the TIMSS not collected in this part of the study. testing also filled out the SUS teacher question- naire, and their school principals filled out the SUS The samples for TIMSS 2019 and the IC literacy school questionnaire. The testing in the schools study do not overlap, so all conclusions from these lasted for two days. The TIMSS testing took place two samples will be considered separately. during the first day, and the next day was used to complete the SUS forms. The questionnaires were completed in digital form. The IC Literacy Test was conducted at the school level and was computer-based. Students took the test and filled out the questionnaires during the back to table of contents 18 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice Scheme 1: Data Overlaps TIMSS 2019 School User Survey Student ID Student ID Context questionnaire Student student School User Survey Student ID TIMSS 2019 Teacher ICL context questionnaire Teacher ID Information and Student ID Communication Literacy (ICL) test Student ID School User Survey School User Survey School (Teacher questionnaire) Source: TIMSS, ICL, and OECD SUS frameworks. Given the limitations of the TIMSS study with regard to the 21st-century skills, the team has identified the skills that combined both traditional math and science, which this study refers to as content domains, and the applying and reasoning skills emphasized by TIMSS 2019, which this study refers to as cognitive domains.17 Table 1 shows how the TIMSS framework measures applying and reasoning in math and science. Table 1: Definitions of the Content and Cognitive Domains in TIMSS 2019 Content domains Mathematics Science TIMSS 2019 for 8th grade students Number 30% Biology 35% generalized for Mathematics and Algebra 30% Chemistry 20% Science Geometry 20% Physics 25% Data and Probability 20% Earth Science 20% back to table of contents 19 17 https://timss2019.org/wp-content/uploads/frameworks/T19-Assessment-Frameworks.pdf Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice Cognitive domains Mathematics Science Applying in Mathematics and Focuses on the ability of students Focuses on using this knowledge to compare, Science: focuses on the ability of to apply knowledge and conceptual contrast, and classify groups of objects or students to apply knowledge and understanding to solve problems or materials; relating knowledge of a science concept conceptual understanding to solve answer questions. to a specific context; generating explanations; and problems or answer questions This domain includes the following solving practical problems. skills: This domain includes the following skills: • Determine efficient/appropriate • Compare/contrast/classify, identify, or describe operations, strategies, and tools for similarities and differences between groups of solving problems for which there organisms, materials, or processes; and distinguish, are commonly used methods of classify, or sort individual objects, materials, solution. organisms, and processes based on characteristics • Represent/model and display and properties. data in tables or graphs; create • Relate knowledge of an underlying science equations, inequalities, geometric concept to an observed or inferred property, figures, or diagrams that model behavior, or use of objects, organisms, or materials. problem situations; and generate • Use models, like a diagram or other model to equivalent representations for demonstrate knowledge of science concepts, to a given mathematical entity or illustrate a process, cycle, relationship, or system, or relationship. to find solutions to science problems. • Implement strategies and • Interpret Information and use knowledge of operations to solve problems science concepts to interpret relevant textual, involving familiar mathematical tabular, pictorial, and graphical information. concepts and procedures. • Provide or identify an explanation for an observation or a natural phenomenon using a science concept or principle. Reasoning for Mathematics and It goes beyond solving routine Includes using evidence and an understanding of science Science: goes beyond solving problems to encompass unfamiliar to analyze, synthesize, and generalize, often in unfamiliar routine problems to encompass situations, complex contexts, and situations and complex contexts. unfamiliar situations, complex multistep problems. This domain includes the following skills: contexts, and multistep problems. • Analyze and identify the elements of a scientific problem and use relevant information, concepts, relationships, and This domain includes the following data patterns to answer questions and solve problems. skills: • Synthesize and answer questions that require considera- • Analyze and determine, describe, tion of a number of different factors or related concepts. or use relationships among • Formulate questions/ hypothesize/predict Formulate numbers, expressions, quantities, questions that can be answered by investigation and and shapes. predict results of an investigation given information about the design; formulate testable assumptions; use evidence • Integrate/Synthesize Link different and conceptual understanding to make predictions about elements of knowledge, related the effects of changes in biological or physical conditions. representations, and procedures to • Design investigations and plan procedures appropriate solve problems. for answering scientific questions or testing hypotheses; • Evaluate alternative problem- and describe or recognize the characteristics of well-de- solving strategies and solutions. signed investigations in terms of variables to be measured and controlled and cause-and effect relationships. • Draw conclusions and make • Evaluate alternative explanations; weigh advantages valid inferences on the basis of and disadvantages to make decisions about alternative information and evidence. processes and materials; and evaluate results of inves- • Generalize and make statements tigations with respect to sufficiency of data to support conclusions. that represent relationships in more general and more widely applicable • Draw conclusions and make valid inferences on the basis terms. of observations, evidence, and/or understanding of science concepts; and draw appropriate conclusions that address • Justify by providing mathematical questions or hypotheses and demonstrate understanding arguments to support a strategy or of cause and effect. solution. • Generalize and make general conclusions that go beyond the experimental or given conditions; apply conclusions to new situations. • Justify by using evidence and science understanding to support the reasonableness of explanations, solutions to problems, and conclusions from investigations. back to table of contents 20 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice We believe that the cognitive domains (applying evidence of causality. Rather, we have relied on the and reasoning) in TIMSS are similar to the 21st-cen- theoretical model indicated by the path diagram tury skills described in section 3 of this report. (Figure 23 below) as well as the additional qual- Therefore, the analysis of TIMSS 2019 data in this itative data to suggest the possibility of a causal study includes the performance of students on relationship. Repeated applications of the joint content domains (mathematics and science) and TIMSS-SUS study in the future would generate on cognitive domains (applying and reasoning that a time-series of data that would make it possible are merged for mathematics and science). to confirm or reject causality in the relationship between various school and teacher inputs and This study used multiple analytical techniques, student outcomes. including: The structural equations modeling (SEM) exercise 1. Acorrelation analysis of questionnaire factors and combines confirmatory factor analysis (CFA) with learning outcomes using indexes created by regression analysis as modeled in the path diagram. principal component analysis (PCA). The estimation is performed simultaneously using 2. A regression analysis of the impact of various maximum likelihood estimation. The model posits student-level and school-level characteristics the existence only of the effects displayed in the on student learning outcomes. path diagram; therefore, in this model, the teacher 3. Structural equations modeling (SEM) as multivar- quality cannot influence learning outcomes in iate analysis. the model without impacting their attitudes and 4. Qualitative research that involved identifying beliefs of the students themselves. The reason and interviewing of teachers who use modern why we chose a multigroup SEM procedure was teaching methods. because we wanted to explore differences in the pathways of modern and traditional teaching Using the variables from the context question- styles. In this iterative procedure, we at first naires of TIMSS and School User Survey, the constrained the model to have equal effect param- team constructed a physical environment index, eters and then compared the constrained model to a modern teaching environment index, a school an unconstrained model that allowed the param- atmosphere index, and a bullying index. eters to vary. The models needed to be nested in this way to enable scientifically accurate and valid The report is based mainly on the regression and comparisons of various measures of model good- correlation analyses. The structural equation ness-of-fit.18 modelling is based on conditional correlations that indicate the magnitude of the effect but SEM analysis is based on a model of how the world not causation. A causal model would require a works as it is posited or proposed by the researchers. controlled experiment that would have been Empirical support for the researcher’s hypoth- difficult to perform as it would have required a esis is based on a set of accepted measures that sample of schools with different levels of environ- are built around the concept of goodness-of-fit. mental and teacher quality with students being In using this approach, we knew the actual values randomly assigned to schools and with pre-testing of the dependent variables, and we knew what the and post-testing to determine the causal effect. It model that we built predicts – typically, there will might have been possible to approximate a causal be an error because a model is an abstraction that effect using longitudinal or time-series data of the captures only part of the effects in the real world. single cross-section that we used of the combined Since the differences would be either positive or TIMSS and SUS data. In such a model, estimated negative, they can be added by taking the square values from regressions of earlier cross-sections root of the mean of the squared error terms (RMSE). could serve as instruments for later cross-sections. A model with a lower RMSE is said to dominate a This could be done in the setting of a Bayesian path model with a higher RMSE, but these comparisons model. However, the conditional correlations that can only be done meaningfully for nested models. we have presented do not come with any statistical back to table of contents 21 18 The estimating procedure was conducted with a help of lavaan, which is an R software package for SEM. The missing values were imputed using the Amelia software package that uses a bootstrapped EM algorithm. All of the analyses include controls for students’ socioeconomic status. The R code for this investigation is available here: https://bitbucket.org/zagamog/edsus/src/master/. This code provides end-to-end replicability of the results depicted in this report from the raw data of TIMSS and the SUS. Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice 5 5. Results and Discussion This chapter covers the various areas of analysis that the team used to answer the research questions. These areas are: (i) the physical environment of schools and their impact on learning; (ii) the school environment and climate; (iii) teaching and learning and student performance; (iv) the performance of Russian students in 21st-century skills on TIMSS; (v) the availability and use of ICT in schools and students’ informa- tion and communication literacy; and (vi) a qualitative perspective on modern teaching and learning in Russia. RESULTS AND DISCUSSION 5.1 School Physical Environment A school's physical environment refers to its class- Only 16 percent of teachers reported using the rooms, science laboratories, and sport, leisure, school canteen for teaching purposes at least and administrative spaces inside and outside the two to four times a week, whereas 62 percent educational building within which teacher and of students reported having been taught in the students interact and daily activities occur. The canteen over the previous week. Fifteen percent frequency with which these spaces are used by of teachers reported using the space in a corridor students and teachers can provide us with valuable outside the classroom to teach, while 77 percent of information about the organization and practices students reported having used the same space for of teaching and learning in the school. learning during the previous week. This may point to an important difference in perception of what According to the data from the TIMSS/SUS study, learning means for teachers and students as well 80 percent of Russian teachers use only classrooms as how students really learn in schools. for teaching, while students use classrooms in combination with the canteen and the corridors for their learning activities. Both teachers and students reported that in a typical week during lesson time, they use only their classrooms (see Figure 5 below). Ninety percent of the teachers teach in classrooms assigned to them at least two to four times a week, and 90 percent of students had been taught in a single classroom over the previous week. back to table of contents 22 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice Figure 5: Frequency of Use of Spaces by Students and Teachers in a Typical Week Use of spaces by students during Frequency of use of spaces by lesson time over the previous week teachers in a typical week 13 71 11 6 Sports hall 90 4 32 A workshop space with furniture for woodwork, metalwork, 5 9 28 58 catering, or similar 96 2 7 5 16 72 Kitchen 96 2 A space with furniture or technology for subjects like art, 4 9 29 58 music, or design 95 3 5 16 24 54 Science laboratory 68 16 8 9 62 13 10 15 School cafeteria 81 31 15 9 17 51 23 Hall or auditorium 76 19 42 9 16 47 28 Library 67 22 7 4 77 6 6 11 Space in a corridor outside the classroom 75 7 3 15 Common space for group work (2 and more) of teachers where 8 12 21 60 teachers and students jointly use di erent connected rooms... 80 14 43 A classroom with direct access to other rooms (a cluster of 12 19 19 50 rooms) 69 13 7 10 60 8 9 23 A classroom 7 4 10 80 100 90 80 70 60 50 40 30 20 10 0 0 20 40 60 80 100 Never Once a week 2 to 4 times a week Every day Never Once a week 2 to 4 times a week Every day Source: OECD’s SUS data. In this report, we examine differences between modern teaching styles and traditional ones to understand how school spaces are used in each case. These styles are defined in Table 2 below. Table 2: Definition of Teaching Styles Question Data Source Variable Measurement Details Numbers Traditional style: If students reply that teachers use layouts that support explicit instruction/ presentation or individual work every day and never need to rearrange tables, chairs, or other SUS (Teachers) 24, 26 Teaching style aspects of the space (e.g., sliding partitions) prior to the start of a lesson because a previous user had them in a different position. Modern style: If at least one of the above- mentioned criteria is not met. In modern schools, students and teachers use the common space for group work and classrooms with direct access to other rooms slightly more often than in traditional schools. In terms of the flexibility and adjustability of the spaces (ease of arranging and re-arranging furniture) (see Figure 6 below), 62 percent of teachers agreed that it was easy to move the furniture, while only 38 percent of teachers disagreed with this statement. Sixty-seven percent of teachers also agreed that there was enough space to arrange furniture in different ways, and 44 percent of them agreed that the furniture could easily be moved during lesson time. Teachers did not find it easy to move the technology equipment, such as LCD projectors and whiteboards, to support different teaching arrangements. Only 25 percent reported that they found it easy – 40 percent disagreed, and 35 percent strongly disagreed. back to table of contents 23 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice Figure 6: Teachers' Agreement with Statements about Moving Furniture a) Furniture is easy to rearrange 13 25 44 18 b) Enough time before class to rearrange furniture 11 25 50 14 c) There is enough space in the classroom to arrange furniture in di erent ways 8 25 52 15 d) Furniture can be easily rearranged during the lesson 19 37 35 9 e) Repositioning hardware such as video projectors and 35 40 19 6 interactive whiteboards to rearrange furniture is easy 0 50 100 Strongly disagree Disagree Agree Strongly agree Source: OECD’s SUS data. More than half of all teachers in the sample (64 percent) said that there was enough time to rearrange the furniture before classes began. This is why teachers generally did not encourage students to move around in the space during a class or to move the furniture to suit group work (see Figure 7 below). Only 8 percent of teachers encouraged students to move around the space during a class at least two to four times a week, whereas fewer than a third (28 percent) of the teachers encouraged students to do so one to three times a month. Almost a half (43 percent) of teachers encouraged students to move furniture during class several times per month to facilitate group work or participation in activities, while 11 percent did so at least once a week. Teachers themselves moved furniture around less frequently, either before or during classes. The majority (79 percent) of the teachers never rearranged tables, chairs, or other elements of the space (such as sliding partitions) prior to the start of a lesson, and 10 percent of teachers did so only one to three times a month. Figure 7: Frequency of Teachers Changing the Spatial Arrangements of Teaching Spaces 0 20 40 60 80 100 a) before the beginning of the lesson, rearrange tables, chairs and other interior objects (for example, sliding partitions), because the one who was in this class before 79 10 5 you arranged them in his own way? b) change the layout of the space for the di erent lessons and activities you have planned? (e.g. rearrange 42 43 9 5 tables, put sliding partitions di erently) c) change the layout of the space during the lesson? (e.g. rearrange tables and chairs) 64 25 7 d) Allow students to move furniture during class to form groups or participate in activities? 43 38 11 5 e) allow students to move around the classroom during the lesson? 46 28 9 8 8 Never or almost never 1-3 times a month 1 time a week 2-4 times a week Daily Source: OECD’s SUS data. back to table of contents 24 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice With regard to the use of outdoor spaces for teaching purposes, while 39 percent of students claimed that they had been taught in these areas during the previous week, 95 percent of teachers said that they never teach in these spaces (see Figure 8). Students are using diverse school spaces during their school day, the gap between teachers and students shows a lot of potential for more diverse teaching and learning if teachers would step out from traditional classrooms and utilize spaces more intensively with students. Figure 8: Use of Outdoor Spaces by Students and Teachers Use of outdoor spaces by students during Use of external/outside spaces by teachers class times over the previous week during class times over the previous week 35 16 13 14 22 School yard 66 25 7 9 34 18 19 21 Sports field 87 10 2 An external (outside) hard ball court, 6 20 14 17 43 sports court, hard paved area not 95 4 accessible from a classroom 11 13 13 17 45 Grassy area (not a sports field) not 82 14 2 accessible from a classroom 100 80 60 40 20 0 0 50 100 Never or hardly ever 1 to 3 times a month 2 to 4 times a week Once a week Every day Source: OECD’s SUS data. Technological equipment in schools is an important dimension of education policy in Russia. The data suggest that the existence of this equipment varies among schools and that its use still lags behind its availability. Whiteboards (interactive boards) are not available in the classrooms of 16 percent of teachers, and 72 percent of teachers reported that they had no tablets. The most common pieces of equipment are laptops and desktop computers, which are available in 82 and 84 percent of schools respectively. While a wireless or wired internet connection is generally available, 58 percent of teachers reported a lack of internet access in school spaces. Figure 9: Availability of Technology Equipment and Frequency of Use 0 20 40 60 80 100 a) Interactive display (interactive whiteboard or interactive screen) 24 24 36 16 b) Wireless internet 29 17 27 27 c) Means of transmitting sound and images for a group of students (projector or large TV with audio equipment) 36 32 25 7 d) School laptops / laptops (stored in this room) 18 20 44 17 e) Desktop computers 24 19 41 16 f) Tablets (e.g. iPAd, Surfacebook) 22 72 g) Charging points (for mobile devices) 37 16 25 21 h) Wired internet connection 39 18 32 10 i) Lack of internet access 8 30 58 In all places In most places In some places In none of them Source: OECD’s SUS data. back to table of contents 25 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice As far as the physical characteristics of the learning space are concerned, the results of how respondents perceived temperature were similar to our earlier findings.19 Students and teachers had different percep- tions of temperature. While in 18 percent of cases, students said that they felt too cold in most or all of the spaces, teachers showed more resilience claiming the same only in 7 percent of the cases. A similar pattern is observed in relation to the perception of hot spaces. As research has shown, comfortable temperatures are a strong contributor to students’ academic performance, which is especially important in Russia, which is a cold country in most of its regions. Figure 10: Perceived Temperature in Learning Spaces When It Is Cold Outside 100 6 5 12 12 31 29 30 44 43 42 50 44 38 65 65 40 42 18 15 7 0 Students Teachers Students Teachers Students Teachers I usually feel too cold I usually feel too hot I usually feel about right (neither too hot nor too cold) In none of the spaces In a few of the spaces In most of the spaces In all of the spaces Source: OECD’s SUS data. Students and teachers also differed in terms of their perception of the air quality in learning spaces. While students tended to complain about the air quality, most teachers considered the air to be fresh, and they were less likely than students to notice bad smells (see Figure 11). Fresh air is a key part of the equation for improving students’ learning outcomes, so there is a need to ensure that the air in all learning spaces is fresh. Figure 11: Perceived Air Quality in Learning Spaces 100 11 12 20 22 8 17 57 30 33 27 74 35 50 28 29 50 30 37 22 20 22 8 0 Students Teachers Students Teachers Students Teachers It doesn't smell/the air is usually fresh It smells good It smells bad In all of the spaces In most of the spaces In a few of the spaces In none of the spaces Source: OECD’s SUS data. back to table of contents 26 19 http://hdl.handle.net/10986/32598 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice As far as the lighting of learning spaces is concerned, students tended to claim that light was uncom- fortable more often than teachers. It is surprising that they reported the lighting was too bright more frequently than being too dark. It is also worrisome that up to 20 percent of students reported that they could not see the demonstrations by the teachers when they used traditional boards, LCD projectors, or other technology (see Figure 12). Figure 12: Students’ Perceptions of Visibility in Learning Spaces Lit by Natural Daylight Students: I can see what is drawn or written on the whiteboard or chalkboard without difficulty 57 30 11 Students: I can see what is displayed on the display screen (for example, LCD screen; television screen; 50 30 14 3 projection screen) without difficulty Students: When the teacher is using apparatus for a demonstration, I can see the demonstration 51 28 15 3 without difficulty In all of the spaces In most of the spaces In a few of the spaces In none of the spaces Not applicable Source: OECD’s SUS data. Auditory comfort is important for students to be able to understand what teachers say. The data show that most students could hear the teacher and their peers well, but both students and teachers were often irritated by noise coming from outside the learning space. Perceived auditoryFigure 13: quality byPerceived studentsAuditory Quality and teacher oflearning in Learningspaces Spaces Students: I can hear other students clearly when they are talking to the class 56 28 12 4 Teachers: I can hear the students clearly when they speak 85 14 0 Students: I can hear the teacher's voice clearly 67 24 7 2 Students: I am disturbed by noise from outside the space 8 7 34 51 Teachers: I am disturbed by noise from outside the space 25 73 Teachers: I am disturbed by sounds inside the space (such as airconditioning) 8 90 Teachers: Sound echoes too much in the classroom 7 91 In all of the spaces In most of the spaces In a few of the spaces In none of the spaces Source: OECD’s SUS data. back to table of contents 27 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice To reach a deeper understanding of the data, the team created a physical environment index based on the physical infrastructure variables in the School User Survey (see Table 3). Table 3: Components of the Physical Environment Index Question Answer options 1. When it is COLD outside, what is the temperature in the rooms where you have lessons or where you study on your own? 1) In all of the spaces 2) In most of the spaces a) Usually it's too cold for me there 3) In a few of the spaces b) I'm usually too hot there 4) In none of the spaces c) Usually I feel comfortable there (neither too hot nor too cold) 2. When it is HOT outside, what is the temperature in the rooms where you have lessons or where you study on your own? 1) In all of the spaces 2) In most of the spaces a) Usually it's too cold for me there 3) In a few of the spaces b) I'm usually too hot there 4) In none of the spaces c) Usually I feel comfortable there (neither too hot nor too cold) 3. What is your opinion of the air quality in the rooms where you have lessons or where you study on your own? 1) In all of the spaces 2) In most of the spaces a) Usually it doesn't smell like anything 3) In a few of the spaces b) Usually there is a pleasant smell 4) In none of the spaces c) Usually there is an unpleasant smell 4. When it is light outside, how do you rate the quality of lighting in those rooms where you study in class or on your own? 1) In all of the spaces 2) In most of the spaces a) Usually the light is too bright 3) In a few of the spaces b) It's usually too dark in there 4) In none of the spaces c) Usually there is normal lighting (neither too bright, nor too dark) 5. Do you hear everything well in those rooms where you have lessons or where you study on your own? 1) In all of the spaces 2) In most of the spaces a) I can hear well what the teacher is saying 3) In a few of the spaces b) I can clearly hear what other students say when they answer in the lesson 4) In none of the spaces c) I am disturbed by external noise 6. Can you see everything that is taught clearly in different rooms? a) I can clearly see what they write or draw on the board " 1) In all of the spaces b) I can clearly see what is shown on the screen (LCD screen; TV screen; 2) In most of the spaces movie screen) 3) In a few of the spaces c) When the teacher shows something with the help of a projector, I can see 4) In none of the spaces everything well 7. How comfortable are the desks/tables and chairs in the rooms you use? 1) In all of the spaces a) The chairs are comfortable to sit in. 2) In most of the spaces b) I can choose a chair or adjust the seat height. 3) In a few of the spaces c) I can select a table/desk or adjust its height. 4) In none of the spaces The physical environment index includes 21 strong positive relationship between the physical variables for among others the temperature inside environment in schools and students’ test scores. the classroom when it is cold/hot outside, visual The coefficients of the physical environment and acoustic comfort, noise levels, smell, and the variables were positive and statistically significant comfort of desks and chairs. in all of the model specifications that we tested. We controlled for student gender, age, and soci- The results of the regression analysis confirmed the oeconomic status (SES).20 In models 2 to 3 and 5 back to table of contents 28 20 The SES index combines background information on students’ families such as the parents’ educational attainment, the number of books at home, the possession of a computer or tablet, a study desk, their own room, an Internet connection, their own mobile phone, any musical instruments, a vehicle, an apartment with four or more rooms, and a dishwashing machine. Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice to 6, we also controlled for school territory effects21 A 10 percent improvement in the physical environ- to take into account all unobserved differences ment index is associated with a 0.9-1.0 improve- between schools, which led to the coefficient being ment in TIMSS test scores other things being halved. This is an indication of the homogeneity of constant.22 Students in the top 60 percent of schools in terms of students’ SES, with the differ- schools by the physical environment index scored ence between the average and median student in on average 15 test points higher, which is equiva- terms of their SES being close to zero (-0.03). lent to roughly 3/4 of a year of schooling (Table 4). Table 4: OLS Regressions on Russian Students’ TIMSS-2019 Test scores in Math, Science, Applying, and Reasoning Dependent variable: Math test scores Science test scores (1) (2) (3) (4) (5) (6) Log physical environment index 20.86*** 21.61*** 22.34*** 23.43*** (PEI) (5.14) (4.69) (4.59) (4.42) Log PEI Q2 5.67 7.03 (4.55) (4.29) Log PEI Q3 14.72 13.39*** (4.80) (4.52) Log PEI Q4 15.55 16.33*** (4.78) (4.51) Log PEI Q5 14.33 16.37*** (4.90) (4.61) Fixed territory effects (school) No Yes Yes No Yes Yes Number of obs 1,747 1,747 1,747 1,747 1,747 1,747 Adj R-squared 0.16 0.48 0.48 0.17 0.43 0.43 Dependent variable: Applying test scores Reasoning test scores (1) (2) (3) (4) (5) (6) Log physical environment index 22.70*** 23.59*** 21.01*** 22.14*** (PEI) (4.87) (4.53) (4.62) (4.31) Log PEI Q2 7.65* 6.58 (4.40) (4.18) Log PEI Q3 15.16*** 12.55*** (4.64) (4.41) Log PEI Q4 18.22*** 16.54*** (4.62) (4.40) Log PEI Q5 15.89*** 14.70*** (4.73) (4.50) Fixed territory effects (school) No Yes Yes No Yes Yes Number of obs 1,747 1,747 1,747 1,747 1,747 1,747 Adj R-squared 0.18 0.47 0.47 0.16 0.46 0.46 Source: Authors' estimations based on TIMSS-2019 and SUS-2019 data for Russia. Notes: Standard errors are reported in parentheses. *** Statistically significant at p<0.01. ** Statistically significant at p<0.05. * Statistically significant at p<0.1 back to table of contents 29 21 Unit fixed effects regression models are used for causal inference with longitudinal or panel data (i.e., the data that tracks the same sample at different points in time) in the social sciences (e.g., Angrist and Pischke 2009). Many researchers use these models to adjust for unobserved, unit-specific, and time-invariant confounders when estimating causal effects from observational data. In our study, we use territorial fixed effects by including dummy variables (equal to 0 or 1) for schools in the regressions to capture all unobserved differences between educational institutions. 22 For example, for Math scores in model (1): log(1.1)*20.86=0.863. Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice To depict this difference and its statistical significance, Figure 14 shows the difference in performance between students in schools with a good physical environment and students that reported issues with their schools’ physical environment. Figure 14: TIMSS 2019 Scores by Students in Schools with Good and Bad Physical Environments 560 554 554 555 553 550 550 545 542 538 539 540 537 535 530 525 Math*** Science*** Applying*** Reasoning*** Q1 Q5 Source: Authors’ calculations based on data from the SUS and TIMSS 2019. back to table of contents 30 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice RESULTS AND DISCUSSION 5.2 School Atmosphere Another important factor Figure 15: School Atmosphere Index Components – Gender Differences affecting school life and Teachers are fair to me learning experiences is the atmosphere that prevails in schools. The extent to which I am proud to study in my school students like to be in school or experience hardships or I feel safe in school violence in school will affect their progress in learning and I belong to my school their wellbeing. Schools with a positive and supportive atmosphere for the students I like being in school are those that are safe and free of violence. This kind of 0 20 40 60 80 100 psychological climate is a key girls boys factor in fostering children’s Source: TIMSS survey data. ability to learn. To assess the quality of the psychological atmosphere, we constructed a school atmosphere index based on the TIMSS student's questionnaire. We built the index by using principal component analysis (PCA) on students’ responses on a scale from 1 ("Disagree a lot") to 4 ("Agree a lot") to the following statements: (i) "I like being in school;" (ii) "I feel safe in school;" (iii) "I feel that I belong to this school;" (iv) "My teachers are fair to me;" and (v) "I am proud to study in my school" (see Figure 15). Students’ perceptions of the school’s atmos- the scores of these two groups in applying and phere affect their learning outcomes. Our analysis reasoning, though less strong with a 6 and 3 point showed that this was the case for their TIMSS difference respectively. Again, this confirms that scores on mathematics, applying, and reasoning. there is potential to improve the climate within Scores in math were 11 points higher for students some schools to better support learning (see in those schools where the atmosphere was Figure 16). considered safer and more respectful towards the students than those from schools where the climate felt less accepting and supportive. There was also a statistical significance between back to table of contents 31 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice Figure 16: Student Scores by School Atmosphere Index Quintiles 548 546 543 542 541 Student test scores 539 537 538 535 536 528 Math*** Science Applying** Reasoning* Bottom 20 percent Top 20 percent Source: Authors' calculations based on TIMSS and SUS data. Similar to the school atmosphere index, we also photos); (vi) threatened me; (vii) hurt me; (viii) created a bullying index using principal component excluded me from groups, including online; or (ix) analysis (PCA). The index consisted of students’ damaged my belongings on purpose.” responses on the scale from 1 to 3 (3 meaning at least once a week, 2 meaning one to two times per According to the results, more bullying occurs month, 1 meaning several times per year, and 0 between boys, mostly violent acts and verbal meaning never) to the following nine statements, abuse. Girls spread lies, share each other's secrets, "Other students: (i) made fun of my clothes; (ii) or refuse to talk more often than boys. However, said mean things about me; (iii) shared my secrets the difference in the bullying index between girls with others; (iv) refused to talk with me; (v) spread and boys is not statistically significant (see Figure embarrassing information about myself (including 17 below). Figure 17: Prevalence of Bullying by Gender and Socioeconomic Status of Students a) gender b) socioeconomic status Spread lies about me Spread lies about me Refused to talk Refused to talk Spread lies about me Said mean things Said mean things Refused to talk Shared my secrets to others Shared my secrets to others Said mean things Shared em barrasing photos about m e Shared em barrasing photos about m e Shared my secrets to others Banned (including online) Banned (including online) Shared em barrasing photos about m e Sent mean messages Sent mean messages Banned (including online) Forced to do something Forced to do something Sent mean messages Shared em barrasing info about me Shared em barrasing info about me Forced to do something Hurt me Hurt me Shared em barrasing info about me Insulted family Insulted family Hurt me Stole som ething from me Stole som ething from me Insulted family Damaged property Damaged property Stole som ething from me Threatened Threatened Damaged property Threatened 0 20 40 60 80 0 10 20 30 40 50 60 0 10 20 30 Percent of students Percent of students Girls Boys Percent of st Top 20 percent Bottom 20 percent Top 20 percent Bottom 20 percent Source: OECD’s SUS data. back to table of contents 32 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice More bullying occurs in schools with more students from low SES backgrounds. The SES of students plays a role in the frequency of this bullying as well. The distribution between quintiles shows that the maximum amount of bullying happens to students in the bottom 20 percent of the SES distribution. Figure 18: Bullying Index and Students’ Test Scores, and Socioeconomic Status a. Test scores for bottom 20 percent who b. Bullying index, by SES quintile are not bullied and top 20 percent who 0,04 are bullied 0,02 560 542 541 537 0 540 534 -0,02 Student test score 520 -0,04 500 -0,06 480 -0,08 460 -0,1 440 -0,12 420 -0,14 400 -0,16 Math*** Science** Bottom Second Third Forth Top 20 Bottom 40 percent (not bullied) 20 percent percent (rich) Top 20 percent (bullied) (poo) Source: Authors' estimates based on the OECD’s SUS data. The most common type of bullying is by spreading Figure 19: Bullying by Spreading Lies and lies and rumors, and the index shows that the Spread lies of Learning Outcomes me Students Bullied about impact of this bullying is significant (see Figure 19). 600 The less the students are bullied, the better their learning outcomes are. The students who are bullied every week had the lowest math and science scores 550 549 550 550 545 544 compared to those who suffered from bullying a 543 533 few times per month or per year. The difference was 528 530 as high as 22 points in math scores and 20 points in science, which is equivalent to almost a full year of 500 schooling. Therefore, bullying is not only an issue of Math Science Total security and safety in schools, but it holds students At least once a week Once or twice a month A few times a year back and impairs their ability to learn. Source: OECD’s SUS data and TIMSS scores. Thus, the issue of bullying is a multidimensional A few other factors related to the climate in problem related to safety and security, learning Russian schools emerged from the TIMSS data. For outcomes, and equity. Tackling the issue of instance: bullying can help to solve many problems faced a) Only 62 percent of the sampled schools reported by education systems and targeting the effort to having any facilities or special infrastructure for students from the bottom 20 percent of the SES students with disabilities, while 38 percent said distribution is crucially important. The methods of that they had no such facilities. bullying prevention and encouragement of positive b) Forty-three percent of girls and 37 percent of attitudes among students, development of better boys claimed that they came to school tired every social and communication skills should become a day or almost every day according to TIMSS. focus of teacher training programs. c) Twenty-nine percent of girls and 19 percent of boys said that they feel hungry every day or almost every day. back to table of contents 33 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice These statistics are worrisome and Figure 20: Types of Teaching and Learning Styles as Defined point to the need for education author- by the OECD’s School User Survey ities to ensure that the basic needs of children are satisfied. It will need work with families and schools on tackling the fatigue and hunger. The inclusion needs specific attention as Russian schools a. Presentation b. Group c. Individual d. Team teaching strive to be 100 percent inclusive and accessibility is critical for the school infrastructure. Teaching styles are another important Layouts that support Layouts that support Layouts that support Layouts that support explict instruction or approaches where approaches where approaches where component of schooling that greatly presentation to the students are required students work two or more teachers whole group to collaborate and independently to work collaboratively affect learning outcomes. The OECD's work in small groups write, read, research, with groups of School User Survey categorizes four to share ideas and help each other think, and reflect students sharing the same space types of learning styles and related Teacher Pupils learning spaces: (i) presentation is the Source: OECD’s SUS data. traditional transmission of knowledge via direct instruction; (ii) group work is teaching is when a team of teachers works with a the arrangement of students into smaller groups large group of students. According to this classifi- for discussions and joint work; (iii) individual work cation, these four teaching styles require different is self-paced independent learning; and (iv) team learning spaces and furniture (Figure 20). This full-scale TIMSS/SUS study showed that most common practice. Team teaching is still rare in Russian teachers favor the direct instruction style, Russian schools, with 63.6 percent of teachers using it either every day (77.6 percent) or at least saying that it never happens in their schools, 21.3 two to four times per week (11.6 percent). Individual percent saying it is used one to three times a work is also common for the schools. This shows month, and 2.4 percent saying it is used every day. that the traditional form of teaching prevails in To gain a deeper understanding of team teaching Russian schools. in Russian schools and how teachers perceive it, we asked teachers how they collaborate and use Modern teaching practices, such as team teaching the learning spaces (see Section 6.5). Group work and group work, are used in Russian schools too is used in the sampled schools to different degrees but, as previous studies suggested, are not a but not every day. Table 5: Frequency of Use of Different Teaching Styles by Teachers in their Practice 1-3 times 2-4 per TEACHING STYLE Never 1 per week Every day Missing per month week Direct instruction 1.1 3.1 3.7 11.6 77.6 2.9 (presentation) Small group instruction 20.5 40.3 19.1 12.0 3.2 4.8 Individual learning 28.3 23.5 19.0 15.6 8.8 4.8 Team teaching 63.6 21.3 4.3 2.8 2.4 5.6 Source: OECD’s SUS data. back to table of contents 34 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice The results of SUS survey suggest that there is potential for increasing the application of modern teaching methods in Russian schools. Teachers reported that they would like to use these pedagogical methods if they received the necessary resources and had access to the appropriate learning environments to conduct such lessons (see Table 6). Table 6: Teachers Willing to Implement Traditional and Modern Teaching Styles 1-3 times 2-4 per TEACHING STYLE Never 1 per week Every day Missing per month week Direct instruction 1.2 5.9 11.0 23.2 56.3 2.4 (presentation) Small group instruction 5.4 26.5 29.1 26.7 8.6 3.7 Individual 7.0 22.7 29.4 22.6 14.8 3.6 Team teaching 34.1 42.5 9.7 6.4 2.6 4.7 Source: OECD’s SUS data. When asked about the ability to rearrange spaces This question alone revealed the extent of the in schools, teachers said that few spaces in school deficits in Russian schools. For example, 77 percent allow for a diversity of teaching styles. Most spaces of schools have no common spaces for arranging are designed for direct teaching, though some team teaching, 71 percent have no classrooms with rearrangement is usually possible to accommo- access to breakout spaces or smaller rooms to date group work, but very few of them provide the enable small group work, 46 percent have no art/ necessary flexibility for team teaching. design workshops, 46 percent have no science labs, 38 percent have no workshops, and 35 percent Another question in the SUS asked about the have no kitchens (see Figure 21). availability of certain types of spaces in schools. Figure 21: Types of Spaces Currently Available in Russian Schools According to Teachers, (percentage) Library 94 6 Canteen 91 9 Traditional classroom without access to additional 90 10 / separate rooms Gym 84 16 Space in the hallway outside the classroom 83 17 Hall 79 21 Kitchen 65 35 Workshop 62 38 Science Laboratory 54 46 Art/design workshop 53 47 Traditional classroom with direct access to individual rooms (for example, for team collaboration, project work, or independent work 29 71 Common area for team teaching 23 77 Yes No Source: OECD’s SUS data. back to table of contents 35 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice Table 7: Spaces and Layouts Traditional classroom with Traditional classroom with the Common space for team no access to other rooms / entrance to other spaces (for teaching or for interaction of additional spaces group or individual work) teachers and pupils, for group or individual work Space in the corridor outside Library Hall the classroom Canteen Science laboratory Art / design workshop Kitchen Workshop Sport hall Source of pictures: Letovo school (Moscow oblast), Horoshkola school (Moscow), Chicago School of Woodworking. back to table of contents 36 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice As discussed above, most Russian teachers still findings and expand on them, including the tend to use traditional pedagogical methods, cognitive domains of reasoning and applying. In although some teachers do try to apply modern order to analyze the relationship between modern methods. However, they are often unable to do teaching and learning and student scores, we so because of these inadequate learning facilities. built a modern teaching environment index, again Our findings demonstrate the need to develop using principal component analysis (see Table 8). learning environments that support team teaching This index comprises three variables consisting and group work and to devise other incentives to of: (i) the frequency of use of direct teaching; (ii) encourage teachers to use modern methods more the frequency of use of individual work teaching regularly. practices; and (iii) a dummy variable that equals 1 when furniture is being moved in the classroom Earlier we have established a link between the during lessons (as reported by students). See the existence of a diversity of teaching styles and definition of the traditional and modern styles in student performance. The full sample of TIMSS Table 2 of this report. 2019 in Russia allowed us to look back at those Table 8: Components of the Modern Teaching Environment Index Question Answer options 19. How often do you work in classrooms that match the space/ 1) Never or hardly ever furniture layouts shown in the picture above? 2) 1-3 times a month a) Type A: Furniture arrangement designed for one teacher/ 3) once a week student to speak to the entire group of students 4) 2-4 times a week b) Type B: Furniture arrangement for small group learning 5) Everyday c) Type C: Furniture arrangement for independent student work d) Type D: Furniture arrangement for team teaching (by several teachers) 20. Do you have to rearrange the furniture in the areas where you 1) Yes work most often to ensure that the organization of the space suits 2) No your activities? The results of our regression analysis confirmed the (in models 2 to 3 and 5 to 6) led to the coefficient strong positive relationship between the teaching being halved. This is an indication of the homoge- and learning environment and student test scores. neity of schools in terms of students’ SES, with the The coefficients of the variables for the modern difference between the average and the median teaching environment index were positive and student’s SES index being close to zero (-0.03). statistically significant in all of the model specifi- cations that we tested. We controlled for students; A 10 percent improvement in the learning environ- gender, age, and socioeconomic status. In models ment index is associated with a 1.0-1.2 improve- 2 to 3 and 5 to 6, we also controlled for school ment in TIMSS test scores other things being territory effects to take into account all unob- constant.23 Students in schools where teachers served differences between schools. On this index, practice modern learning styles (top 40 percent boys outperformed girls in all subjects by six to of the modern teaching index, i.e. Q4-Q5) scored seven test points. The SES coefficients were statis- up to 30 test points higher than their peers in tically significant in all of the model specifications, more traditional schools where teachers use direct although controlling for school territorial effects teaching methods most of the time. back to table of contents 37 23 For example, for Math scores in model (1): log(1.1)*53.73=1.106. Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice Table 9: OLS Regressions on TIMSS-2019 Test Scores in Math and Science Dependent variable: Math test scores Science test scores (1) (2) (3) (4) (5) (6) Gender (girl=1) -5.53 -6.36** -5.57* -6.22** -5.79** -4.93* (3.45) (2.88) (2.89) (3.08) (2.71) (2.72) Age 4.55 -0.12 -0.48 7.14** -0.18 -0.60 (3.16) (2.77) (2.77) (2.82) (2.61) (2.61) Log SES index 128.11*** 58.59*** 57.12*** 115.42*** 60.43*** 59.29*** (7.74) (7.64) (7.67) (6.92) (7.20) (7.22) Log modern teaching index (MTI) 26.72*** 23.52*** 26.60*** 24.95*** (5.04) (4.40) (4.50) (4.15) Log LEI Q2 (dummy variable) 4.62 4.55 (4.53) (4.27) Log LEI Q3 (dummy variable) 17.09*** 16.72*** (4.62) (4.35) Log LEI Q4 (dummy variable) 8.82* 14.05*** (4.96) (4.67) Log LEI Q5 (dummy variable) 26.32*** 29.88*** (4.79) (4.51) Fixed territory effects (school) No Yes Yes No Yes Yes Number of obs. 1,747 1,747 1,747 1,747 1,747 1,747 Adj R-squared 0.16 0.48 0.48 0.17 0.43 0.43 Table 10: OLS Regressions on TIMSS-2019 Test Scores in Applying and Reasoning Dependent variable: Applying test scores Reasoning test scores (1) (2) (3) (4) (5) (6) Gender (girl=1) -8.03** -7.96*** -7.18* -6.44** -6.71** -5.95** (3.27) (2.78) (2.79) (3.10) (2.64) (2.66) Age 5.95** 0.14 -0.27 4.79* -0.13 -0.53 (2.99) (2.67) (2.67) (2.84) (2.54) (2.54) Log SES index 127.98*** 64.46*** 62.97*** 114.39*** 51.70*** 50.32*** (7.33) (7.38) (7.41) (6.96) (7.02) (7.05) Log modern teaching index (MTI) 29.66*** 26.47*** 25.97*** 23.96*** (4.77) (4.25) (4.53) (4.04) LEI Q2 (dummy variable) 6.84 5.72 (4.38) (4.16) LEI Q3 (dummy variable) 18.24*** 16.19*** (4.47) (4.25) LEI Q4 (dummy variable) 12.41*** 11.12** (4.79) (4.55) LEI Q5 (dummy variable) 29.79*** 28.56*** (4.63) (4.40) Fixed territory effects (school) No Yes Yes No Yes Yes Number of obs. 1,747 1,747 1,747 1,747 1,747 1,747 Adj R-squared 0.18 0.47 0.47 0.16 0.46 0.45 Source: Authors’ estimations based on TIMSS 2019 and SUS 2019 data for Russia. Notes: Standard errors are reported in parentheses. *** statistically significant at p<0.01. ** statistically significant at p<0.05. * statistically significant at p<0.1. back to table of contents 38 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice This analysis confirms the findings of our previous studies in a more robust way as the larger sample enabled the team to control for the socioeconomic status of the students. With all controls, it is clear that modern teaching can have a strong effect on student performance (see Figure 22). Figure 22: TIMSS 2019 Test Scores by Modern Teaching Environment Index 570 559 557 558 560 553 550 540 533 531 530 529 530 520 510 Math*** Science*** Applying*** Reasoning*** Q1 Q5 Source: Authors’ calculations based on SUS and TIMSS 2019 data. During the study, we also used a deeper analytical lens to try to capture the particular factors that affect the learning outcomes of students as well as to draw causal links among teacher quality, students’ learning environment, their academic achievements, and their wellness. Specifically, this analysis compares the mediation effects of teacher quality and learning space on students’ outcomes through their distinct teaching styles – modern or traditional – in math. RESULTS AND DISCUSSION 5.3 Teaching and Learning and Student Performance In previous chapters, we demonstrated a link our measures of modern and traditional teaching between a diversity of teaching styles and student styles. Since many of these variables are not performance. The TIMSS 2019 dataset provided directly observable, we estimated them as latent us with a representative sample for the entire constructs. We also posit the existence of sequen- Russian Federation that allowed us to expand on tial or structural relationships between these these links. In this section, we explore the relation- latent constructs – for instance, we posit that ships between three variables – teacher quality, the teacher quality affects student motivation, which quality of physical space, and students’ attitudes in turn affects student test scores. Our estima- towards learning, – and how these variables in tion of latent constructs was based on a method turn affected student achievement as measured in called confirmatory factor analysis or CFA, which is TIMSS 2019. very similar to the PCA method used earlier in the analysis. Both are ways to reduce the dimension- We further seek to understand how, if at all, the ality of the measurement of a construct, with CFA nature of these inter-relationships varies between being more appropriate to use in this context. The back to table of contents 39 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice subsequent estimation of regression equations complicated. The complication arises from the using the latent CFA constructs is the so-called unobserved nature of many of the variables. For structural part of the model, and taken together, instance, we do not have a direct or explicit measure the set-up is referred to as structural equation of such constructs as teacher quality or student modelling or SEM (Pearl, 2012). motivation. Rather, we posit that teacher quality is a variable that reflects qualities such as being The findings of the SEM model are laid out in well-organized, having an orderly set of teaching Figure 23, which is called a path diagram in the materials, and being easily understood by their SEM literature. The path diagram includes direc- students when they explain something. This is not tional arrows, which are conceptually equivalent to an exact measure of good teaching, but it seems the left-hand side and right-hand side of conven- reasonable to posit that teaching quality can be tional regression equations. The variable at the meaningfully measured by students’ perceptions end of the arrowhead is the dependent variable in of a teacher’s orderliness and students’ under- the regression, and the variables at the tail end are standing of the material, data on both of which the independent variables in a regression. (Figure are available in the TIMSS dataset. In Figure 23, it 23 uses science scores as the ultimate outcome is clear that, in the case of both modern and tradi- variable, while path diagrams for math scores, tional teaching styles, there is a strong relation- math applying, and student wellness as outcomes ship between teacher quality and positive student are presented in Annex 1.) A path diagram such attitudes. The factors are constructed to have a as Figure 23 is a simple way to present regression mean of zero and a variance of 1, so this means an coefficients of a model that is inherently somewhat effect of nearly 1 for both teaching styles. Figure 23: Path Diagram with Science Scores as an Outcome Modern Teaching Style Traditional Teaching Style Math Important Math Strong Math Important Math Strong Understandable Teacher Quality Math Enjoy Understandable Teacher Quality Math Enjoy Orderliness Orderliness 0.98 0.91 Audibility Audibility Visibility Conditions -0.09 Student Attitudes Visibility Conditions 0.08 Student Attitudes Temperature 0.05 Temperature 0.02 0.12 0.09 0.1 0.03 Safety Science Safety Science 0.11 -0.03 Technology Technology School Space School Space Comfort furniture Comfort furniture Source: OECD’s SUS and TIMSS 2019 data. Note: Blue links depict positive statistically significant effects, while grey links depict statistically insignificant effects. back to table of contents 40 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice Data The data in the path diagram came from the SUS survey and TIMSS study, which were both conducted in 2019 and which we merged in such a way that all observations displayed unique entries representing students so all teacher-related characteristics were averaged by students. Table 11 specifies the measurement details of the variables used in the analysis, while Annex Table 1.1 contains descriptive statistics of the main observed characteristics that we studied. Table 11: Description of the Variables Q№ Variable / Construct Measurement Details TIMSS Math and science scores Reflective factors constructed on the basis of the five - and their subcategories plausible values for each subject. (applying and reasoning) A reflective factor created on five statements about students’ feelings: “I like being at school,” “I like to see my classmates at school,” “I am proud to go to this 13 Wellness school,” “I feel I am part of this school,” and “teachers at the school are fair to me.” The concept sets out students’ overall perception of a school, its teachers, and its activities. A second-order reflective factor is composed of three Students' beliefs and factors representing whether students enjoy math, if 16, 19, 20 attitudes towards they think math is important, and if they are good at mathematics math. A formative construct, which included two reflective 17, 18 Teacher quality factor components depicting if a teacher is understandable and if classes are well-organized. SUS Teachers Traditional style: If students reply that teachers use layouts that support explicit instruction/presentation or individual work every day and never need to rearrange tables, chairs, or other aspects of the space 24, 26 Teaching style (e.g., sliding partitions) prior to the start of a lesson because a previous user had them in a different position. Modern style: If at least one of the above-mentioned criteria is not met. SUS Students The number of learning tasks for which technology 25 Technology devices were used in a typical week. A reflective factor based on manifest variables about 24 Safety feeling at ease in the school. A reflective factor based on items about the comfort 17 Comfortable furniture of desks/tables and chairs in the rooms that students use. A second-order reflective construct composed of three factors: audibility (students hear everything clearly in the rooms where they have lessons), visibility 15, 16, 11, 12 Conditions (students clearly see everything that is shown in different rooms), and temperature (the temperature in the rooms is comfortable). back to table of contents 41 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice The results of the measurement model (the initial The evaluation of the structural models for each construction of latent variables) indicated that the learning outcome also revealed acceptable fit model presented in Figure 23 is based on latent parameters. They encompassed teacher quality variables that are uni-dimensional and valid (Hair components and school environment charac- et al, 2009). All indicators were significantly (p teristics as exogenous variables and students’ < 0.001) loaded on the respective unobserved attitudes and learning achievements as endog- variables and were comparatively balanced. As enous variables. As depicted in Table 12, teacher maximum likelihood estimation is based on a grid quality positively affected all outcomes by culti- search algorithm that seeks to correctly find a vating students’ positive attitudes towards math. global maximum, we examined a measure called Likewise, technology use (operationalized as the composite reliability, which appeared to be larger number of learning tasks for which technology than 0.7 for each construct. This indicated the devices were used in a typical week) demonstrated sound convergent validity of the measurement a positive mediated influence on all outcomes models. The average variance extracted values under consideration. The rest of the effects were for all latent variables were above the required not statistically significant. cut-point of 0.5, signifying the adequate discri- minant validity of the factor solutions. Discrimi- Table 12 (rows 1 through 5) presents the details of nant validity is another measure of the quality of a the model schematically represented in Figure 23 model. Overall, these findings are evidence of the for science scores as an outcome variable. The table good approximating capacity of the models that also presents results for overall math scores, for we used and their internal consistency. the applying and reasoning domains of the math scores, and for the outcome of student wellness. Statistically significant results are highlighted. Table 12: Structural Mediated Associations for Full Sample and Subsamples Full Sample Traditional Teaching Modern Teaching Est SE Est SE Est SE 01 Teacher Quality -> Students’ 0.095 0.013 *** 0.086 0.016 *** 0.119 0.025 *** Attitudes -> Science Comfortable Furniture 02 -> Students’ Attitudes -> 0.000 0.003 -0.003 0.004 0.013 0.008 † Science 03 Safety -> Students’ Attitudes 0.003 0.004 0.002 0.005 0.006 0.009 -> Science 04 Conditions -> Students’ 0.003 0.005 0.007 0.005 -0.010 0.010 Attitudes -> Science 05 Technology -> Students’ 0.005 0.001 *** 0.003 0.002 * 0.012 0.004 ** Attitudes -> Science 06 Teacher Quality -> Students’ 0.203 0.015 *** 0.195 0.018 *** 0.230 0.029 *** Attitudes -> Math 07 Comfortable Furniture -> -0.001 0.007 -0.008 0.009 0.023 0.015 Students’ Attitudes -> Math 08 Safety -> Students’ Attitudes 0.009 0.009 0.006 0.010 0.013 0.018 -> Math 09 Conditions -> Students’ 0.006 0.010 0.015 0.012 -0.022 0.020 Attitudes -> Math 10 Technology -> Students’ 0.011 0.003 *** 0.007 0.003 * 0.023 0.006 *** Attitudes -> Math 11 Teacher Quality -> Students’ 0.348 0.025 *** 0.358 0.030 *** 0.327 0.042 *** Attitudes -> Wellness back to table of contents 42 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice Full Sample Traditional Teaching Modern Teaching Est SE Est SE Est SE Comfortable Furniture 12 -> Students’ Attitudes -> 0.003 0.012 -0.011 0.015 0.035 0.020 † Wellness 13 Safety -> Students’ Attitudes 0.015 0.014 0.009 0.018 0.018 0.024 -> Wellness 14 Conditions -> Students’ 0.015 0.017 0.034 0.022 -0.027 0.027 Attitudes -> Wellness 15 Technology -> Students’ 0.019 0.005 *** 0.014 0.006 * 0.032 0.008 *** Attitudes -> Wellness 16 Teacher Quality -> Students’ 0.199 0.015 *** 0.189 0.017 *** 0.230 0.028 *** Attitudes -> Applying Comfortable Furniture 17 -> Students’ Attitudes -> 0.000 0.007 -0.008 0.008 0.024 0.015 Applying 18 Safety -> Students’ Attitudes 0.009 0.009 0.005 0.010 0.013 0.018 -> Applying 19 Conditions -> Students’ 0.006 0.010 0.015 0.012 -0.022 0.020 Attitudes -> Applying 20 Technology -> Students’ 0.011 0.003 *** 0.007 0.003 * 0.023 0.006 *** Attitudes -> Applying 21 Teacher Quality -> Students’ 0.204 0.016 *** 0.197 0.018 *** 0.226 0.029 *** Attitudes -> Reasoning Comfortable Furniture 22 -> Students’ Attitudes -> 0.000 0.007 -0.008 0.009 0.023 0.015 Reasoning 23 Safety -> Students’ Attitudes 0.009 0.009 0.006 0.010 0.013 0.017 -> Reasoning 24 Conditions -> Students’ 0.006 0.010 0.015 0.012 -0.021 0.020 Attitudes -> Reasoning 25 Technology -> Students’ 0.011 0.003 *** 0.007 0.003 * 0.022 0.006 *** Attitudes -> Reasoning Sig: ***p < 0.001, ** p < 0.01, * p < 0.05, † p < 0.1; Blue tint refers to the Figure 23 The results showed that the positive effects of both the quality of the furniture and the use of technology on math, science, applying scores, and technology in the classroom – provides additional wellness were greater in magnitude for the modern achievement-related benefits to students who are style than for the traditional teaching model. The taught using a modern teaching style. Within the comfort of classroom furniture is associated with confines of the modeled environment, these are higher academic achievements in science and causal relationships. wellness, but this applies only to students exposed to modern learning, whereas for the traditional style model, these associations were not shown to differ in the general sample. No statistically signif- icant group discrepancies were identified in the effect of teacher quality on students’ academic outcomes and wellness. The findings allow us to conclude that both modern and traditional educational models are conducive to producing better grades in math and science and also to enhancing the applying and reasoning of mathematical knowledge. The findings also indicate that the built environment – including back to table of contents 43 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice RESULTS AND DISCUSSION 5.4 Nurturing 21st-century Skills in Teaching and How This Impacts TIMSS Results In this carrying out analysis, we have had to contend questions were related to certain teaching and with a lack of data and instruments for assessing learning practices that support the development students’ acquisition of 21st-century skills. TIMSS of five specific 21st-century skills, namely: prob- 2019 itself does not directly measure socio-emo- lem-solving, decision-making, critical thinking, tional skills, but the TIMSS categories of reasoning communication and collaboration, and project and applying represent higher-order cognitive work. The detailed distribution of the questions skills, so we chose to include them in our analysis. and their coding is provided in Table 13 below. The TIMSS questionnaire also asks a lot of process questions of teachers and students, the answers We also constructed indexes for each of these skills to which have helped us to make inferences about with variables corresponding to the questions in certain kinds of behavior and abilities that teachers TIMSS 2019 and correlated them with the cognitive nurture in their students that relate to 21st-century and subject domains of TIMSS 2019. The indexes skills. were created using teachers' responses about how often they took certain actions during classes as In this section, we discuss these teaching practices self-reported in their replies to the TIMSS ques- that are related to the stimulation of higher-order tionnaire. Students were split into two groups using cognitive and social skills. We conducted an analysis the median values of the indexes for the frequency of the teacher and student context questionnaires of actions taken by teachers during their classes. provided by TIMSS for the 8th-grade study. The Table 13: Twenty-first Century Skills and Teaching Practices in Schools Skill description Questionnaire type Question and response option 12. How often do you do the following in teaching this class? c) Ask students to complete challenging exercises that require them to go beyond the instruction Teacher questionnaire (mathematics) 15. In teaching mathematics to this class, how often do you ask students to do the following? g) Work on problems for which there is no immediately obvious solution Problem-solving 12. How often do you do the following in teaching this class? C) Ask students to complete challenging exercises that require them to go beyond the instruction. Teacher questionnaire 15. In teaching this class, how often do you ask students to (science) do the following? b) Observe natural phenomena and describe what they see e) Conduct experiments or investigations k) Use formulas and laws to solve problems back to table of contents 44 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice 12. How often do you do the following in teaching this class? F) Ask students to come up with their own problem- Teacher questionnaire solving procedures (mathematics) 19. C. How often do you do the following with the mathematics homework assignments for this class? b) Have students correct their own homework Decision making 12. How often do you do the following in teaching this class? F) Ask students to come up with their own problem- Teacher questionnaire solving procedures (science) 18. C. How often do you do the following with the mathematics homework assignments for this class? b) Have students correct their own homework 12. How often do you do the following in teaching this class? A) Relate the lesson to students’ daily lives E) Link new content to students’ prior knowledge G) Encourage students to express their ideas in class Teacher questionnaire 19. C. How often do you do the following with the (mathematics) mathematics homework assignments for this class? b) Have students correct their own homework 23. In the past two years, have you participated in professional development in any of the following? e) Improving students’ critical thinking or problem- solving skills 12. How often do you do the following in teaching this Critical thinking class? A) Relate the lesson to students’ daily lives E) Link new content to students’ prior knowledge G) Encourage students to express their ideas in class 15. In teaching this class, how often do you ask students to do the following? Teacher questionnaire (science) h) Use evidence from experiments or investigations to support conclusions 18. C. How often do you do the following with the homework assignments for this class? b) Have students correct their own homework 22. In the past two years, have you participated in professional development in any of the following? e) Improving students’ critical thinking or inquiry skills back to table of contents 45 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice 12. How often do you do the following in teaching this class? B) Ask students to explain their answers Teacher questionnaire D) Encourage classroom discussions among students (mathematics) 15. In teaching mathematics to this class, how often do you ask students to do the following? h) Work in mixed mathematical ability groups i) Work in same mathematical ability groups 15. In teaching this class, how often do you ask students to do the following? G) Interpret data from experiments or investigations Teacher questionnaire 15. In teaching this class, how often do you ask (science) students to do the following? m) Work in mixed ability groups n) Work in same ability groups Communication 12. Do you use the Internet to do any of the following and collaboration tasks for schoolwork (including classroom tasks, homework, or studying outside of class)? c) Collaborate with classmates on various projects 16. To what extent do you agree/disagree with the following statements about your mathematics lessons? n) My teacher asks us to discuss with each other how to solve problems Student questionnaire 21. To what extent do you agree/disagree with the following statements about your biology lessons? n) My teacher encourages me to discuss the results of our biology experiments 25. To what extent do you agree/disagree with the following statements about your earth science lessons? n) My teacher encourages me to discuss the results of our earth science experiments 20. How much importance do you place on the Teacher questionnaire following assessment strategies in mathematics? (mathematics) e) Long-term project 19. How much importance do you place on the Project work following assessment strategies in science? Teacher questionnaire e) Long-term project (science) 15. In teaching this class, how often do you ask students to do the following? d) Plan experiments or investigations Source: Authors’ compilation based on TIMSS 2019. back to table of contents 46 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice Once we had identified the five skills from the teacher questionnaire, we looked at the effect of the teaching activities designed to stimulate the acquisition of those socio-emotional and higher-order cognitive skills on student learning outcomes. Our hypothesis was that these activities would be producing better learning outcomes in the subject domains of TIMSS 2019 and in the cognitive domains of reasoning and applying. Figure 24 shows the results of the correlations based on the categorization of the use of certain teaching and learning practices as “rare” or “often.” It is clear that if teachers use these practices often, then the effect on the subject and cognitive outcomes of students is positive and strong in all skills except communication and collaboration. Figure 24: The Interdependence of Stimulated 21st-century Skills and TIMSS 2019 Scores 560 555 550 545 540 535 530 525 520 515 Applying*** Applying*** Applying*** Applying*** Math Applying Math*** Reasoning*** Math*** Reasoning*** Math*** Reasoning*** Math*** Reasoning*** Reasoning Science*** Science*** Science*** Science*** Science Problem solving Decision making Communication and Critical thinking Project work collaboration Rare Often Source: Authors’ estimates based on the TIMSS 2019 data. While there is a strong correlation between the indexes representing the stimulated skills (apart from the communication and collaboration) and the cognitive and subject domains of TIMSS 2019, we were also interested to see if this correlation held in the case of students from families with low SES. We found a stronger correlation across all tested indexes, including communication and collaboration, but unfortu- nately, the scores of students from the bottom 40 percent of the socioeconomic distribution were far lower than the average scores in TIMSS 2019 (see Figure 25). Figure 25: The Interdependence of Stimulated 21st-century Skills and the TIMSS 2019 Scores of Students in the Bottom 40 percent of the Socioeconomic Distribution 535 530 525 520 515 510 505 500 495 490 485 Applying*** Applying*** Applying*** Applying*** Applying*** Math*** Reasoning*** Math*** Reasoning*** Math*** Reasoning*** Math*** Reasoning*** Math*** Reasoning*** Science*** Science*** Science*** Science*** Science*** Problem solving Decision making Communication and Critical thinking Project work collaboration Rare Often Source: Authors’ estimates based on the TIMSS 2019 data. back to table of contents 47 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice What we conclude from the findings of this analysis is that helping students to acquire 21st-century skills plays a key role in enhancing students’ academic performance, especially those from poorer socioeco- nomic backgrounds. Implementing these teaching and learning practices at scale has the potential to generate enormous gains in learning outcomes in Russia. RESULTS AND DISCUSSION 5.5 ICT in Schools and Students’ Information and Communication Literacy ICT is widely used in Russian schools, but any and reliable learning materials. This platform will issues with availability or connectivity have been continue to be developed to ensure the education exacerbated by the abrupt transition to distance system has the resilience to withstand any future learning because of the COVID-19 crisis. In this crises. process, the Russian government relied on its previous experience with equipping all schools The use of ICT by teachers in their day-to-day with internet connectivity and reforming the practice is one of the predictors of their students’ curriculum towards the use of ICT in teaching. IC literacy. The statistics provided by teachers in the While hardware was widely available, software School User Survey showed that students’ most support and learning platforms were not well-de- common ICT-related activity is homework, which is veloped. The most reliable platform that was mostly unrelated to teachers, and they very rarely being used by Russian teachers was the Moscow use ICT for such activities as collaborating with Electronic School (https://hundred.org/en/inno- other students or doing online research. Other ways vations/moscow-electronic-school#1a343044). in which they use ICT are for watching the videos, The Ministry for Education also quickly created preparing presentations, creative expression, and the Russian Electronic School (https://resh.edu.ru/) assessing and practicing skills, most of which they to respond to the sudden massive need for quick do at least once a week (Table 14). Table 14: Frequency of Use of ICT by Students in Different Activities Never Once a week 2-4 time a week Every day a) Online research 54.28 33.25 10.71 1.76 b) Complete an assessment task 21.13 43 25.38 10.5 c) Watch a video 7.49 40.95 33.08 18.48 d) Listen to audio 35.89 31.86 20.4 11.84 e) Complete homework 18 29.13 19.88 33 f) Practice skills 15.38 55 24.38 5.25 g) Express ideas creatively 20.18 55.33 18.65 5.84 h) Prepare presentations/reports 5.74 64.71 22.69 6.86 i) Collaborate with students in other schools 83.86 11.6 3.15 1.39 j) Collaborate with students in other countries 96.46 2.02 1.14 0.38 Source: Authors’ estimates based on the OECD’s SUS data. back to table of contents 48 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice In terms of the devices and equipment available for teaching and learning, the availability is strongly corre- lated with the use. The most widely available and used devices are projectors and video players. Tablets, laptops, and charging points are not widely available in Russian schools. And tablets are the least used devices. This accords with our finding that the traditional way of teaching – demonstration in classrooms – is the most common teaching approach used in Russian schools. Figure 26: Availability and Frequency of Use of Types of Technology by Modern Teachers in their Practice Availability Frequency a) Interactive display (interactive whiteboard 27 21 38 14 or interactive screen) 36 7 18 34 32 14 32 22 b) Wireless internet 42 8 10 36 c) Means of transmitting sound and images for a 37 33 27 3 group of students (projector or large TV with audio 8 15 9 31 38 equipment) d) School laptops / laptops (stored in this 19 23 43 15 room) 46 17 8 22 26 24 38 12 e) Desktop computers 34 13 7 12 34 43 25 68 f) Tablets (e.g. iPAd, Surfacebook) 80 8 64 44 18 22 16 g) Charging points (for mobile devices) 35 7 9 42 39 16 35 10 h) Wired internet connection 34 6 10 47 0 20 40 60 80 100 0 50 100 In all places In most places Never or almost never 1-3 times a month In some places In none of them Once a week 2-4 times a week Source: Authors’ estimates based on the OECD’s SUS data. At the same time, when we correlated the frequency of technology use in schools with student scores on TIMSS, the results were positive and statistically significant for all technology types. Figure 27: Frequency of Technology Use and Learning Outcomes by Technology Type a) Interactive display (interactive whiteboard or b) Wireless internet interactive screen) 558 560 556 555 553 552 555 552 550 551 550 550 547 545 545 539 542 540 539 538 540 541 536 540 538 535 530 535 525 530 520 Often Rare Often Rare Often Rare Often Rare Often Rare Often Rare Often Rare Often Rare Math Science Applying Reasoning Math Science Applying Reasoning c) Projector or large TV with audio equipment f) Tablets (e.g. iPAd, Surfacebook) 560 556 575 573 552 554 550 567 569 570 550 564 565 540 560 530 555 521 550 547 519 518 518 545 545 520 545 542 510 540 535 500 530 490 525 Often Rare Often Rare Often Rare Often Rare Often Rare Often Rare Often Rare Often Rare Math Science Applying Reasoning Math Science Applying Reasoning Source: Authors’ estimates based on the OECD’s SUS and TIMSS 2019 data. back to table of contents 49 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice The data suggest that, when teaching is supported computer or a device. After completing the assess- by technology, learning outcomes improve. In ment, each test taker receives recommendations section 5.4, we showed that technology is a on how to improve his or her IC literacy. Educators positive contributor to student learning outcomes then use the summarized test results to improve in science and their wellness. However, the main the education provided in each classroom and in factor in that case was the use of the modern the entire region. teaching approaches and the overall quality of teaching. We also attempted to understand the In 2019, the IC Literacy Test was administered in 11 relationship between students’ learning outcomes regions of Russia24 on a sample of 12,060 grade 9 and their IC literacy. So, we turned to the data from students. A subset of 4,958 students also filled out Russia’s IC Literacy Assessment. IC Literacy Test is the SUS questionnaire related to the learning envi- an instrument that measures how well students ronment. Because of the data entry errors (dupli- transitioning from middle school to high school cations or incorrect student IDs), the complete IC (at the end of grade 9 in Russia) use modern IC Literacy/SUS dataset comprises information only technologies in their communication and research about 3,952 students. The distribution of test- activities. The test assesses the level of student takers by age and gender is provided in Tables 15 skills to operate in ICT rich environment rather than and 16. their computer literacy or ability to operate the Table 15: Distribution of Students Taking the IC Literary Test by Age Age of students Number of students As percentage of total Cum. 12 33 0.84 0.84 13 22 0.56 1.39 14 877 22.19 23.58 15 2,792 70.65 94.23 16 228 5.77 100 Total 3,952 100 Table 16: Distribution of Students Taking the IC Literary Test by Gender Gender Number of students As percentage of total Cum. Girls 2,027 51.29 51.29 Boys 1,925 48.71 100 Total 3,952 100 The majority of students (85 percent) said that they use computers only during information tech- nology classes even though they believed that the use of computers in school facilitates learning. Also, 40 percent of students said that it was important for computers to be used by students them- selves rather than just by teachers. In all, 53 percent of students said that they used computers often or in every lesson to search for information in the Internet; 42 percent said they used them to prepare course work; 35 percent for creating presentations; 35 percent for conducting independent work; 33 percent for watching videos; 33 percent for doing group work with classmates; 23 percent for using training simulators; and 20 percent for conducting experiments or doing lab work. Computers are most often used by teachers for projecting presentations (as stated by 71 percent of students) and for demonstrating learning material (as stated by 67 percent of students). back to table of contents 50 24 Altai, Kaliningrad, Krasnoyarsk, Novosibirsk, Pskov, Bashkortostan, Rostov, Tyumen, Chelyabinsk, Yamalo-Nenets Autonomous Okrug, and Yaroslavl. Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice According to the students’ self-assessment of their own IC literacy skills, the majority of them are profi- cient in all activities, from finding the information they need to prepare texts and presentations to keeping the computer safe from viruses (Figure 28). Figure 28: Self-assessments by Students of Their IC Literacy Skills Very q uickly find th e words of a song on the Internet 94% Quickly un ders tand if the site con tains th e information you need 94% Distinguish a site wh ere you can download a son g for free from the one that you req uire 93% Use Intern et sources for homework 92% Fin d the sites from which you b orrowed materials for you r school project 92% Quickly fin d a file th at you d own loaded fro m th e Internet and saved on your comp uter for several 91% Use information from several sites to find out the truth ab out your life 89% Keep yo ur compu ter safe from viruses 88% Design (format) documents in Word so that they are easy to pars e 87% Describe you r interests on Facebook or VKontakte so that people with similar interests 85% Create beautiful an d meaningful PowerPoint presentations 85% Distinguish sites where you can infect your computer with a virus from thos e that do n ot 85% Use search on sites that are NOT search engines, that is, NOT Google or Yand ex 78% 0% 50% 100% Source: Authors’ estimates based on the OECD’s SUS data and IC Literacy Test. Note: Answers categorized by “can perform very well,” “can perform well,” and “able to perform.” back to table of contents 51 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice The IC literacy test produces the results in five levels ranging from developing to the advanced. Table 17 below provides a quick description of student abilities per level. Table 17: Description of IC levels A student who has an advanced level of IC literacy works at a high level with tasks that require the ability to: • correctly formulate the problem, • find information from various sources, Advanced level • organize information according to specific criteria, • evaluate the quality of information and the reliability of its sources, • compare and summarize information from different sources, • make correct conclusions based on existing information, • transfer information to other people. A student who has an above intermediate level of IC literacy works at a high level with tasks that require the ability to: • correctly formulate the problem, • find information from various sources, • organize information according to specific criteria, Above Intermediate level • make correct conclusions based on existing information, • transfer information to other people, but can only satisfactorily cope with tasks requiring the ability to: • evaluate the quality of information and the reliability of its sources, • compare and summarize information from different sources. A student who has an intermediate level of IC literacy satisfactorily copes with tasks that require the ability to: • correctly formulate the problem, • find information from various sources, Intermediate level • organize information according to specific criteria, • evaluate the quality of information and the reliability of its sources, • compare and summarize information from different sources, • make correct conclusions based on existing information, • transfer information to other people. A student with an intermediate level of IC literacy satisfactorily copes with tasks that require the ability to: • correctly formulate the problem, • find information from various sources, • organize information according to specific criteria, Below Intermediate level • make correct conclusions based on existing information, • transfer information to other people, but may not cope with tasks that require the ability to: • evaluate the quality of information and the reliability of its sources, • compare and summarize information from different sources. A student who has a developing level of IC literacy cannot solve problems corresponding to the level of "below average", that is, he cannot cope with tasks that require the ability: • correctly formulate the problem, Developing level • find information from various sources, • organize information according to specific criteria, • make correct conclusions based on existing information, • transfer information to other people. back to table of contents 52 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice To make it easy to interpret results we proposed experienced region where the IC Literacy testing the groupings of levels by low, average, and high was conducted two times on a representative IC Literacy levels. Based on the description above sample. In addition, Krasnoyarsk region (along we suggest including developing level as low, since with Perm, Kaluga, and Stavropol regions) partic- this level demonstrates functional illiteracy. The ipated in the E-Learning Support Project25 (ELSP) average level includes intermediate and below in 2004-2008 financed by the loan of the World intermediate. Lastly, high level unifies above inter- Bank. The project supported the development of mediate and advanced levels. digital learning materials, teacher training in ICT, and creation the regional support networks for According to the results of the IC Literacy testing, urban and rural areas that developed capacity of 24 percent of students from participating regions school systems in ICT education. Therefore, some are not proficient in accessing and using infor- regions of Russia could be better prepared for such mation with means of modern technologies. On test. average only 11 percent of students scored high. The distribution of scores by regions of Russia is This study hasn’t identified differences in IC literacy presented in the figure below. Novgorod region between boys and girls. It might be an indication of is leading because the sample is skewed towards the inclusiveness of ICT in Russian schools or equal students from urban schools. Krasnoyarsk is an access to ICT in Russian families. Figure 29: IC Literacy levels per participating Russian regions 100% 14.5% 12.3% 12.9% 8.8% 11.4% 11.2% 9.8% 10.6% 9.7% 10.6% 10.8% 18.4% 75% 64.3% 67.9% 64.5% 64.6% 66.1% 64.9% 64.7% 61.8% 61.4% 67.6% 67.7% 50% 65.2% 25% 20.0% 22.8% 23.2% 24.0% 24.1% 24.2% 24.4% 25.6% 27.6% 27.8% 16.5% 17.9% 0% ns n n on n on on n on on on on o o o o io gi gi gi gi gi gi gi gi gi gi gi eg re re re re re re re re re re re sr ol od n k rm y in k ga o h rs sk ns ha ez ov al p s lu r Pe ya ro ro od kh ya go n er ak Ka ro ac av no Br m Sa ov or tr Vo St Ke As as e eg N ag Kr izh er N Av low average high Source: Authors’ estimates based on the IC Literacy Test. Distribution of IC Literacy scores by school type Distribution of IC Literacy scores by students’ aspi- demonstrates that there is a higher share of rations to higher education shows that student students with low IC literacy levels in regular motivation to enter university is a good predictor schools (21 percent) compared to specialized of IC Literacy scores. Students that plan to enroll schools (9-13 percent). Students in gymnasiums in higher education scored higher – 19 percent and lyceums have higher shares of students in with high proficiency level – almost twice more the top proficiency level (22 percent) compared to than those who don’t intend to continue beyond regular schools (14 percent). The share of students secondary education. in the medium/average proficiency level is similar across all school types. back to table of contents 53 25 https://projects.worldbank.org/en/projects-operations/project-detail/P075387 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice Figure 30: IC Literacy levels by type of school and attitudes towards higher education a) Type of school and IC levels b) Intention to enroll in university 100% 100% 14% 12% 10% 22% 22% 19% 27% 75% 75% 65% 61% 50% 66% 64% 66% 64% 50% 65% 25% 25% 21% 13% 12% 9% 0% 30% Gymnasium Lyceum School with in- School 22% 15% depth study of individual 0% subjects Yes Don't know No low average high low average high Source: Authors’ estimates based on the IC Literacy Test. A high share of students at low IC literacy (34 percent) never uses computers or other gadgets during leisure as compared to 24, 26, and 24 percent respectively for students who use gadgets rare, often or all the time. So, usage or/and access to devices may define a gap of up to 10 percentage ponts between those who have low proficiency. Figure 31: Impact of Using IT for Leisure and Homework on Students’ IC Literacy How often do you use computer or other How often do you use computer or other gadgets during Use of computer for homework leisure? leisure? gadgets during Use of computer for homework 100% 100% 90% 90% 24% 26% 24% 80% 34% 80% 31% 25% 25% 22% 70% 70% level 1 level 1 60% 60% 41% 38% 40% level 2 level 2 50% 32% 50% 37% level 3 38% 40% 38% level 3 40% level 4 40% level 4 30% level 5 30% level 5 24% 25% 25% 25% 20% 20% 25% 25% 26% 30% 10% 6% 7% 7% 10% 9% 0% 0% Never Rare Often All the time Never Rare Often All the time Source: Authors’ estimates based on the IC Literacy Test. Surprisingly, there was no statistically significant difference in ICT levels by students’ socioeconomic status.26 This is also true for many other variables that the team studies. It seems that the school level characteristics are not effecting student IC literacy. It might be the case since the school curriculum does not include the work on information and communication literacy. It is clear from these results that IC literacy of Russian students need to be developed further. This would require a more frequent use of technologies in teaching practices, the capacity for students to be able to use technology in schools, and deeper integration in general between the ICT and teaching and learning. The topic itself requires a more in-depth analytical work in the future. back to table of contents 54 26 The index for socioeconomic status included the following variables: educational attainment of mother and father, the number of books at home, and the possession of a PC/laptop, tablet, smartphone, smartwatch, or e-reader. Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice RESULTS AND DISCUSSION 5.6 Qualitative Analysis of Modern Teaching in Russian Schools As has been discussed earlier in the report, the to participate in the interviews, and 15 teachers OECD School User Survey gathers information from both urban and rural schools agreed to partic- about different teaching styles and physical envi- ipate. They represented the following Russian ronments that support and shape each pedagogical regions: the Republic of Tatarstan, the Republic practice (see Figure 20 in Section 5.2). According to of Chuvashia, the Republic of Marii El, the Ivanovo the answers collected during the TIMSS/SUS pilot region, and the Yaroslavl region. In addition, we study in 2019 and the full-scale TIMSS/SUS study interviewed the principal of the school in Moscow in 2019, Russian teachers use both traditional and City where team teaching is included in the daily modern pedagogical methods. Although the data curriculum with dedicated learning environments. indicated that the frequency with which teachers The average length of each interview was about 40 use modern methods such as group work and minutes. The number of teachers present at each team-teaching in their daily practices remains low, interview varied from one to four. it was not clear to us how exactly teachers define team teaching and in what ways they use group The teachers who were interviewed said that work in their daily classes. they only used team teaching in their pedagog- ical practice during extracurricular events, such as Therefore, we decided to conduct open-ended Geography Week or Polytechnic Week. Thus, this interviews with teachers to explore these questions. type of lesson is used only once or twice in each We identified a subset of SUS data, which included academic year. The usual practice was for two a list of schools where the teachers confirmed that subject teachers to jointly conduct a so-called “inte- they use both team teaching and group work in grated lesson” that combined two subjects such as their classes. Then we developed an interview guide chemistry and biology, English and chemistry, or consisting of three sets of thematic questions to be geography and biology (see Box 1). The teachers asked during the course of the interview covering tended to work together to teach several classes team teaching, group work, and the impact of and divided the students into smaller groups to COVID-19 on daily teaching. The structure of the complete the assignments or experiments set in interview was deliberately designed to be flexible these integrated lessons. In many cases, these inte- in terms of the sequence in which the questions grated lessons consisted of short thematic lectures were asked, so that the interviewer could try to and group assignments or were organized in the gain a more detailed or “in-depth” answer to any format of a game for which the teachers served as particular question ((see Annex 2 for the interview judges. If their school had children with disabilities, guide).). The regional TIMSS coordinators contacted the teachers also tried to involve those pupils in the selected schools and invited a group of 52 teachers extracurricular events. Box1: Example of an Integrated Lesson Conducted during an Extracurricular Event An integrated lesson called “World Ocean” was taught jointly by the teachers of biology and chemistry. The chemistry teacher talked about chemical elements, while the biology teacher explained about endemic species and how they adapt to their living conditions. Source: Interview with chemistry teacher, Republic of Marii El. back to table of contents 55 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice The teachers confirmed that the preparation for this type of lesson takes more time and coordination among colleagues than traditional teaching practices, and they found it difficult to use it in their daily practice. Some teachers complained that key parts of the curriculum are taught at different times of the school year in different subjects, making it difficult to combine the lessons for team teaching (see Box 2). Thus, to encourage more teachers to adopt team teaching, there is a need to coordinate the teaching schedules of the different subjects before the start of the academic school year (see Box 2). Box 2: Planning of Integrated Lessons Biology and chemistry are seemingly related in our educational program. However, the teaching of theory is so scattered across academic year that we have no way to connect it. If I, for example, teach the theory of proteins in the 10th grade, I do so at the beginning of the school year for me as a teacher of biology, while in the chemistry course, this topic is scheduled to be discussed at the end of the school year. But it's still easy for me, because my second specialty is chemistry. And I try to include some part of the meta-knowledge during the biology lesson. Source: Interview with biology teacher, Ivanovo region (rural school). During the focus-group interview, only one group of educators representing the same school confirmed that they used team teaching in their daily practice, but this was technically possible only because of the small number of students in that school. To decide whether to include the integrated lesson in the schedule, the teachers assemble together and make an assessment at the end of the academic year to identify any problems with children not understanding the educational material. If it turns out that the students expe- rience difficulties understanding one discipline because of cross-related knowledge in another subject that they have not yet mastered, then the school administrators plan for integrated lessons on the topic to be taught by a team of teachers the following academic year. Box 3: Team Teaching in Moscow City School Work format: The approximately 100 to 130 children, who are based in one building and represent a group of classes, get together in an open space. They are divided into several teams consisting of five or six students to engage in project activities. They are assisted by various teachers, usually from two different subject disciplines such as a computer science teacher and a music teacher. The children conduct role-playing games. For example, in the lobby, they design a "city of the future" and build a prototype. The older children (in grade 8) assemble an electric scooter from a special construction set. Before that, they created robotic auto watering stations. These activities are conducted within the framework of a computer science and technology collective lesson. The movement of objects is taught in joint physics and mathematics lessons. Other combinations might be English and tech- nology or literature and social science. We include these lessons in the daily curriculum plan. Number of teachers: Two teachers for 60 pupils during a collective lesson is better than one teacher for 30. If you can arrange a group of three teachers for 60 pupils, it would be an ideal situation. However, there is no main teacher. These are three equally important teachers. They are not allocated to specific children. A child can come to any teacher and ask a question. Planning of team-teaching lessons: We start planning a collective teaching lesson for the following academic year in April when the intensive work of preparing for the next year begins. We dedicate May and June to this. Source: Interview with school principal, Moscow City School. back to table of contents 56 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice Usually, the teachers use the halls or science labo- All of the interviewed teachers confirmed that they ratories to conduct these integrated lessons. The move furniture in order to adapt the learning envi- teachers claimed that they need bigger learning ronment to group work. However, in many cases, spaces with adequate capacity to accommodate they pointed out that standard school furniture is a substantial number of students to conduct team quite heavy, so such rearrangements take a lot of teaching. Additionally, the teachers confirmed time and effort (see Box 4). The teachers agreed that it is vital to have light and movable furniture, that furniture with wheels would be more useful. adequate equipment, and enough special material Another important piece of equipment is a digital for experiments (particularly in the case of physics whiteboard, which the teachers actively use in many and chemistry). classes. However, this type of equipment is not distributed equally among the different learning The interviewed teachers said that they also actively spaces in schools. The teachers also expressed a used group work during their daily practice within desire for these digital whiteboards to have wheels the framework of the standard curriculum. The to make them easier to move. main aim of using group work is to start learning a new topic or to repeat and master knowledge and One group of teachers were informed that their skills learned previously. Usually, the teachers divide school would be getting a new combined space a class into several small groups (each consisting of for science lessons and group work as part of two to five students) organized in accordance with the national project in education called “Point of the type of experiment or assignment being done. Growth.” The teachers were involved in the prepa- Sometimes, the groups are chosen by seating ration of a technical brief and the development of arrangements of the students in the classroom a design for the learning space and were able to (the teacher split the students into the groups by request that the space would have movable and rows) or by academic achievement (mixing strong flexible furniture, a request which has been granted. and weak students in one group). The teachers said that the children should feel comfortable in a working group. Box 4: The Use of Furniture during Group Work It is difficult [to move furniture] during the lesson, as desks are heavy. Changing furniture is difficult. But when an open event is scheduled, we rearrange the class into groups in advance. If we have prepared a lesson only in the group format, then we shift and change the furniture. When we teach a mini group, we don't move furniture. The first student turns to another student at the desk behind him and so on. They already know how to do it. Source: Interview with teacher, Yaroslavl region, rural school. During the period of quarantine due to COVID-19, all of the teachers confirmed that they worked in a distance mode using various digital tools and platforms. They confirmed that it was difficult to experiment with pedagogical methods, and they spent all of their energy simply on adapting to a new way of remote working. Some of their students had no access to computers or a proper internet connection, while some parents were against distance learning, especially the need for their children to use computers every day. Therefore, the teachers had to adjust the learning process in each case and provide a viable solution to keep children studying (see Box 5). back to table of contents 57 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice After the end of lockdown and when the schools were finally open, the Ministry of Education introduced new guidance for onsite operations. In most cases, teachers confirmed that their students are assigned to a classroom and must remain in that one space throughout the school day. The teacher must move between different classrooms and carry around all necessary learning equipment (see Box 5). It has become a challenge for many teachers of science (especially for chemistry and physics teachers) because some of their equipment cannot easily be carried to other places because of their size and for safety reasons. They expressed strong worries that, if students are unable to carry out practical exercises, this will negatively affect their learning outcomes and skills in the long term. In many cases, the teachers are trying to replace real experiments with YouTube videos and other digital materials. One teacher mentioned using a digital resource database called School-collection.edu.ru, which was developed under the E-Learning support project27 financed by the World Bank in Russia in 2004 to 2008. Box 5: Teaching Practices during COVID-19 “During distance learning, there were no joint lessons with other teachers. There were classes in which children were not able to access the lesson over Zoom because they lacked the necessary technology in their homes. There were some parents who were against their children learning over the Internet. For these students, the teachers wrote down the lessons in notebooks, and parents fetched the notebooks from the school and brought them back to school for their children’s work to be checked. Most students accessed Zoom through their smartphones. A significantly smaller proportion of children used a computer.” Source: Interview with a teacher of geography, Republic of Marii El, Yoshkar-Ola city “I follow the students. Our classes are all separated. The school is big. We have three floors. There is a recreation area on the right side of the building. Children come out of the building through only one entrance from the stairs. And we have everything planned out. Their arrival at school is planned. Senior classes come at half past seven. And then a bus brings children from the countryside from nearby villages. Then there are the middle classes, 6th and -7th grades. Before being admitted to school, everyone needs to have their temperature taken and to sanitize their hands. And all go out directly to the left. Other streams of people, who need to enter, go to the right. Each class is regulated. And the canteen also works in the same way.” Source: Interview with a teacher of physics, Yaroslavl region, rural school Overall, the results of the interviews revealed that: • Team teaching is used in some schools but usually only once or twice a year and mostly during extra- curricular activity weeks. • Usually, the team consists of teachers of two different subjects such as science and math or science and a foreign language (English or French). • The introduction of team teaching into a school’s daily practice requires advance planning and good cooperation between teachers and the school administrators. • Group work is often used in daily teaching but in order to easily reorganize the learning space to accommodate this kind of work, light and movable furniture is needed. • The current COVID-19 prevention regime limits the ability of teachers to conduct practical exercises and use modern teaching methods, thus potentially generating learning losses even after schools have reopened and are back to face-to-face learning. back to table of contents 58 27 https://projects.worldbank.org/en/projects-operations/project-detail/P075387 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice 6 6. Recommendations for Policymakers High-quality teaching can be delivered using both menting these modern practices in schools nation- traditional and modern styles. The effectiveness of wide would develop students’ 21st-century skills each will depend on the specific school context. Our and thus improve their learning outcomes. data suggest that high-quality teaching, whether traditional or modern, results in high TIMSS scores. Modern teaching practices that support the devel- Teachers are the main contributors to the quality of opment of students’ 21st-century skills can be learning and in the academic performance of their realized at each level of education, and Russia can students. With that in mind, our data also confirms learn from many examples of international best that student’s positive attitudes improve their practices such as: learning outcomes. Therefore, education policies related to teachers should encourage the develop- • A child-centered approach and play-based ment not only of their technical knowledge of the learning in early childhood education: A number subject that they teach but also ways to encourage of Southeast and East Asian education systems students and foster a positive attitude towards are using child-centered pedagogic practices to learning. foster holistic learning. They are expanding the definition of learning outcomes to include social, The use of modern teaching practices can be emotional, physical, and higher-order thinking supported by expanding the availability and use skills and are recognizing that holistic learning of technology in learning and by enhancing the requires new format of teaching to fully accu- quality of furniture in classrooms. The data show mulate child potential.28 Another example is the that these two factors – technology and comfort- Montessori model, characterized by multi-age able furniture – play a significant role in enabling classrooms, student-chosen learning activities, modern teaching practices such as group work and and minimal instruction. It has been shown to team teaching and enhance both students’ perfor- be more effective than conventional education mance and their well-being. The interviews that we in improving children’s executive functions. With conducted with teachers confirmed this finding successful local adaptations, Montessori and and showed that most teachers who practice other child-centered approaches – including modern teaching would like more flexibility in their Steiner, Reggio Emilia, and Tools of the Mind – school environment, including mobile laboratories, can be found in diverse settings from Haiti to transformable spaces, and movable furniture. Kenya. The Russian education system has strong theoret- • Social and emotional learning (SEL) programs ical and empirical foundations to expand modern in general education: SEL teaches students teaching. For example, team teaching is similar to 21st-century skills, while improving their the well-defined collective method of education academic outcomes. The most successful SEL propounded by Vitaly Dyachenko, Manuk programs are those that follow a whole school Mkrtchyan, and other Russian and international approach by: (i) integrating SEL into the regular scholars who have promoted modern teaching and curriculum; (ii) introducing dedicated SEL curric- learning methods. This report finds that imple- ular activities as necessary; and (iii) ensuring a back to table of contents 59 28 https://vuir.vu.edu.au/33645/1/Lessons_that_matter_what_should_we_learn_from_Asias_school_systems.pdf Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice safe and supportive school climate. Teachers (NYP) established very close and active collab- can deliver their regular curricular activities oration with partners in industry. The NYP’s while integrating SEL strategies such as inter- Teaching Factory concept integrates academic active learning, the establishment of goals studies with real-life work experience on indus- and rules, and use of greetings and closings try-based projects. Industry-based projects that have been shown to improve students’ enable students to develop hands-on, real- academic skills as well as their socio-emotional world problem-solving skills and to be ready to skills. For example, the US Responsive Class- work upon graduation (https://www.nyp.edu. rooms program takes a student-centered social sg/). In the Netherlands, the Dutch Vocational and emotional learning approach to teaching Colleges provide entrepreneurial courses with and discipline. The program includes a set of the objective of improving students’ non-cog- practices designed to create safe, joyful, and nitive skills such as teamwork and self-confi- engaging classrooms and school communities dence. for both students and teachers. • Making technology available in schools on an • Twenty-first century skills modules and prac- equitable basis to improve student learning and tice-based teaching in tertiary education: enhance teachers’ professional development. TVET and higher education programs across This would require equipping teachers with the the world use practice-based teaching. In skills needed to use technologies in teaching as Japan, KOSEN, a renowned TVET and five-year well as providing them with ongoing support. engineering education program KOSEN for The types of technology used in teaching students from 15 years old (with an additional include computers, the Internet, projectors, and two-year further study option), includes scien- any other learning tools that can help students tific experiments, workshop-based training, and with a diversity of learning needs and prefer- training in practical manufacturing skills. KOSEN ences. Portable devices as laptops and tablets makes extensive use of project-based learning are considered as the most suitable tools to to nurture critical thinking and problem-solving support these learning needs. In addition, skills where multi-disciplinary thinking and technological tools provide teachers with an entrepreneurship are strongly encouraged. enhanced array of strategies for their pedagogy. Internships and cooperative education provided Our study revealed that students scored higher in partnership with local industries are also part in all of TIMSS subject domains in those schools of KOSEN’s teaching and learning. There are where teachers frequently used different kinds a number of local and global competitions for of technologies in their lessons. These higher KOSEN students to demonstrate their engi- scores for different subjects and different types neering skills. (https://www.kosen-k.go.jp/ of technology ranged from the equivalent of english/). In China, Lenovo, an IT and computer one-third to two-thirds of a year of schooling. manufacturer, is working with tertiary institutes Even after controlling for students’ socioeco- to train vocational students in high-tech areas nomic characteristics, the difference and statis- such as cloud computing using practice-based tical significance in these scores remained for curricula, practitioner-led instruction, and all subjects, especially for schools that use the professional certification. In Hong Kong SAR, Internet (both wireless and wired), desktop China, an additional year of general education computers, and projectors/large TVs with audio was added to undergraduate programs in 2012 equipment. This is especially important given that is focused on developing students’ prob- the hybrid learning formats that have been lem-solving, critical thinking, communication, adopted as a result of the COVID-19 pandemic. leadership, and lifelong learning skills. This extra year seems to be producing graduates with • Helping teachers to adopt innovative teaching the attributes and abilities prized by the labor methods and exchange knowledge and expe- market. In Singapore, Nanyang Polytechnic riences. Only a small share of teachers use back to table of contents 60 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice modern teaching practices, but when asked if groups. The government and NGOs should they would be prepared to use these practices, unite to develop training for teachers in bullying many teachers said that they would. However, prevention, psychological support services for most current learning spaces are not conducive students in and outside the school, and infor- to adopting these innovative methods. What mation campaigns and learning material for is needed is more flexible spaces and movable parents to help them to support their children furniture that can be rearranged to enable more if they suffer from bullying. Additionally, schools group work and team teaching. It is important should explore how they can redesign existing for education decision-makers to make it learning environments to prevent bullying possible for team teaching to be used in daily behavior (for example, by creating open and practice instead of only during extracurric- transparent spaces with physically comfortable ular time (which was identified as the common furniture). practice in Russian regional schools in our qual- itative interviews with teachers). This would • Adjusting schooling in response to COVID-19. require changing the traditional way of planning The pandemic required schools to adopt new the academic schedule, providing teachers with and varied ways to ensure that education more time to design team teaching lessons, and could continue safely. Students were required supplying appropriate learning materials. Given to interact with their teachers remotely or in the diversity of Russia’s regions, it would be classrooms adapted to minimize interactions useful for policymakers to set up a knowledge and ensure safe distances between students. exchange among regions and between schools Returning to full-time face-to-face schooling in urban and rural areas to promote best is involving the continuation of many safety practices across the country. measures, which means that learning losses are still happening despite the return of students • Preventing bullying and developing supporting back to schools. If education policymakers were measures to maintain a positive school climate: to invest in new school equipment and ICT and Our findings showed that bullying can negatively in modern ways of teaching, it might be possible affect students' learning outcomes, especially to compensate for these learning losses. those of students from lower socioeconomic Implementing a program to improve teaching and learning based on these recommendations could yield significant improvements in learning outcomes, potentially equivalent to a half to a full year of learning. This would go a long way towards compensating for the learning losses resulting from COVID-19 and bring Russia back on track in terms of improving its learning outcomes. The post-pandemic period may be a chance to rethink the Russian education sector and to introduce innovations in the teaching of traditional subjects such as reading, mathematics, and science as well of 21st-century skills. This will require policy- makers to focus heavily on teachers and finding ways to support them in the process of changing their pedagogical practices. back to table of contents 61 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice 7 7. Conclusion As this study has shown, Russia’s education system tests. Teacher training, group work, team teaching, has done well in terms of creating equitable access related spaces that enable such activities, and to different education levels and achieving good fostering the development of 21st-century skills learning outcomes but has the potential to do in students may also make the education system better. One way to improve the academic perfor- more equitable and efficient. Using ICT in teaching mance of Russia’s students would be for schools to and learning can also add to these performance focus on developing students’ socio-emotional and gains, and the increased use of technology during higher-order cognitive skills in addition to teaching the COVID-19 pandemic has created an opportu- academic knowledge and skills. Global evidence nity to consolidate and extend this trend nation- suggests that the development of 21st-cen- wide, thus unlocking the potential of the Russian tury skills such as complex problem-solving and education system. teamwork is a sound and necessary investment to equip students with the ability to apply their Russia has already allocated a significant part academic skills effectively in the workplace and in of education public funding within the national life. While there is already demand on the labor projects up to 2030. The focus of those projects market in Russia for such skills, education policies includes strengthening education infrastructure, are not yet sufficiently focused on their develop- fostering the professional development of teachers, ment. digitizing education, and helping all students to excel. Adding the development of 21st-century skills In this study, we analyzed several datasets and agenda to the goals of the national projects would conducted a qualitative review of innovative encourage the use of modern teaching practices teaching practices in Russian schools in order to and improve learning environments, at the same explore the relationship among teaching practices, time creating better and more welcoming space in learning environments, the school environment, schools. In turn, this would yield better academic and students’ learning outcomes. outcomes, thus helping to overcome the learning losses associated with COVID-19 and providing This study expands on the earlier findings of TIMSS Russia with a skilled and productive workforce for pilot study that suggested a connection between the future. learning environments and learning outcomes as measured by TIMSS 2019 and also the importance of modern teaching approaches in developing 21st-century skills. How students perceive their schools, how secure they feel, and the frequency of bullying are all directly connected with learning outcomes, as the more positive students’ experi- ences of school, the more likely they will perform well. The findings of our study show that solving some problems (such as severe bullying) could help to raise students’ scores on learning achievement back to table of contents 62 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice References Angrist, Joshua D., and Jorn-Steffen Pischke. 2009. ¨ Mostly Harmless Econometrics: An Empiricist’s Companion. Princeton, NJ: Princeton University Press. Assessment and Teaching of 21st Century Skills. (n.d.). Website. http:// www.atc21s.org Barrett, Peter; Treves, Alberto; Shmis, Tigran; Ambasz, Diego; Ustinova, Maria. 2019. The Impact of School Infrastructure on Learning : A Synthesis of the Evidence. International Development in Focus;. Washington, DC: World Bank. © World Bank. https://openknowl- edge.worldbank.org/handle/10986/30920 License: CC BY 3.0 IGO Blackmore, J. et al. (2011), Innovative Learning Environments Research Study, Department of Education and Early Childhood Develop- ment, Melbourne. Care, Esther (2018). Education System Alignment for 21st Century Skills: Focus on Assessment. Center for Universal Education at The Brookings Institution. Chen, F. F. (2007). “Sensitivity of goodness of fit indexes to lack of meas- urement invariance.” Structural equation modeling, 14(3), 464-504. Claned Group ja ELE Finland. (2017). Road to 21st Century Competence. Evaluation Framework for Transversal Competence in the Finnish Curriculum. Retrieved from http://www.ele.fi/assets/evaluation_ framework_microsoft_final.pdf Council of Ministers of Education, Canada. (n.d.). Global Competencies. Retrieved from https://www.cmec.ca/682/Global_Competencies. html Council of Ministers of Education, Canada. (n.d.). CMEC Pan-Canadian Global Competencies Descriptions. Retrieved from http://www. ontariodirectors.ca/CODE-TLF/docs/tel/PanCanadian_Global_ Competencies_Backgrounder_EN.PDF Department of Education and Skills Ireland. (n.d.). Junior Cycle for Teachers. Whole School CPD Key Skills. Retrieved from https:// www.jct.ie/perch/resources/wholeschool/wscpdday2keyskills- booklet.pdf back to table of contents 64 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice Department of Education and Skills Ireland. (n.d.). Primary. Overview: Assessment Guidelines for Schools. Retrieved from https://www. ncca.ie/media/1351/assessment-guidelines.pdf Department of Education and Skills Ireland. (2016). Ireland’s National Skills Strategy 2025. Retrieved from https://www.education.ie/en/ Publications/Policy-Reports/pub_national_skills_strategy_2025. pdf Dyachenko V.K. 2004. Collective way of teaching. Didactics in dialogues. - M .: Public education.-- 352 p. European Commission. Council Recommendation on Key Competences for Lifelong Learning. Retrieved from https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=ur- iserv%3AOJ.C_.2018.189.01.0001.01.ENG&toc=O- J%3AC%3A2018%3A189%3ATOC Griffin P., Care E. (2015) The ATC21S Method. In: Griffin P., Care E. (eds) Assessment and Teaching of 21st Century Skills. Educational Assess- ment in an Information Age. Springer, Dordrecht International Society for Technology in Education (ISTE). (2019). ISTE Standards for Students. Retrieved from https://www.iste.org/ standards/for-students International Society for Technology in Education (ISTE). (2007). ISTE Standards Students. Retrieved from https://id.iste.org/docs/ pdfs/20-14_ISTE_Standards-S_PDF.pdf Hair, Joseph F. et al (2009). Multivariate Data Analysis: A Global Perspec- tive. 7th ed. Upper Saddle River: Prentice Hall. Lebedintsev V.B., Gorlenko N.M., Gornostaev A.O., Comma O.V., Emelin O.I., Katyshuk S.M., Klepets G.V., Litvinskaya I.G. 2012. The concept of future school. // Scientific and pedagogical school of M.A. Mkrtchyan. - Krasnoyarsk, Russia .-- 64 p. Ministry of Education Singapore. (2018). 21st Century Competencies. https://www.moe.gov.sg/education/education-system/21st-centu- ry-competencies Mealha, Ó. et al. (eds.) (2017). “Citizen, Territory and Technologies: Smart Learning Contexts and Practices, Smart Innovation,” Systems and Technologies 80, DOI 10.1007/978-3-319-61322-2_5 Ministry of Education of the Russian Federation. (2019). National Project “Education”. Retrieved from https://edu.gov.ru/national-project/ back to table of contents 65 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice Ministry of Education of the Russian Federation. (n.a.). State Federal Education Standards. Retrieved from https://fgos.ru/ Meeting Minutes, Project Committee on the Main Direction of Strategic Development of the Russian Federation: • Project Modern School («Современная школа») • Project Success of Every Child («Успех каждого ребенка») • Project Support for Families with Children («Поддержка семей, имеющих детей») • Project Digital Education Environment («Цифровая образовательная среда») • Project Teacher of the Future («Учитель будущего») • Project Young Professionals («Молодые профессионалы (Повышение конкурентоспособности профессионального образования») • Project New Opportunities for Everyone («Новые возможности для каждого») • Project Social Activity («Социальная активность») • Project Education Export (Экспорт образования) • Project Social Lift («Социальные лифты») • National Education Project OECD (2017), PISA 2015 Results (Volume V): Collaborative Problem Solving, PISA, OECD Publishing, Paris, https://doi. org/10.1787/9789264285521-en. OECD (Organization for Economic Cooperation). (2018). The Future of Education and Skills. Education 2030. Retrieved from https://www. oecd.org/education/2030/E2030%20Position%20Paper%20 (05.04.2018).pdf OECD. (2019). OECD Learning Compass 2030 Concept Notes. Retrieved from https://www.oecd.org/education/2030-project/teaching- and-learning/learning/all-concept-notes Ontario Ministry of Education. (2016). 21st Century Competencies: Foundation Document for Discussion. Retrieved from http://www. edugains.ca/resources21CL/About21stCentury/21CL_21stCentury- Competencies.pdf Pearl, J. (2012). The causal foundations of structural equation modeling (Vol. 370). California Univ Los Angeles Dept of Computer Science. Pearl, Judea (2009). Causality: Models, Reasoning, and Inference. 2nd ed. Cambridge University Press. P21 (Partnership for 21st Century Learning). (2019). Framework for 21st century learning. Retrieved http://www.battelleforkids.org/ networks/p21/frameworks-resources back to table of contents 66 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice Reimers, F. M., Chung, C. K. (2019). Teaching and learning for the twen- ty-first century: Educational goals, policies, and curricula from six nations. Harvard Education Press. Shmis, Tigran and Suhas D. Parandekar. (2018). “Education Equity in the Russian Federation: Summary Report” (English). Washington, DC: World Bank Group. http://documents.worldbank.org/curated/ en/139291530189329351/Education-Equity-in-the-Russian-Fed- eration-Summary-Report. Slavin, R.E. (1995). Cooperative learning: Theory, research, and practice (2nd ed.). Boston: Allyn and Bacon. Slavin, R.E. (2013). “Classroom applications of cooperative learning” S. Graham (ed.), APA handbook of educational psychology. Wash- ington, DC: American Psychological Association. Vasiliev, Kirill; Lukiyanova, Anna; Chugunov, Dmitry; Maltseva, Inna; Shulga, Ivan; Rutkowski, Jan J.; Cahu, Paul Marie Michel; Travkin, Pavel; Roshchin, Sergey; Nellemann, Soren (2013). "Developing skills for innovative growth in the Russian Federation" (English). Wash- ington, D.C.: World Bank Group. http://documents.worldbank.org/ curated/en/460821468107067600/Developing-skills-for-innova- tive-growth-in-the-Russian-Federation University of Helsinki. (n.a.). Transversal competencies in Finnish basic education. Retrieved from https://tuhat.helsinki.fi/ws/portalfiles/ portal/124343473/L_HETETTY_Transversal_skills_in_Finnish_ curriculum.pdf World Economic Forum. (2015). New Vision for Education Unlocking the Potential of Technology. Retrieved from http://www3.weforum.org/ docs/WEFUSA_NewVisionforEducation_Report2015.pdf Webb, N. M. (2008). “Learning in small groups” in T. L. Good (ed.), 21st century education: A reference handbook (pp. 203-211). Los Angeles: Sage. “World Bank. 2019. World Development Report 2019: The Changing Nature of Work. World Development Report; Washington, DC: World Bank. © World Bank. https://openknowledge.worldbank.org/ handle/10986/30435 License: CC BY 3.0 IGO.” Zhao, Y. (2015). Lessons that matter: What should we learn from Asia?, Mitchell Institute discussion and policy paper No. 04/2015. Mitchell Institute for Health and Education Policy, Melbourne. Available from: www.mitchellinstitute.org.au back to table of contents 67 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice About Authors Tigran Shmis holds a specialist degree (2001) as a school teacher of computer sciences and economics. He also completed postgraduate study in educational ICTs and holds a PhD (candidate of sciences) degree (2004) from the Russian Academy of Education. Later he completed the MEd (2007) program at the Moscow branch of the University of Manchester on management in education and educational policy. Tigran has worked on educational projects in the Russian Federation, Belarus, Kazakh- stan, Kyrgyz Republic, Romania, Serbia, and Peru. Among those projects were the Yakutia Early Childhood Development (ECD) Project, Russian Education Aid for Development (READ), Khanty-Mansyisk ECD infra- structure development technical assistance, Kyrgyz ECD Project, Belarus Education Modernization Project, Serbia ECD Project, and technical Assistance on Safer Schools development project in Peru. He delivered several cooperation programs for the OECD Center for Effective Learning Environments (CELE) and Early Childhood Education and Care (ECEC) networks and UNESCO. Areas of research and professional interests are ECD, innovative learning environments, and international assessment work. Tigran leads work on innovative learning environments, ECD quality initiatives, and capacity building of Russia in international development aid in education. Maria Ustinova works as a consultant at the World Bank office in Moscow, where she supports various technical assistance and lending projects in the field of education, social protection and inclusion. She also serves as an associate researcher at the Urban Health Games Research Group, which is a part of the Architecture Department at the Technical University of Darmstadt, Germany. She contributes to research projects investigating how urban planning and design influence human health and well-being, particularly focusing on school learning environments. Additionally, she teaches a course on “Learning Environments Design Foundations” at Moscow City University. Maria holds a double master’s degree in international cooper- ation and urban development from Darmstadt University of Technology, Germany, and Tor Vergata University of Rome, Italy. Prior to joining the Bank, she worked as a consultant for the United Nations Economic Commission for Europe as well as for various projects of the Directo- rate-General for Education and Culture at the European Commission and the Education, Audiovisual and Culture Executive Agency of the European Union. Her areas of research and professional interest include education facilities design, early childhood development, health promoting design, universal design and participatory practices in education facilities design. back to table of contents 68 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice Dmitry Chugunov is an education economist with over seven years of international expe- rience working at the national and international level on education and capacity building projects. He has global work experience with devel- opment aid agencies in Africa, Europe, and Central Asia and special- ized in education policy analysis, impact evaluation (pre- and post-in- tervention), cost analysis and projection, mobile data collection, student assessment, and workforce and skills development. He has authored or co-authored over 30 analytical reports, articles, and knowledge products in the above areas and is proficient in large dataset management and economic modeling. Dmitry is dedicated to creating evidence-based policy recommendations through high-quality analysis. Dmitry holds a PhD in economics with a specialization in public economics and a master’s degree in economics with a specialization in public finance from the National Research University Higher School of Economics in Moscow. Suhas Parandekar is a Senior Economist in the World Bank’s Global Education Practice, where he leads and participates in task teams in the areas of education, science, technology and innovation. He has been working at the World Bank for nearly twenty years on the forefront of development policy formula- tion and implementation setting up cross-functional teams that design projects and perform policy analysis together with partners from govern- ment and other agencies. He has worked in projects in 24 countries, with the bulk of his experience from the countries of El Salvador, Nicaragua, Argentina, Brazil, Laos and Vietnam. He was also working on projects in the Russian Federation, Ukraine and Georgia. Currently he has been working on issues related to computational literacy and the teaching of coding to children; extra-curricular education; financial education and financial literacy and higher education policy. Ekaterina Melianova worked as a consultant at the World Bank office in Moscow, where she supported quantitative analysis in various education projects. Her research and professional interests revolve around the application of advanced statistical methodologies to policy analysis, particularly in the field of education and public health. Ekaterina has contributed to projects dedicated to the examination of extra-curricular education and returns to schooling in the Russian Federation. She aspires to exploit novel and rigorous data-analytic techniques to improve policy decision-making processes. Ekaterina is a quantitative social scientist by training; she holds master’s degrees in comparative social research and applied statis- tics from the National Research University Higher School of Economics in Moscow. Ekaterina is currently a PhD researcher in Advanced Quantita- tive Methods programme at the University of Bristol, UK. Lucy Kruske worked as a consultant at the World Bank headquarters in Washington DC, specializing in skills and workforce development policy projects. Her research and professional interest focus on skills policies and systems, employer engagement in skills development, and the future of work. Prior to joining the World Bank, she conducted research on technical vocational education and training for the Harvard Ministerial Leadership Program and led human capital development in the private sector. Lucy holds a master’s degree in international education policy from Harvard Univer- sity and is currently a McCloy Fellow at the Harvard Kennedy School. back to table of contents 69 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice Annex 1: Tables and Figures Annex Table 1.1: Descriptive Statistics for Main Variables, weighted Variable Mean SD Min Max Math teacher quality BSBM17A 3.07 0.78 1 4 BSBM17B 3.04 0.79 1 4 BSBM17C 3.27 0.77 1 4 BSBM17D 3.45 0.71 1 4 BSBM17E 3.48 0.68 1 4 BSBM17F 3.22 0.76 1 4 BSBM17G 3.45 0.72 1 4 Teacher’s orderliness in math BSBM18A 2.66 0.94 1 4 BSBM18B 2.82 0.95 1 4 BSBM18C 3.12 0.93 1 4 BSBM18D 3.04 0.96 1 4 BSBM18E 3.17 0.92 1 4 BSBM18F 2.76 1.02 1 4 Students' beliefs BSBM16A 2.93 0.83 1 4 BSBM16B 2.90 0.95 1 4 BSBM16C 3.01 0.85 1 4 BSBM16D 3.02 0.80 1 4 BSBM16E 2.91 0.89 1 4 BSBM16F 2.62 0.88 1 4 BSBM16G 2.62 0.90 1 4 BSBM16H 2.30 0.84 1 4 BSBM16I 2.43 1.01 1 4 BSBM19A 2.83 0.84 1 4 BSBM19B 2.63 0.92 1 4 BSBM19C 2.60 0.99 1 4 BSBM19D 2.50 0.86 1 4 BSBM19E 2.73 0.92 1 4 BSBM19F 2.34 0.81 1 4 BSBM19G 2.46 0.88 1 4 BSBM19H 2.68 0.98 1 4 BSBM19I 2.78 0.95 1 4 BSBM20A 3.17 0.84 1 4 BSBM20B 3.24 0.75 1 4 BSBM20C 3.04 0.93 1 4 BSBM20D 2.97 0.94 1 4 BSBM20E 2.26 0.94 1 4 BSBM20F 2.94 0.89 1 4 BSBM20G 3.06 0.84 1 4 BSBM20H 3.23 0.78 1 4 BSBM20I 3.11 0.82 1 4 back to table of contents 70 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice Variable Mean SD Min Max Math BSMMAT01 546.54 80.83 227 841 BSMMAT02 546.79 80.33 255 819 BSMMAT03 547.61 80.59 242 805 BSMMAT04 546.57 82.36 305 827 BSMMAT05 546.92 81.07 274 810 Science BSSSCI01 546.04 74.19 230 793 BSSSCI02 545.93 74.40 273 823 BSSSCI03 546.20 74.80 223 793 BSSSCI04 545.19 75.63 267 774 BSSSCI05 544.58 75.75 252 772 Math applying BSMAPP01 545.98 81.49 250 809 BSMAPP02 547.02 81.68 253 805 BSMAPP03 545.94 81.10 278 810 BSMAPP04 546.33 81.28 268 795 BSMAPP05 547.01 81.84 261 805 Math reasoning BSMREA01 537.35 84.12 245 797 BSMREA02 538.52 84.27 257 807 BSMREA03 540.48 84.98 270 827 BSMREA04 539.12 84.33 265 804 BSMREA05 541.69 82.98 257 809 Science applying BSSAPP01 544.75 84.43 253 808 BSSAPP02 546.89 84.12 175 813 BSSAPP03 546.23 83.27 209 829 BSSAPP04 544.64 82.45 177 819 BSSAPP05 545.26 85.87 228 838 Science reasoning BSSREA01 546.31 74.43 288 787 BSSREA02 543.31 72.74 234 763 BSSREA03 549.23 71.95 265 797 BSSREA04 544.69 74.29 243 781 BSSREA05 546.94 72.77 281 784 Safety n24_0 3.00 1.07 1 4 n24_1 3.36 0.86 1 4 n24_2 3.34 0.87 1 4 n24_3 2.88 1.18 1 4 n24_4 3.29 0.92 1 4 Technology n_tech 6.82 2.44 0 10 Temperature n11_2 2.98 0.87 1 4 n12_2 2.85 0.94 1 4 Audibility n15_0 3.56 0.72 1 4 n15_1 3.38 0.84 1 4 n15_2 3.26 0.92 1 4 Visibility n16_0 3.42 0.78 1 4 n16_1 3.24 0.93 1 4 n16_2 3.25 0.91 1 4 Comfortable furniture n17_0 3.11 0.92 1 4 n17_1 1.92 1.04 1 4 n17_2 1.77 1.03 1 4 Traditional teaching style 0.69 0.46 0 1 back to table of contents 71 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice Annex Figure 1.1: Path Diagram for Math Scores as an Outcome Modern Teaching Style Traditional Teaching Style Math Important Math Strong Math Important Math Strong Understandable Teacher Quality Math Enjoy Understandable Teacher Quality Math Enjoy Orderliness Orderliness 0.98 0.91 Audibility Audibility Visibility Conditions -0.09 Student Attitudes Visibility Conditions 0.07 Student Attitudes Temperature 0.06 Temperature 0.03 0.23 0.21 0.1 0.03 Safety Math Safety Math 0.1 -0.04 Technology Technology School Space School Space Comfort furniture Comfort furniture Annex Figure 1.2: Path Diagram with Wellness as an Outcome Modern Teaching Style Traditional Teaching Style Math Important Math Strong Math Important Math Strong Understandable Teacher Quality Math Enjoy Understandable Teacher Quality Math Enjoy Orderliness Orderliness 1.08 0.98 Audibility Audibility Visibility Conditions -0.09 Student Attitudes Visibility Conditions 0.09 Student Attitudes Temperature 0.06 Temperature 0.03 0.3 0.37 0.1 0.04 Safety Wellness Safety Wellness 0.12 -0.03 Technology Technology School Space School Space Comfort furniture Comfort furniture back to table of contents 72 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice Annex Figure 1.3: Path Diagram with Math Applying as an Outcome Modern Teaching Style Traditional Teaching Style Math Important Math Strong Math Important Math Strong Understandable Teacher Quality Math Enjoy Understandable Teacher Quality Math Enjoy Orderliness Orderliness 0.97 0.91 Audibility Audibility Visibility Conditions -0.09 Student Attitudes Visibility Conditions 0.07 Student Attitudes Temperature 0.06 Temperature 0.02 0.24 0.21 0.09 0.03 Safety Applying Safety Applying 0.1 -0.04 Technology Technology School Space School Space Comfort furniture Comfort furniture Annex Figure 1.4: Path Diagram with Math Reasoning as an Outcome Modern Teaching Style Traditional Teaching Style Math Important Math Strong Math Important Math Strong Understandable Teacher Quality Math Enjoy Understandable Teacher Quality Math Enjoy Orderliness Orderliness 0.98 0.92 Audibility Audibility Visibility Conditions -0.09 Student Attitudes Visibility Conditions 0.07 Student Attitudes Temperature 0.06 Temperature 0.03 0.23 0.21 0.1 0.03 Safety Reasoning Safety Reasoning 0.1 -0.04 Technology Technology School Space School Space Comfort furniture Comfort furniture back to table of contents 73 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice Annex 2: A guide for interviewing teachers Respondents: Grade 8 teachers from general education schools, who took part in the TIMSS/School User Survey and specified in their questionnaires that they were using collaborative learning and group work techniques in their teaching activities. Purpose of interview: To get an insight into how the teachers understand the essence of collabora- tive learning and group work techniques, understand when such techniques are employed in teaching practices, identify the barriers and opportunities for developing the educational environment of a school supporting the use of such methods. Unit 1. Introduction Objectives of the unit: To get an insight into the teacher’s professional experience Suggested questions: What is your length of employment? What subjects do you teach? Which other grades of students do you teach in addition to grade 8? What is the average number of students in the classroom who attend your lesson? What school classrooms do you use for teaching lessons? Unit 2. Collaborative learning activities Objectives of the unit: To get an insight into how the teachers understand the essence of collaborative learning techniques, and in what way they employ such techniques in their teaching practices. Suggested questions: 1. Could you please tell me if you have experience in co-teaching with another teacher(s)? 2. What is the number of your teacher colleagues with whom you have collaborated in co-teaching? 3. What is the number of students that you have worked with using collaborative learning activities and co-teaching? 4. With what subject teachers have you conducted collaborative learning lessons? 5. How did you plan such lessons? 6. What school facilities and classrooms have you been using to provide for collaborative learning activities? 7. From your perspective, what classrooms do you need to provide for collaborative learning activities? 8. From your perspective, what school facilities do you need to provide for collaborative learning activities? back to table of contents 74 Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice Unit 3. Group learning Objectives of the unit: To get an insight into how the teachers understand the essence of the group work technique, and in what way they employ such method in their teaching practices. Suggested questions: 1. Could you please give examples of how you or your colleagues separate students into small groups in your teaching practices? 2. For what purpose do you use this method in teaching your subject? 3. How many students have you worked with using group work? 4. What challenges have you noticed when your students were working in group? 5. What school facilities and classrooms have you been using to provide for group work? 6. What school facilities and classrooms have you been short of to provide for group work? 7. From your perspective, what classrooms are needed to provide for group work? 8. From your perspective, what school facilities are needed to provide for group work? Conclusion of the interview, acknowledgement of participation back to table of contents 75 Report No: AUS0002233 Russian Federation New Skills for New Century: Regional policy 6/25/2021 © 2021 The World Bank 1818 H Street NW, Washington DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org Some rights reserved