World Bank Education Global Practice: Report P164840 Education Equity in the Russian Federation Summary Report Education Worldwide Relationship between Index of Economic, Social and Cultural Status and PISA 2015 Science Performance PISA 2015 Science Score Effect Count ESCS Effect This report is written by staff and consultants of the World Bank. The findings, interpretations and conclusions ex- pressed herein are those of the authors and should not be attributed in any manner to the Board of Executive Directors the World Bank or the governments they represent, the Government of the Russian Federation, or any of the reviewers. Any mistakes found in the report are the sole responsibility of the authors and the World Bank does not guarantee the accuracy of the data included in this work. © The World Bank, The World Bank Group, 2018 1818 H Street NW, Washington, DC 20433 Телефон: 202-473-1000; Веб-сайт: www.worldbank.org Cover fotos: authors’ archives. Lay-out and prepress: alexpublishers.ru 3 Acronyms BART Bayesian Additive Regression Trees BSE Basic Stata Exam CLPS Collaborative problem-solving skills ECA Europe and Central Asia ECD Early childhood development ESCS PISA index on economic, social and cultural status EU European Union GDP Gross domestic product IBRD International Bank for Reconstruction and Development ICT Information and Communication Technology IEA International Association for the Evaluation of Educational Achievement ISSWB Index of School Social Well-Being IT Information Technology MCMC Markov Chain Monte Carlo Algorithms OECD Organization for Economic Co-operation and Development PIAAC Program for the International Assessment of Adult Competencies PIRLS Progress in International Reading Literacy Study PISA Program for International Student Assessment R&D Research and Development SCMATEDU Index of quality of the school's educational resources TIMSS Trends in International Mathematics and Science Study TrEC Trajectories in Education and Careers UNESCO United Nations Educational, Scientific and Cultural Organization USE Unified State Exam VET Vocational Education and Training World Bank Education Global Practice: Report P164840 4 Acknowledgments This report was prepared by a World Bank team led by Tigran Shmis (Senior Education Specialist, Education Global Practice) and Suhas Parandekar (Senior Economist, Education Global Practice) and includes contributions from Shizuka Kunimoto, Dmitry Chugunov, and Polina Zavalina. A team of consultants, including Sergei Kosaretsky, Marina Pinskaya, Alexandra Mikhailova, Roman Zvyagintsev and Andrey Zakharov from the National Research University Higher School of Economics (HSE) contributed to the qualitative and economic analysis of the report. The work was carried out under the guidance of Andras Horvai, Russia Country Director, Dorota Nowak, Country Program Coordinator, Christian Aedo, Practice Manager, and benefited from comments and discussions with peer reviewers Amer Hassan, Toby Linden, and Anna Olefir. Education Equity in the Russian Federation. Summary Report 5 Abstract This report outlines the findings from an analysis of inequity in Russian pre- university education. The report seeks to answer to the following questions: what is the magnitude of educational inequity in Russia and what are its sources; what practices might potentially tackle inequity in Russian schools; and what are the potential economic benefits to the Russian Federation of dealing with educational inequities. The report uses data from standardized international assessments to compare Russian performance with other countries. The study uses the latest quantitative methods including a machine learning model to understand the dynamics connecting social inequity and inequity in educational outcomes. Further, the report examines the prevention and remediation measures by looking at the resilient schools in Russia and identifies the factors of schools’ resiliency. This part includes assessment and the regional case study on the access, quality, and equity in ECD in The Sakha Republic (Yakutia) and in Tomsk Oblast and the analysis of the resilient schools and factors of success, including the management, pedagogy and school surroundings. Finally, a brief part of the report depicts the positive effects on economic growth from investments to bring up the low performing students to an average level of performance. The report summarizes the five analytical papers. The five papers include: 1. Good Performance and Getting Better: Comparative Education Equity in Russia; 2. Social Equity, Vocational School and Educational Performance; 3. Collaborative Problem-Solving Skills; 4. Resilient Schools; and 5. Short Note on the Economic Value of Improvement in Education Equity. The abstracts of these papers are presented in the Annex to this report, while all papers will be available as separate analytical pieces. World Bank Education Global Practice: Report P164840 6 1. Comparative educational equity performance of the Russian Federation 1. The Russian Federation does well on educational equity, a fact that is perhaps not as well-known as Russian performance on levels of international assessments. It is by now quite well known that the Russian Federation exhibits a good and improving performance in standardized international assessments such as Programme of International Student Assessment (PISA), Trends in International Mathematics and Science Study (TIMSS) and Progress in International Reading Literacy Study (PIRLS). Perhaps it is not yet as well known that the Russian Federation also does well on equity of educational performance. Russia is the highest ranked country on PIRLS, which measures primary school reading – a country does not get to be first ranked without all groups doing consistently well. However, Russian performance on PISA is close to the OECD average, which means that there are about thirty countries ranked above PISA and makes PISA an interesting realm to study education equity in the Russian Federation. 2. Russian students show the lowest ‘achievement gap’ between students from the highest and lowest socio-economic groups. When we compare achievement by students from the lowest socio-economic quintile with those the highest socioeconomic quintile, the OECD countries as a whole, and the European Union countries as an average, showed a difference of nearly 100 PISA points for 2015. This means that students who came from well-off families, with high levels of family wealth, parental education and high-status occupation, tended to score 100 PISA points higher, across the board – for Science, Mathematics, and Reading, as shown in Annex Figure 1. 100 PISA points is a big difference, equivalent to one standard deviation or, according to the OECD, about three years of instruction1. The gap is present in the Russian Federation as well, but is almost half of the European average, and ranks Russia ahead of all other European PISA countries in terms of the lowest size of the gap. Similarly, the share of poor performers from the poorest quintile in Russia was 29.1 percent (see Figure 1). This proportion in Russia is smaller compared to other countries. For instance, the share of poor performers from the poorest quintile in Brazil and Finland in PISA 2015 mathematics is 90% and 60.2 % respectively. 3. Russian education equity performance is likely on account of strong and sustained educational policy measures and not a historical artifact of higher income equity for former socialist countries. Performance on equity in the Russian Federation has been improving over time, and this trend in the right direction contrasts with some key European comparators. Annex Figure 1 shows that some of the countries with the highest achievement gap by socio-economic status include Bulgaria, the Czech and Slovak Republics, and Hungary. Annex Figure 2 indicates how Russian performance on equity tracks Singapore and China (Hong Kong) more than the European countries of Finland and the Slovak Republic that unfortunately show an increasing trend of education inequity over time. We carried out a micro-econometric analysis of the data on test scores and ESCS to examine the nature of the relationship more closely. This analysis, which is detailed further in the papers that make up this report, clearly shows the relationship between past 1 OECD (2016), PISA 2015 Results (Volume I): Excellence and Equity in Education, PISA, OECD Publishing, Paris. P. 65 Education Equity in the Russian Federation. Summary Report 7 Figure 1. The share of Poor Performers by ESCS in PISA 2015 Mathematics (%) Bottom ESCS All students Source: World Bank staff calculations based on PISA 2015 database, OECD and current educational reform measures and the ensuing superior educational equity performance. These policy measures are detailed in the next section of this summary report. 4. Russia has set itself the objective of being amongst the ten highest ranking countries by educational quality – to achieve this objective it will be vital to address inequity by bringing up the low performers. While Russia achieves the highest rank of performance on the PIRLS test that measures early grade reading and is already within the top ten of countries on the TIMSS test that measures curricular achievement in mathematics and science in Grade 8, Russia’s performance is about average or below average on PISA measurement of competencies for 15 year- olds. There are about 75 countries that take part in PISA and Russia would need to improve its performance by about one standard deviation to move from its current position to be in the top ten. Our analysis indicates two facts: a) it is feasible to achieve this task by bringing up the scores of low performers so that all students had a minimum score of 500; and b) achieving this would yield substantive gains for the Russian economy, by our estimates, a gain equivalent to one-third of Russian GDP over a period of about 20 years. 2. Policies and programs associated with better equity 5. Russia has consistently focused on learning and assessment of that learning through sample-based international standardized assessments as well as a national assessment. Participation in PISA, TIMSS, PIRLS, and PIAAC has made international benchmarking possible. The Unified State Examination (popularly known as EGE after the Cyrillic alphabet acronym for Единый Государственный World Bank Education Global Practice: Report P164840 8 Экзамен) became a mandatory examination for university entrance, with nearly a million students now taking the exam. Russian students perform very well in early grades (PIRLS) but Russia’s ranking falls off considerably by the time they reach high school, and few Russian universities rank amongst the top 500 universities in the world. In recent years, policy attention has turned to competencies and problem- solving skills, which remains a highly important area from an equity viewpoint. 6. The endowment effect of having a highly educated parent population is an important factor influencing educational equity in the Russian Federation. One of the big contributions to better equity in the Russian Federation is the fact that Russian parents have a much higher level of education compared to the average level of OECD. According to the official OECD figures, about 10% of the lowest socio-economic group students for OECD as a whole have parents who have a tertiary level education, the figure for the Russian Federation is 68%. More educated parents not only means better support at home, one of the findings of this research has been an even distribution of parental participation at school. Policies that harness parental support and involvement in school would enable Russia to capitalize further on this substantive endowment advantage for equity. 7. Consistent policy attention and investment have been made in the Russian Federation to provide access to Pre-school education. Pre-school enrollment in the Russian Federation is comparatively at a high level as recorded through the reported experience of PISA students and from general statistics. As with other policy issues, when levels of a variable are high, it is a corollary that coverage is even across groups – the same goes for pre-school coverage. The report discusses the findings of the positive relationship between Pre-school and educational equity. One important related finding from other studies will be useful to mention. Around the world, there is growing evidence that the quality of the Pre-school is very important in determining its impact on future benefits. While this may appear to be obvious or tautological, it does imply that policy should track not only access but also the quality of pre-school. 8. Policies directed towards providing adequate infrastructure are likely to be associated with improved equity. Regional difference in economic output and living standards are well known. However, these differences are muted when it comes to educational performance. A probable reason for smaller differences in educational performance is fewer differences in educational inputs. The quantity and quality of educational infrastructure and the quality and quantity of teachers across Russian regions are similar, even as economic output is not similar. Regression equations reported as findings tend to confirm such a hypothesis. Further research is required to help determine the kind of infrastructural investments that need to be deepened for better educational performance, especially to bring students up from the lower end of the performance distribution. 9. Family cultural possession, motivation calibration, and sense of belonging among students play a role in equity and performance. Families that have more orientation towards art and culture in their homes tend to positively influence the science scores of Russian students even if the families are not affluent. Same as in the EU, in Russia students with well-calibrated motivation from bottom ESCS show very close results to the students with poorly calibrated motivation from top ESCS (motivation calibration Education Equity in the Russian Federation. Summary Report 9 is a measure of a student’s ability to recognize motivation in others or the extent to which the student’s definition of motivation agrees with the standard definition). The sense of belonging in students also play a role in their performance. Schools and regional authorities may expand the school cultural resources, engage students in creative work and arts as well as in other extracurricular creative work. The motivation for achievement and pursuing high-quality outcomes may become an important dimension for the education policy of the Russian Federation. Students with mindsets towards achievement would be able to perform and show better results regardless of their socio-economic status. 3. Equity concerns regarding Collaborative Problem Solving Skills (CLPS) 10. The equity performance differential of Russia with European/OECD countries is weak for collaborative problem-solving skills, which raises a concern about the future. This is a case of relative performance differential - while Russia does much better than the OECD average on equity for Science, Mathematics, and Reading, the gap for CLPS is narrow. Furthermore, the reason that Russia performs better on equity ground for CLPS is a sort of `wrong’ reason – it is because Russian performance is not high even for students from higher socioeconomic levels, as measured by ESCS, the Economic, Social and Cultural Status index used throughout this report. ESCS is a combination of family wealth, parental education level, parental occupation status and cultural possessions at home such as works of art and musical instruments. Reducing inequity in CLPS needs to be an issue of policy concern with a view to Russian performance on 21st-century skills. 11. Regional differences in educational performance for traditional subjects are relatively low in the Russian Federation, but CLPS scores show significant differences. There are regional differences in performance – it is not the case of perfect equality or something like that, for instance the Northern Economic Region shows very good performance on Reading – it would be a top ten performer as a country by itself, but overall Reading performance for Russia is about the same as the OECD average. These are qualitative conclusions as PISA sample size in the Russian Federation is not adequate for regional representation, unlike other countries such as regions in Spain and states in Brazil. Performance of some regions on CLPS is quite low – uncharacteristically for the Russian Federation, CLPS performance in the North Caucasus would be similar to the performance of Thailand or Mexico. 12. Measurement of CLPS involves children interacting with a set of computerized artificially intelligent agents and CLPS assessment in a game-like manner has much to recommend itself. The report provides a quick reference to OECD literature regarding the CLPS assessment itself, which is quite useful for policymakers to get a quick understanding of how it is conducted. The report documents the close correlation between CLPS and performance in Science, Mathematics, and Reading. It would appear that CLPS assessment is less stressful for the students and less susceptible to strategic preparation and answering. If it is able to uncover performance that combines cognitive and non-cognitive skills, there is potential for World Bank Education Global Practice: Report P164840 10 all testing to be carried out similarly as CLPS. In any event, from a policy perspective on assessment, investment by the Russian Federation to deepen the OECD CLPS test development efforts should be considered by policymakers. 13. Analysis of the determinants of the CLPS performance to understand how CLPS equity may be improved points towards infrastructure as an important element. While inequities in infrastructure provision may be less severe in the Russian context as compared to the OECD, the data shows that the infrastructure index is related to the CLPS performance. PISA includes a variable that measures school principal reports of deficiencies in infrastructure such as laboratories, educational equipment, heating and lighting, and the school grounds. We find for the Russian Federation that this deficiency index reported by the low CLPS performing groups is quite high for the low performing group. This finding is confirmed by the reported regression analysis for CLPS. Infrastructure improvements are likely part of the policy solution to reduce inequity of CLPS performance. 14. ICT related education and the policies and resources for digital competency may not be functioning adequately. CLPS is a generic set of skills not necessarily linked to a particular subject or specific modalities like ICT. However, digital competency/computational science has a very strong relation to CLPS because of various reasons, including instant feedback in activities such as coding and the importance of collaboration in identifying problems and sharing solutions. The report finds a potentially disturbing negative correlation between the use of ICR resources at school and the CLPS performance. This relationship does not hold for home use or entertainment use. It is possible that this anomalous result stems from either poor curriculum or poor instruction. The ongoing curriculum review for general education in the Russian Federation should closely examine the issue of the ICT curriculum across subjects and the implementation in schools. 15. Teacher behavior and skills lead to a positive school climate in the Russian Federation that provides the potential for comparative advantage similar to the one due to the parental endowment. While not as big in magnitude as for the ICT issue mentioned above, there is an anomalous result with regard to teacher behavior. The Russian Federation benefits from a teacher population that is proportionally highly educated as can be seen from the data on parents. The data from TIMSS indicates that nearly all Russian teachers of eighth-grade students in the 2011 implementation of TIMSS had a university degree and that while half of them had subject level specialization only, the other half had both subject level specialization and an additional qualification on pedagogical training. The data suggest that teacher behavior problems (how teachers are getting along with students and colleagues) are statistically significantly associated with improved learning. It is possible that this result may not withstand deeper scrutiny, but it may be correct in the context of possible over-regimentation of curriculum and instruction, which would show up for CLPS but likely not for traditional subjects. It would appear to be sensible from a policy viewpoint to incorporate teachers closely in the ongoing curriculum reform and more broadly with autonomy and agency in the instructional process. Education Equity in the Russian Federation. Summary Report 11 4. Enhancing Image of Vocational Education for Improved Equity 16. Using a cutting-edge machine learning analytical technique, this report reaches a causal inference regarding the socio-economic level of a school’s 9th- grade student population and the consequent choice of a vocational education stream rather than general education. The simplest way to examine this finding is by looking at the diagram on the following page. The lefthand side shows the impact on educational performance, the PISA Mathematics score, the right-hand side shows the impact on student choosing vocational education. The phenomenon causing the impact is a student belonging to a school where more than one-third of the students are from the lowest socio-economic quintile as defined by the ESCS index, called the ‘treatment schools’. The black bars represent frequency counts of different types of economic models, hundreds of models, each model considering a different combination of explanatory variables like student self-confidence, whether the student went to pre-school, the infrastructure index for the school and so on. The dotted white line shows the origin or the point of zero impact, and the important thing to note in Figure 2 is the number of black bars that are to the left or right of the white dotted line at zero. 17. Students from schools which have a disproportionately high number of students from lower socio-economic levels (ESCS) do not have a strong tendency for poor academic performance, but they have a strong tendency to go on to vocational school. We can see from the left panel that a bit more than half of the black bars are on the negative side of the white line. The graph indicates that there is a weak impact on student test performance, with nearly forty percent of the models showing a positive impact on test scores. Contrast this finding with the righthand side graph, where the large majority of black lines are to the right of the white dotted line – this can be seen as clear and strong tendency for the students from the `treatment group’ schools to go in for the vocational stream. We discuss the policy implications of his finding presently, after pointing out some more key findings in the report. 18. When we examine the variables that are related to improved performance and to the choice of vocational education, once again, we see the role of the school infrastructure, preschool education and teacher preparation. In addition to a discussion of the treatment impact, the report also presents the role of the explanatory variables used in the model. ESCS of the individual student and a related variable focused on cultural possessions are related to performance. Teachers having dual qualification also significantly affects student performance. The variable representing an index of the use of computers at school initially increases and then decreases – this tends to confirm the recommendation reported earlier regarding Collaborative Problem-Solving Skills, that the deployment and use of the ICT curriculum in Russian schools need to be investigated closely. World Bank Education Global Practice: Report P164840 12 Figure 2. Impact Assessment of Low Socio-Economic Level Probability Density Probability Density Distribution of Average Treatment Effect Distribution of Average Treatment Effect Mathematics Achievement Vocational School Choice 19. Student Confidence is an important variable that can be influenced by an educational policy that appears to be significantly related to both test performance and student choice of vocational education. Student confidence is an index variable from the TIMSS dataset that combines information from student responses about their self-assessment regarding mathematics and science. Students respond to statements such as “Mathematics is not one of my strengths,” or “I am good at working out difficult mathematical problems.” The findings appear to indicate that students who go to vocational school may be doing so for sociological and acculturated reasons rather than objective weakness regarding academic performance. The variable regarding student confidence needs to be studied more closely than was possible during this study to arrive at definitive policy conclusions. The preliminary conclusion would be that the issue of the image of vocational education in terms of the cognitive demands from students should be monitored closely, especially as vocational education is geared to support the development of the 21st-century worker. 5. Lessons from Resilient Schools 20. The resilient schools in Russia provide a rich basis for policy analysis, however, the research methods require time and resources. The research of resilient schools is a mixed method approach that includes (i) the quantitative side based on data collected using contextualization questionnaires as well as academic results, and (ii) the qualitative part represented by interviews with all participants of school educational process from resilient and deprived groups. The research in two regions of the Russian Federation - the Tomsk Oblast (Region) and the Republic of Sakha (Yakutia) required physical travels and conducting interviews with schools. Education Equity in the Russian Federation. Summary Report 13 21. International PISA data (2006-2015) shows that Russia is among several countries that have managed to increase the share of academically resilient students. The OECD defines academic resilience as the ability of 15-year-old students from disadvantaged backgrounds to perform at a certain level in PISA in reading, mathematics, and science that enables them to play an active role in their communities and prepares them to make the most of lifelong-learning opportunities. The percentage of resilient students in Russia has almost doubled since 2006 (see Figure 3). 22. High expectations in relation to students, stimulation of their educational activity and clear requirements for the results are identified as the most common features of resilient schools in Russia. The analysis revealed these characteristic features of Russian schools working in adverse social conditions, teaching the most difficult groups of students and nevertheless achieving high educational results. These requirements and expectations probably spread not only to the most successful students; they constitute the school culture. Figure 3. Percentage of resilient students in selected countries, 2006 and 2015 % 0 10 20 30 40 50 60 Hong Kong (China) Macao (China) Estonia Japan Canada Finland Chinese Taipei Korea Netherlands Slovenia Germany Norway Denmark Poland Australia United Kingdom Spain Russia 13% 25% France Austria Latvia Czech Republic Hungary Brazil Indonesia Source: OECD (2018), PISA 2006 and PISA 2015 Databases. World Bank Education Global Practice: Report P164840 14 23. Education policy pursued by resilient schools is similar to those fostered by sound high-performing schools. However, this policy differs from the policy found in schools with the same social well-being level but poor learning outcomes. If schools develop a lively and rich learning environment, keep the vibrant pace of living filled with learning events, promote partnership with students and involve them in school-level decision making, teachers and school administrations are to exert a lot of efforts. In schools where the learning process is not so active, and education activities are not aligned with learning objectives and re detached from relevant students’ needs education quality in general; suffers. Despite high academic performance requirements, schools create a student-friendly environment of psychological comfort and mutual trust. Of special note is the fact that, just like with resilient schools, the key figure that propels school development is an outstanding leader, school principal. 24. Regional level policies on schools and teachers support do play a role in fostering school resilience. In Tomsk region education authorities develop policies aimed at: (i) enhancing the professional exchange between schools, (ii) promoting joint cross-school learning events (contests, competitions, visiting lessons), (ii) supporting learning outcomes monitoring, (iii) monitoring and preventing negative fluctuations in teachers’ performance, and (iv) identifying, helping and supporting schools that are at risk. In Yakutia region, the study revealed another important finding - those teachers who are willing to cooperate and share experience do it rather at their initiative and no special support from the school administration. 25. The study identified a substantial variation in the teaching and learning objectives, expectations and requirements to students in different schools. Some schools do not have high expectations, do not guide students towards higher education and do not set clear requirements for outcomes. The success of schools is not related to the social context of the analyzed low performing schools in Yakutia; their context does not differ from the context of the most successful schools. It is relevant both to the school premises and student families. Students of the two low performing schools are not more ‘difficult’ than students of well-performing schools. 26. One of the important characteristics of school resilience is the vision of school goals. The very existence of a common vision, common values, shared an understanding of the school goals and objectives by the teaching team and the principal is key to school success. In Yakutia well performing and resilient schools all have such features, while low performing schools are missing these goals. 27. The resilient schools in two Russian regions share the common features including fostering performance, support of students learning, full-day programs, engaging in local traditions and community. The study identified the following school policy elements: (i) unconditional priority is given to education quality, solid knowledge, and strong performance, schools have high expectations of their students and teachers, closely monitor achievements, and provide tailored support to students based on their needs and abilities; (ii) schools support learning activity of their students, involve them in academic competitions and contests, create conditions for their travel to other regions and countries; (iii) schools keep students busy throughout the day and even during vacations providing for Education Equity in the Russian Federation. Summary Report 15 meaningful and educational leisure time; (iv) socialization is viewed as introduction of students to traditional agricultural practices, preparation for informed choice of occupations and familiarization with other cultural and social contexts; and (v) schools have a place of socio-cultural centers in their locations, enjoy support from local communities and in turn contribute to their development. 6. Exploring future ideas and new directions to enhance educational equity 28. Russian Federation needs to deepen its involvement with international assessments in three ways: Expand the sample sizes for regional representativeness; Invest in test development using modern techniques such as artificially intelligent agents; and Invest in analytical capacity to exploit modern analytical methods involving Big Data and machine learning. This report, like other elements of a substantive policy analytic literature on education in the Russian Federation, has benefited from readily accessible data regarding internationally benchmarked student assessment and the associated context. The resulting openly available data has enabled researchers and policy analysts to track educational progress in Russia over the past nearly two decades. Russia has now (May 2018 Presidential decree) set itself the task of being amongst the top ten countries in the world with regard to educational quality. In terms of PISA Russia is currently at the level of the OECD average, while in PIRLS and TIMSS Russia is already on top or high in the rankings respectively. The task of coming up from below average to average level can be seen in retrospect as easier to achieve than the pending task of getting to the top ten. The three suggested measures regarding international benchmarking have hopefully been shown by this report to deserve serious consideration: to increase the sample size, so that performance can be compared robustly, as has been done in other countries; to assume a leadership role regarding the development of modern testing methods to benefit not only Russia but other countries; and to deepen the institutional capacity to carry out analysis using the latest methods that provide very high clarity of policy insight. The investigation of this report regarding equity hints at how attention to equity would help attain the task of being in the top ten, but to meet the goal will require deepening of the investments that have already made in Russia on the issue of student assessment. 29. The investments in building robust and using reliable internationally benchmarked data regarding quality need to be supplemented by rigorously designed impact evaluation of programs. One phenomenon which is strangely lacking in the Russian context with an abundance of international benchmarking is accompanying literature on impact evaluation of programs. Recent years have seen massive investments from the Federal Government for the 5 out of 100 universities, innovative regional universities, enhancing the built environment program and the World Skills initiative to transform the image and reality of vocational education. Plans are being made for further investments. The effectiveness, efficiency and equity impact of these interventions would be considerably heightened if accompanied by rigorously designed impact evaluation. There appears to be a belief prevalent in policy circles that reputed international experts can be invited to Russia to advise about the `best practices’ that are followed in the rest of the world World Bank Education Global Practice: Report P164840 16 and which can be productively applied to the Russian context. While openness to new ideas is always a good thing, the fact remains that design and implementation of educational policies are highly dependent on history and culture, and rigorous scientific data in the Russian context is imperative for sound policy decisions – the most renowned Nobel prize winning economists cannot substitute for such data. The report’s investigation of equity associated with vocational education suggests a prime candidate for impact evaluation – the World Skills program, widely considered to be an effective means to improve the image of vocational education. 30. The role of digital education and how the transformation in digital education may be productively linked to the looming digital transformation of the economy requires a deep investigation only hinted at in the current study on educational equity. While exploring the issue of educational equity for collaborative problem solving skills and the issue of vocational education stream, this report describes a problematic finding regarding ICT in schools in the Russian Federation. It is possible that there are issues with both the design and the delivery of the curriculum. One emerging policy area concerns the topic of coding as a literacy. A forthcoming World Bank report will examine the issue of how in countries around the world, policy makers are introducing measures to teach computational literacy, that many consider essential for the 21st century, along with the traditional literacies of reading and mathematics. Coding as a literacy is only one aspect of the broad area of 21st-century skills and competencies, with the larger issue of ICT role in education needing a close examination from the viewpoint of policy and implementation. 31. A final emerging area of policy interest suggested in the report concerns extra-curricular education – early attention may prevent the emergence of an equity problem in this area. This report has pointed to the relationship between extra-curricular education and educational outcomes, particularly with regard to collaborative problem-solving skills, but also in relation to vocational education choice. While Russia performs better on education equity than OECD comparators, extra-curricular education may be an exception requiring policy attention. The report mentions how school provision of extra-curricular activities may suffer from inequity and how it is related to performance. This topic is another area where the World Bank is likely to devote attention within the realm of critical educational policy issues for 21st-century human capital. As described in the report, policy attention will prevent a scenario where students from lower socio-economic levels can close the gap on cognitive skills, only to see new gaps opened up regarding emerging areas such as collaborative problem-solving skills enhanced through modes such as extra-curricular education. Education Equity in the Russian Federation. Summary Report 17 Annex: Abstracts of five papers in this series Paper 1: Good Performance and Getting Better: Comparative Education Equity in Russia This paper summarizes the descriptive statistics of equity issues in Russia and examines the relationship of education performance and social equity. It also looks at the McKinsey methodology of motivation calibration as well as at other student’s characteristics to draw some viable and practical lessons. Lastly, the paper analyses the equity from the classical Gini perspective and provides information about Russia’s particular story of education equity. Paper 2: Social Equity, Vocational School and Educational Performance This paper examines the relationship between social equity and educational performance and the choice of going to vocational school. The paper uses a unique dataset combining PISA and TIMSS and utilizes a recently developed Bayesian econometric method that allows causal inference from non-experimental data. We find that socio-economic profile does impact educational performance, but it is not rare to overcome disadvantages, a phenomenon investigated further through another paper in this series that examines resilient schools. Paper 3: Collaborative Problem Solving Skills This paper examines the equity in the achievement of collaborative problem solving skills (CLPS). CLPS equity in Russia is better than OECD, but mainly because of the relative lack of high performers. Policies to improve levels of CLPS performance will likely not face an equity trade-off. These policies need to revise ICT related pedagogy, emphasize extracurricular activities and continue reducing disparities in infrastructure within federal subjects. Paper 4: Resilient Schools; This paper provides qualitative insight into the practices of resilient schools in Russia. The methodology of the paper was based on preliminary identification of the resilient schools based on the statistical and performance data, and further visiting the schools. The visits included structural interviews and observations. The school policies were analyzed in resilient schools, low performing schools, and high performing schools. Paper 5: Short Note on the Economic Value of Improvement in Education Equity The paper combines time series data on regional GDP and information about variation in cognitive achievement in cross section across the federal subject in the Russian Federation. Using conservative extrapolated estimates based on the international literature regarding the causal link from improvement in cognitive World Bank Education Global Practice: Report P164840 18 outcome to long-term impact on economic growth, the paper determines that a one standard deviation improvement in cognitive achievement will lead to greater economic growth of approximately 25 trillion Russian Rubles equivalent to about USD 400 billion or one-third of current Russian GDP. Figure 4. Difference in PISA 2015 Scores Between Top and Bottom Socioeconomic Quintiles – Different Countries Education Equity in the Russian Federation. Summary Report Source: World Bank staff calculations based on PISA 2015 database 19 World Bank Education Global Practice: Report P164840 Figure 5. Share of Low Performers among Bottom ESCS Students by Subject (%) Math Science Reading 88,1 91,9 85,7 73,9 61,6 74,2 60,9 65,2 60,2 47,0 47,0 43,6 29,1 30,6 32,2 24,4 25,5 28,0 19,4 19,8 16,7 26,6 17,7 19,0 10,4 10,0 12,7 2000 2003 2006 2009 2012 2015 2000 2003 2006 2009 2012 2015 2000 2003 2006 2009 2012 2015 FIN HKG RUS FIN HKG RUS FIN HKG RUS Source: World Bank staff calculations based on PISA 2000-2015 database 20 Education