92913 Strengthening Kazakhstan’s education system An Analysis of PISA 2009 and 2012 Education Global Practice Europe and Central Asia Region Acknowledgments This report was prepared by Lucas Gortazar, with analytical or writing support from Martin Moreno, Mark Zelman (Consultants), Jeremie Matthew Amoroso, Daniel Kutner, and Katia Herrera-Sosa (Education Sector, Europe and Central Asia Region, the World Bank). The report is part of the World Bank’s education sector knowledge and advisory services provided to the Government of the Republic of Kazakhstan between 2013 and 2015 under Joint Economic Research Program and led by Keiko Inoue (Senior Education Specialist, Europe and Central Asia Region, the World Bank). This report is part of the PISA Country Series conducted by the Education Unit in the World Bank’s Europe and Central Asia Region and is the second World Bank analysis of Kazakhstan’s education system’s performance based on analysis of PISA data. The team is deeply grateful to Alberto Rodriguez, Andrea Guedes, Igor Kheyfets, Mohammed Ihsan Ajwad, Aliya Bizhanova and Anara Sheshmukhanova for their helpful guidance, comments, and assistance. Finally, the team would like to acknowledge the contribution of art director Nicholas Dehaney and the editor, Amy Gautam. Contents Abbreviations and Acronyms Executive Summary 02 1 An Overview of Kazakhstan’s Performance on PISA 06 2 Determinants of Education Quality: Winning at What Cost? 14 Expenditures in education 15 Trends in performance 2009-2012 17 Inequities in performance 18 Gender disparities 19 Urban-rural disparities 19 Linguistic disparities 22 What is in hands of policy makers? 22 The impact of preschool 22 School types and social stratification 23 3 Drilling Down Further Into Math and Reading Skills 26 Math skills in PISA 2012 27 Reading skills in PISA 2009 29 Teacher practices and student learning strategies 30 4 Findings and Recommendations 32 References 34 Annex 36 Figures Figure 1 Kazakhstan’s PISA performance by discipline, 2009 and 2012 09 Figure 2 Comparison of PISA 2012 scores for select countries and ECA/OECD averages 09 Figure 3 Distribution of Kazakh students by proficiency level in math and reading 10 Figure 4 Kazakh students’ performance in reading in PISA 2009 and 2012 by socioeconomic quintile 12 Figure 5 Relationship between public education expenditures per pupil and PISA 2012 reading performance, selected countries 15 Figure 6 Change in performance by subject and student performance group from 2009 to 2012 16 Figure 7 Math performance analysis by characteristics, 2009-2012 17 Figure 8 Reading performance analysis by characteristics, 2009-2012 18 Figure 9 Index of equality of opportunities: Kazakhstan and other ECA countries, 2012 19 Figure 10 Kazakh students’ performance in reading and math by gender, 2009 and 2012 20 Figure 11 Kazakh students’ performance in reading and math by geographic location, 2009 and 2012 20 Figure 12 Kazakh students’ reading performance by medium of instruction and region, 2009 and 2012 21 Figure 13 Kazakh students’ performance in math and reading by medium of school instruction, 2009 and 2012 22 Figure 14 Index of School Social Stratification for OECD and Kazakhstan, 2009 and 2012 23 Figure 15 Kazakh students’ performance in reading by school type, 2009 and 2012 24 Figure 16 Math contents and process categories in PISA 28 Figure 17 Kazakhstan’s PISA 2012 performance on different math subscales compared to the average math performance 29 Figure 18 Kazakhstan’s PISA 2012 performance on different reading subscales compared to the average reading performance 30 Figure 19 Kazakhstan reading gap with comparator countries as a function of teacher practices and learning strategies 31 Boxes Box 1 Kazakhstan’s Education System 08 Box 2 PISA’s Index of Economic, Social, and Cultural Status 11 Tables Table 1 Determinants of achievement in PISA by individual and school characteristics 16 Abbreviations & ECA ECD ESCS Acronyms Europe and Central Asia Early Childhood Development Economic, Social, and Cultural Status (Index) GDP Gross domestic product OECD Organisation for Economic Co-operation and Development OLS Ordinary least squares PIRLS Progress in International Reading Literacy Study PISA Programme for International Student Assessment RIF Re-centered influence functions SES Socioeconomic status TIMSS Trends in International Mathematics and Science Study TVE Technical Vocational Education UN United Nations UNESCO United Nations Educational, Scientific and Cultural Organization An Analysis of PISA 2009 and 2012 1 2 Strengthening Kazakhstan’s Education System: Executive Summary Kazakhstan’s PISA1 2012 performance improved markedly compared to 2009, and indicated a narrowing achievement gap among students. Math and science performance improvements equivalent to more than half a year of schooling were achieved. According to the OECD, 40 points in PISA is equivalent to what students learn in one year of schooling. The improvements reduced the gap with other countries in Europe and Central Asia (ECA) by almost half. Moreover, the performance improvements of the lowest achievers in 2009 and 2012 outpaced those of their higher achieving peers at a rate that compares favorably against OECD countries. However, performance on reading improved only marginally and overall reading achievement remained low, with some groups of students actually performing worse in 2012. Kazakhstan’s PISA performance places it significantly behind other countries, especially in reading. Countries with income per capita levels similar to Kazakhstan’s (namely Turkey and Russia) performed significantly better in math, science, and reading. Most importantly, Kazakh reading scores still lag about one year of schooling behind the ECA average and almost two years of schooling behind OECD. Public expenditures on education are lowest in Kazakhstan compared with other PISA 2012 participating countries, which likely hampers the country’s ability to ensure effective learning for all. Any increase in public spending on education will have to be aligned with careful policy reform decision making, since resources alone do not guarantee attainment of desired education outcomes. Kazakhstan’s PISA 2012 performance improved markedly compared to 2009, and indicated a narrowing achievement gap among students An Analysis of PISA 2009 and 2012 3 For every improvement in PISA performance, Kazakhstan took one step back. Despite improvements in math and science, reading achievement suffered. A more in-depth look at country performance across groups reveals a series of challenges that need to be addressed in the short term. Although performance in reading remains the major challenge for Kazakhstan, achievement disparities across all three subjects will require the attention of policy makers with respect to resource allocation, human resource deployment, and policies in the near future. Special emphasis needs to be placed on: 1 Boys’ reading: Although there is no gender disparity in math or science performance, girls perform better than boys by the equivalent of one full year of schooling in reading. 2 Low socioeconomic status (SES) students: A gap equivalent to 1.5 years of schooling exists between students from families in the highest and lowest socioeconomic quintiles. 3 Kazakh language schools:2 Students at Russian language schools perform the equivalent of 1.5 years higher in reading and one year higher in math than peers attending Kazakh language schools. 4 Professional school (technical vocational education, or TVE, lyceum) students: Students in professional schools are behind their peers in basic and general secondary schools and colleges by more than 1.5 years of schooling in reading and one year of schooling in math. 5  azakh language spoken at home: Students from households where Russian is spoken K performed better by about one year of schooling in reading and by about 1.5 years of schooling in math. An in-depth analysis into math and reading skills shows large variations in performance compared to other countries. PISA rotates the in-depth assessment of skills by subject area each time it is administered. PISA 2009 focused on reading, while PISA 2012 focused on math; PISA 2015 will focus on science. While all three subjects are tested each time, the rotating in-depth assessment of subject-specific skills allows for the evaluation of content and competency specific skills in each area. In both PISA 2009 and 2012, there was a large variation in achievement across subscale categories compared with other 4 Strengthening Kazakhstan’s Education System: countries. For math, PISA 2012 shows that students performed relatively better in content skills in algebra and geometry, but not as well in problems related to data and uncertainty. Moreover, students had difficulty relating mathematical problems to their real-life contexts and deriving applications of the problems. In the 2009 PISA in-depth reading assessment, students better understood what PISA calls continuous texts (such as information presented in a paragraph format) than non-continuous texts (such as information accompanying a figure or chart). As with math, skill weaknesses demonstrated a disconnect between what students read and their ability to apply that to real-life situations: in particular, Kazakh students’ learning strategies (as reported by the PISA students’ questionnaire) showed important gaps relative to those of students in OECD countries. The following policy recommendations could be explored to build on the impressive overall gains made in PISA 2012: Ensure that basic reading skills are achieved. Reading performance appears to be the greatest (i)  challenge in Kazakhstan’s education system. Reading is a skill that begins with a solid foundation in early grade literacy. Rather than waiting for national or international large-scale assessments mostly focused on grades 4 and above, teachers should regularly assess early grade literacy in the classroom and then intervene with the appropriate combination of pedagogical approaches. Early grade literacy is a critical stepping stone for further skills acquisition, and a comprehensive approach that brings together teachers’ professional development, learning materials, and assessment tools development is necessary. Kazakhstan should consider implementing a program that includes five important steps to improve reading in the early grades: (i) train teachers how to teach reading; (ii) maximize instruction time in the classroom; (iii) put appropriate books in the hands of children; (iv) implement appropriate language policies and provide mother tongue-based instruction; and (v) measure reading skills. International evidence shows that the quality of teaching is the most important school-based predictor of student learning; thus ensuring teacher effectiveness should be prioritized. Teaching quality is affected by many factors, from teachers’ skills to their practices. In Kazakhstan, teachers appear to be especially challenged in how they manage assignments and relate knowledge taught to students’ lives. In turn, students are disadvantaged by limited knowledge and ineffective learning strategies, such as relating new knowledge to other contexts and applying study methods other than memorization. This study’s results also found large discrepancies in performance between Russian and Kazakh language schools, as well as among regions. While relatively low student performance points to the need to enhance teacher effectiveness at the national level, improving the pedagogical skills of teachers in Kazakh language schools and underperforming regions should be a priority. Incentive packages to ensure that the highest performing teachers are assigned to underperforming schools could also be considered. An Analysis of PISA 2009 and 2012 (ii)  Establish a comprehensive Early Childhood Development (ECD) program, with an emphasis on at-risk groups. The analysis of the determinants of quality of education concludes that unlike in most countries, attending preschool in Kazakhstan has no impact on student achievement levels at age 15, controlling for other relevant factors. These results may imply that the quality of the preschool system was poor in the late 1990s and early 2000s. In recent years, Kazakhstan has allocated considerable resources through the Balapan program to expand access to preschools, but the focus has not been on improving quality or ensuring that young children acquire key cognitive and non-cognitive skills that serve as the building block for lifelong skills formation. Kazakhstan therefore should assess the quality of preschools, either by observing teachers and their effectiveness, or by monitoring child development outcomes to ensure that additional investments have the intended impact. Recent evidence also suggests that adequate health, nutrition, and stimulation for children aged 0-3 are critical for brain development, which also affects skills acquisition. Currently, social services for children in the age groups 0-3 and 4-6 are not well coordinated. An effective ECD program should therefore integrate health and social services for children aged 0-3, while ensuring a smooth transition to quality preschool programs and then primary school readiness. The return on investments in ECD is highest for at-risk children and their families, thus in a context of limited resources, a targeted approach is recommended. Develop an integrated assessment system. An integrated assessment system should ensure that (iii)  student assessments reflect the content of the curriculum and are objective and fair, and that results are provided in a way that supports the system to identify areas where further research or remedial action is needed. While the PISA results present a good starting point to discuss education reform, there are limits to the conclusions that can be drawn. For example, PISA is a competency-based, rather than a curriculum-based, assessment. There are thus limitations to the data and some information gaps are evident in the determinants analysis, especially for understanding the factors driving the improvements in math and science scores. Summative national student assessments, conducted at key milestone grades, could be reformed to collect data on the missing variables. Interim and formative assessments are also necessary to assess student progress with respect to the curriculum or to drill down deeper on mastery of a specific skill. Education policy makers may wish to consider not only results from international student assessments such as PISA but also national assessments at various levels. The results from this report will hopefully contribute to the ongoing debate about student assessment system reform in Kazakhstan. 6 Strengthening Kazakhstan’s Education System: 1. An overview of Kazakhstan’s performance on PISA An Analysis of PISA 2009 and 2012 7 Education and skills are critical for the development of countries and individuals. International evidence suggests that quality of education is one of the most important determinants of long-term economic growth.3 Hanushek and Woessmann (2007 and 2010) looked at a wide range of student assessment surveys from 1960 onward, including the Trends in International Mathematics and Science Study (TIMSS), the Programme for International Student Assessment (PISA), and the Progress in International Reading Literacy Study (PIRLS). They estimated that an improvement of 50 points in PISA scores would imply an increase of 1 percentage point in the annual growth rate of GDP per capita.4 Beyond economic growth, education improves the living standards of individuals, as those with more education are able to acquire higher-order skills, making them more productive and employable, and extending their labor market participation over their lifetime. These factors lead to higher earnings and a better quality of life. Formal schooling also contributes to development of behavioral skills like longer attention span, motivation, self-confidence, and physical and emotional health, all important determinants of socioeconomic mobility. Education is a key ingredient for reducing inequality and increasing shared prosperity. Understanding the determinants of education quality can play an important role in shaping effective evidence-based education policy. The PISA database is a great resource in the pursuit of this analysis. PISA is a tool for measuring education quality across countries. Introduced in 2000 by the Organisation for Economic Co-operation and Development (OECD), PISA is a worldwide study of 15-year-old school students’ performance in three different disciplines: mathematics, science, and reading. PISA is administered every three years and focuses on the competence of students and their ability to tackle real-life problems in the three disciplines and emphasizes skills that are critical for individuals’ personal and professional development. PISA only assesses students who are enrolled in school, making it the most realistic, and internationally comparable, snapshot of a country’s education system. However, if dropout rates are high, the results may not be representative of a country’s cohort of 15-year-olds. PISA’s scoring system is standardized so that the mean score for each discipline among OECD countries in year 2000 is 500 points, with a standard deviation of 100 points. According to the OECD, 40 points in PISA is equivalent to what students learn in one year of schooling.5 Kazakhstan’s education system (see Box 1) was assessed in the two most recent PISA rounds of 2009 and 2012. Kazakhstan’s participation in PISA allows us to benchmark its performance with other countries, measure the extent of the country’s success in promoting education quality, and gauge whether system inequities have been reduced over time. Kazakhstan’s PISA 2012 performance in math and science is a marked improvement above 2009 levels (Figure 1). Since 40 points are equivalent to one year of schooling, the math and science gains were each equivalent to more than half a year of schooling. Performance in reading improved marginally by 3 points, remaining at quite low levels. On average, significant changes tend to be observed only in score comparisons exceeding 10 points; thus the limited improvement in reading is within the margin of error. International comparisons aside, Kazakhstan has achieved significant performance gains in math and science since its first assessment in 2009. 8 Strengthening Kazakhstan’s Education System: Box 1 kazakhstan’s education system Kazakhstan has a population of 17.2 million people, with Kazakhs comprising 64 percent of the population and Russians 23 percent. Other ethnic groups include Uzbeks, Ukrainians, and Uyghurs. The education system serves more than 4.5 million students from the pre-primary level through tertiary education. According to UN estimates, Kazakhstan’s school-age population is projected to grow by more than 20 percent between 2015 and 2030, reflecting the impact of high fertility and low emigration. Primary education is the first stage of compulsory education and spans a period of four years. The majority of children enroll at age seven, but six-year-olds can be admitted by passing an entrance test. Secondary education starts at fifth grade and consists of two levels: basic comprehensive (grades 5-9) and senior comprehensive (grades 10-11). After basic comprehensive, students can continue to senior comprehensive for two more years or enter technical vocational schools (colleges) for three years of study. After completing secondary education, students may progress to higher education institutes and universities. School System Structure Age Grade Level of Education Preschool education 5-6 Preschool Kindergartens, crèches Pre-primary (day nurseries) Secondary (complete) education 6-11 1-4 Primary comprehensive Secondary 11-16 5-9 Basic comprehensive Secondary 16-18 10-11 Senior comprehensive, Secondary gymnasia 16-19 10-12 Technical vocational lyceums and colleges Secondary Higher education 19-22 13-16 Bachelor’s degree University Diploma of Specialist Universities, academies, institutes 22+ 17+ Graduate studies University Source: UNESCO World Data on Education, 6th edition, 2006-07. An Analysis of PISA 2009 and 2012 9 Figure 1 Kazakhstan’s PISA performance by discipline, 2009 and 2012 430 +27 +25 +3 400 390 393 405 432 400 425 370 Reading Math Science Source: PISA 2009 and 2012. n 2009 n 2012 Figure 2 Comparison of PISA 2012 scores for select countries and ECA/OECD averages 520 One year of 500 schooling 480 460 432 425 440 393 420 400 380 360 Kazakhstan Malaysia Chile ECA Russia Turkey OECD Malaysia Kazakhstan Turkey Russia Malaysia Kazakhstan Chile Turkey ECA Russia Chile ECA OECD OECD Source: PISA 2012.6 Reading Math Science 10 Strengthening Kazakhstan’s Education System: Figure 3 Distribution of Kazakh students by proficiency level in math and reading 70% 60% 50% n 2009 n 2012 40% 30% 20% 10% 0% Basic Level 2 Level 3 Level 4 Level 5+ Basic Level 2 Level 3 Level 4 Level 5+ level 2 level 2 Lacking basic Lacking basic math skills reading skills Source: PISA 2009 and 2012. Despite this improvement, Kazakhstan’s 2012 Kazakhstan has reduced its overall share results remained below the levels of regional of students below basic proficiency levels, comparator countries, especially in reading. although the share remains high. PISA The Achilles heel of Kazakhstan’s education system categorizes scores in six levels of proficiency; students is reading performance, in which students score who score below level 2 in the math and reading tests lower than other countries with similar economic are considered functionally innumerate and illiterate, development, such as Malaysia and Chile. And respectively.7 According to the 2012 data, around 45 despite a marked improvement in math and science percent of 15-year-old students in Kazakhstan score performance compared to its initial assessment in below level 2 in math, meaning that they are unable to 2009, Kazakhstan’s scores are still lower than those understand and solve simple math problems, severely of similar Europe and Central Asia (ECA) countries, limiting their development and skill acquisition namely Turkey and Russia. Kazakh scores lag 35 process. The picture is worse for reading – about 57 points behind the ECA average for math and 71 points percent of Kazakh students lack basic reading skills – behind for reading, and even further behind the OECD with little improvement since the 2009 assessment. averages for all disciplines. The performance gap According to OECD standards, 57 percent of students between Kazakhstan and the OECD is equivalent to are able to only “locate one or more independent more than 1.5 years of schooling in math and science pieces of explicitly stated information” or make and 2.5 years in reading. “simple connections.” That being said, an important An Analysis of PISA 2009 and 2012 11 The Achilles heel of Kazakhstan’s education system is reading performance, in which students score lower than other countries with similar economic development, such as Malaysia and Chile. part of the progress made by Kazakhstan since 2009 Box 2 PISA’s Index of Economic, Social, was due to improvements of students performing and Cultural Status below level 2 in both math and science, but not reading. Countries like Turkey and Russia have a Created by the OECD, PISA’s Index of lower share of students below level 2 (42 percent Economic, Social, and Cultural Status (ESCS) and 24 percent, respectively), and their progress in is a multidimensional measurement that the last decade was driven mainly by the improved incorporates information reported by students performance of low achievers. on their family’s wealth and occupational, educational, and cultural background. It is derived from a combination of three other indexes: (i) an index of the highest occupational status of parents, indicating not only labor market status, but also the type of job held by parents; (ii) an index based on the highest level of parental education in years of schooling; and (iii) an index of family home possessions, which itself consists of a combination of the family’s possessions (such as cars, bathrooms, or technological devices) and educational resources (such as desks, computers, textbooks, and the number of other books), as well as the type of cultural possessions (such as the type and genre of books or works of art). The ESCS Index is the most important determinant of student achievement and is therefore crucial for analysis of the quality of education. Source: OECD 2014. 12 Strengthening Kazakhstan’s Education System: Figure 4 Kazakh students’ performance in reading in PISA 2009 and 2012 by socioeconomic quintile 480 +8 +6 +5 +1 -10 440 400 360 353 375 381 391 396 407 408 436 426 345 320 1 2 3 4 5 Bottom 20% Top 20% Source: PISA 2009 and 2012. n 2009 n 2012 5x5=25 6x5+30 7x5=35 An Analysis of PISA 2009 and 2012 13 The lowest socioeconomic status (SES) students that since 2009, the group in the bottom 20 percent of improved the most over 2009, while students SES improved the most in all three subjects. Students from the highest SES actually performed worse from the top 20 percent improved less in math in reading. In the PISA 2009 report, it was reported than their peers in the bottom SES group, and even that without sustained improvements for all, poor experienced lower performance in reading between students are unlikely to increase their future living the 2009 and 2012 assessments – a decline of 10 standards. While the report pointed out that average points (Figure 4). PISA 2012 differences in math and score growth is important, it is also crucial to foster reading scores between students in the highest and improvements among students from the lowest lowest quintiles of socioeconomic status are 60 and socioeconomic groups of a country’s population. In 73 points, respectively (between 1.5 years and 2 years this report, the OECD’s Index of Economic, Social, of education), while the OECD average differences and Cultural Status (ESCS) is used as a measure of are 100 points in math (2.5 years) and 90 points in students’ SES background (see Box 2). Results show reading (2.25 years). 14 Strengthening Kazakhstan’s Education System: 2.Determinants of education quality: Winning what costs at ? An Analysis of PISA 2009 and 2012 15 Expenditures in education Although Kazakhstan’s level of public expenditure per student is low relative to other countries in ECA, this doesn’t fully explain its low PISA performance (Figure 5). Kazakhstan spends the lowest amount on education per pupil among the ECA countries that participated in PISA 2012, even though the country’s GDP per capita is close to the ECA average. Expenditure per pupil in Kazakhstan is equivalent to 11.7 percent of GDP per capita, much lower than that of PISA top performers like Poland (23.9 percent), Japan (22.8 percent), Switzerland (27.1 percent), and Estonia (25.8 percent). Other upper-middle-income countries with similar economic development indicators, like Chile (15.3 percent) and Malaysia (19.1 percent), also devote more national resources than Kazakhstan. To ensure high quality provision of education, a minimum level of financial resources in education is necessary. However, higher levels of expenditures do not necessarily imply better learning outcomes. Countries like Poland or Estonia devote less per student compared to countries with similar economic development, but manage to ensure that the vast majority of students are well prepared in literacy and numeracy skills by the end of their secondary education. While a causal link between public expenditure per pupil and improved education outcomes is not a certainty, Kazakhstan’s low expenditure per pupil should be reconsidered, particularly given the country’s aim to rank among the world’s high-income countries by 2030. Figure 5 Relationship between public education expenditures per pupil and PISA 2012 reading performance, selected countries East Asia 550 Japan Hong Kong 530 Korea Europe & Poland Finland Estonia Central Asia 510 Czech Republic Netherlands Switzerland Australia Belgium PISA score in mathematics, 2012 France United Denmark 490 Hungary Italy Iceland Austria Norway Latvia Spain States Slovenia Sweden 470 Croatia Isreal Portugal Lithuania Slovak Republic Western Europe 450 Thailand Serbia Cyprus & US/Canada 430 Romania Chile Bulgaria Mexico 410 Latin American Indonesia Colombia Malaysia 390 Argentina PeruKazakhstan 370 350 0 2000 4000 6000 8000 10000 12000 14000 16000 Public expenditures per pupil (in PPP dollars), UNESCO 2012 or latest Source: PISA 2012 and UNESCO 2012. Note: The curve represents a logarithmic approximation of the scatter plots. 16 Strengthening Kazakhstan’s Education System: Figure 6 Change in performance by subject and student performance group from 2009 to 2012 40 Performance difference between each content/process Low-Achiever 30 subscale and the average mathematics scale 20 Middle-Achiever 10 0 -10 Reading Math Science -20 High-Achiever Source: Authors’ calculations based on PISA 2009 and 2012. Note: Low-, middle-, and high-achieving students are defined as those in the 20th, 50th, and 80th percentiles of performance, respectively. Table 1 Determinants of achievement in PISA by individual and school characteristics Individual characteristics Age Gender Socioeconomic status (ESCS Index) Ethnicity Grade Participation in pre-primary education School characteristics Peer Average socioeconomic status (ESCS Index) School dropout rate Share of minorities School Quality of educational resources (Index) resources Student-teacher ratio Location (urban or rural) Parental engagement Type of school (public or private) School Responsibility over curriculum and autonomy assessment (Index) Responsibility over human and financial resources (Index) Source: Greenwald, Hedges and Laine 1996; Hanushek 2009. An Analysis of PISA 2009 and 2012 17 Figure 7 Math performance analysis by characteristics, 2009-2012 Low Achiever 45 Average Achiever Middle Achiever High Achiever 40 35 30 25 nE  ndowment: School resources 20 n Endowment: Peer characteristics 15 n Endowment: Individual 10 characteristics n Systemic change 5 n Other 0 n Actual change 2009-2012 -5 -10 Source: Authors’ calculations based on PISA 2009 and 2012. Note: Determinants analysis was done using a threefold Oaxaca-Blinder method on RIF-regressions for each quantile of the distribution of performance (Firpo, Fortin and Lemieux 2009). Low-, middle-, and high-achieving students are defined as those in the 20th, 50th, and 80th percentiles of performance, respectively. See more in Table A.2 of the Annex. Trends in performance 2009-2012 The performance improvements of the lowest The drivers of change in math (Figure 7) and achievers in 2009 and 2012 outpaced those reading (Figure 8) performance among low-, of their higher achieving peers in all three middle-, and high-achievers from 2009 to 2012 subjects (Figure 6). The substantial improvements are varied and do not show consistent patterns. for low achievers show that the education system Table 1 categorizes PISA variables into individual has successfully narrowed the gap among student and school characteristics, with subgroups of performance groups. However, high achievers only variables within school characteristics: peer, school marginally improved in math and science, and even resources, and school autonomy. An analysis of the decreased their performance in reading. Although drivers of change in performance between 2009 the system has ensured gains for those students with and 2012 split students into low, middle, and high the most difficulties, it is fundamental to guarantee achievers. Besides the four categories of individual that all students have the opportunity to reach their and school characteristics, “systemic change” was also full potential, to ensure the credibility of the education considered, referring to the change in the effect that system. In other words, when a country is trying to individual and school characteristics had on learning improve overall student learning outcomes, gains by between 2009 and 2012. There are also residual the lowest achieving students should not come at the effects, called “others,” that are not captured by PISA expense of the top achievers. data; these include governance and management practices, social and cultural norms, and teacher 18 Strengthening Kazakhstan’s Education System: Figure 8 Reading performance analysis by characteristics, 2009-2012 Low Achiever 40 Average Achiever Middle Achiever High Achiever 30 20 nE  ndowment: School 10 resources n Endowment: Peer 0 characteristics n Endowment: Individual -10 characteristics n Systemic change n Other -20 n Actual change 2009-2012 -30 Source: Authors’ calculations based on PISA 2009 and 2012. Note: Results decomposition was done using a threefold Oaxaca-Blinder method on RIF-regressions for each quantile of the distribution of performance (Firpo, Fortin and Lemieux 2009). Low-, middle-, and high-achieving students are defined as those in the 20th, 50th, and 80th percentiles of performance, respectively. See more in Table A.2 of the Annex. and principal effectiveness. The results show that Inequities in performance improvements in math performance between 2009 and 2012 were largely driven by systemic change Access to quality education is relatively equal variables for low and middle achievers. In this case, in Kazakhstan. As discussed above, the difference the substantial improvements in the performance in math scores between students in the highest and of students from Kazakh language and rural schools lowest SES quintiles is large in Kazakhstan, but has primarily benefited the poor performers (further narrowed since 2009. Analysis indicates that the details below). For high achievers, the improvements main individual characteristics (gender, age, and were largely explained by peer characteristics. The SES) have less impact on performance in Kazakhstan only negative effect on change in math performance compared with other countries in the region (Figure was from the set of “other” variables, which was 9), and explain only one-fifth (21 percent) of the especially large for high achievers. For the change differences in students’ reading achievement.8 in reading performance, the pattern was different: Compared to neighboring countries, the weight of systemic change had an overall negative effect, individual characteristics in Kazakhstan is among especially for high achievers. The improvement in the lowest, reflecting the high equality of educational low achievers’ performance was partly explained opportunities. by improvements in individual and school characteristics, but the “other” variables largely explained the change. An Analysis of PISA 2009 and 2012 19 Figure 9 Index of equality of opportunities: Kazakhstan and other ECA countries, 2012 0.35 0.30 More equality PISA 2012 Scores of opportunities 0.25 0.20 0.15 0.10 Estonia Russia Kazakhstan Croatia Poland Czech Republic Turkey Romania Slovenia Montenegro Serbia Lithuania Latvia Hungary Slovak Republic Bulgaria Source: Authors’ calculations based on PISA 2012. Note: The index is the percent of the variance in reading scores explained by the main predeter mined charactristics (age, gender, and socioeconomic status) in a linear regression (Ferreira and Gignoux 2011). Gender disparities Urban-rural disparities Performance in math does not vary by gender, The gap between rural and urban students is while reading remains a major challenge for shrinking quickly in all three subjects, especially Kazakh boys. In math, the increase was shared math and science (Figure 11). Rural students by boys and girls, such that both improved their significantly improved their performance in math, performance equally. Similar to OECD countries, equivalent to a full year of schooling. The difference Kazakhstan has no gender gap in math performance. between rural and urban students’ reading scores However, Kazakh girls significantly outperformed is 32 points and only 13 points in math. The score boys in reading, both in PISA 2009 and 2012. In many difference between urban and rural locations in math countries, girls score higher than boys on the reading is less than half of the ECA average of 27 points. The scale, and Kazakhstan is no exception. For example, shrinking gap in reading, however, is also partly due to girls score 46 points higher in reading in Turkey and lackluster performance by urban students in Russian 40 points higher in Russia. Although high in absolute language schools in 2012. terms, Kazakhstan shows similar gender difference relative to these countries (40 points, or about one The performance gap is narrowing between year of schooling). students in Kazakh and Russian language schools (Figure 10). In urban areas, Russian language schools performed worse in 2012 than in 2009, by the equivalent of half a year of schooling. 20 Strengthening Kazakhstan’s Education System: Figure 10 Kazakh students’ performance in reading and math by gender, 2009 and 2012 Girls Boys Girls Boys +27 +27 440 +2 400 +5 405 432 405 432 412 414 369 374 360 Math Reading Source: PISA 2009 and 2012. n 2009 n 2012 Figure 11 Kazakh students’ performance in reading and math by geographic location, 2009 and 2012 485 +14 440 -5 +8 +39 395 413 408 368 376 424 438 386 425 350 Urban Rural Urban Rural Reading Math n 2009 n 2012 Source: PISA 2009 and 2012. An Analysis of PISA 2009 and 2012 21 Figure 12 Kazakh students’ reading performance by medium of instruction and region, 2009 and 2012 Kazakh School Russian School Kazakh School Russian School 485 +10 -23 +15 -7 440 395 377 387 461 438 351 366 416 409 350 Urban Rural n 2009 n 2012 Source: PISA 2009 and 2012. The 40 point gain in math performance among Kazakh language schools is equivalent to one year of schooling (Figure 13), more than three times the improvement seen in Russian language schools. 22 Strengthening Kazakhstan’s Education System: Figure 13 Kazakh students’ performance in math and reading by medium of school instruction, 2009 and 2012 Kazakh School Russian School Kazakh School Russian School 485 +40 -12 +16 -13 440 395 377 417 446 458 355 371 444 431 350 Math Reading n 2009 n 2012 Source: PISA 2009 and 2012. However, although Kazakh language schools in the in 2009 (1.5 years of schooling) to 41 points in same areas improved in the same period, the gap 2012 (one year of schooling), although it remains between the two types of schools remained 51 points high even after controlling for socioeconomic in urban areas, equivalent to more than one full year characteristics.9 However, there was a decline in of schooling. In rural areas, Kazakh language schools the share of Russian language schools, from 40 performed worse than those in urban areas, but the percent in 2009 to 36 percent in 2012. One reason gap between these schools and Russian language for the declining performance of Russian language schools narrowed to slightly more than one year of schools may be the emigration of high-achieving schooling. In urban and rural areas, the performance Russian-speaking students, which may explain the of Russian language schools declined but was within decreased performance of the remaining Russian- the margin of error in rural Russian language schools. speaking students in reading (but not in math). The performance of urban and rural Russian language Linguistic disparities schools declined in all subjects. Kazakh language schools showed significant improvements in all three subjects, while What is in hands of policy Russian language schools showed less makers? impressive improvements (and a decline in reading performance). The 40 point gain in The impact of preschool math performance among Kazakh language schools is equivalent to one year of schooling (Figure 13), Analysis of the determinants of education more than three times the improvement seen in quality shows that preschool education does Russian language schools. The math gap between not play an important role in predicting student the two types of schools declined from 69 points performance at age 15. Socioeconomic factors, An Analysis of PISA 2009 and 2012 23 Figure 14 Index of School Social Stratification for OECD and Kazakhstan, 2009 and 2012 More segregated 0.54 0.52 OECD Average 0.50 Less segregated 0.48 system Ü 0.46 Kazakhstan 0.44 0.42 Less segregated 0.40 2009 2012 Source: Authors’ calculations based on PISA 2012. Note: A higher index value (between 0 and 1) indicates a higher correlation between student and school SES. gender, grade when PISA was taken, or the type of School types and social stratification school attended by a student may be important (see Table A.1 in Annex), but the analysis of determinants Stratification in Kazakhstan’s education system did not find a linkage between student performance has decreased since 2009 (Figure 14). The Index and preschool attendance. Although 35 percent of of School Social Stratification is defined as the students who took the PISA 2012 attended at least correlation between the PISA individual SES and one year of preschool, this did not have an impact the average school SES.11 In a world without social on their performance.10 In fact, the gap in scores stratification (thus an index equal to zero), families between students who attended at least two years of from different socioeconomic backgrounds would preschool and those who did not attend any preschool randomly settle across the country and students from was only 32 points, compared to an average 66 and 67 different backgrounds would study together, resulting point difference in non-OECD and OECD countries in more diverse schools. However, households tend participating in PISA, respectively. Global evidence to co-locate in neighborhoods with other households shows that providing quality preschool education is similar to them, and students tend to attend school crucial for promoting children’s social, emotional, with peers who have a similar socioeconomic status physical, and cognitive development; it also increases as a result of spatial inequalities. A lower score in school readiness, which helps with the acquisition of the stratification index suggests that socioeconomic important skills throughout the lifecycle (Heckman diversity in schools is higher. While Kazakhstan’s and LaFontaine 2010; Heckman 2008; Engle et al. change between 2009 and 2012 is not significant 2011). Kazakhstan recently invested considerable based on this scale, school stratification is important amounts in expanding access to preschool. The PISA for policy decisions since it reflects how households results suggest that high-quality preschool education segregate themselves by living in areas with similar is critical to achieve the magnitude of positive impact households; it shows that higher-income families have on student performance seen in most countries. a tendency to send their children to schools with more 24 Strengthening Kazakhstan’s Education System: Figure 15 Kazakh students’ performance in reading by school type, 2009 and 2012 480 440 +3 +1 +3 -5 400 360 391 394 396 397 398 401 331 326 320 Basic Secondary General Secondary Colleges Professional Schools n 2009 n 2012 Source: PISA 2009 and 2012. We love school:) An Analysis of PISA 2009 and 2012 25 rigorous admission policies; and it can also reflect the schools (TVE lyceums) since these students are one education system’s selection mechanism available step closer to joining the labor market when they to students and their parents for transitioning into complete their studies. Not only did the reading selective upper secondary education. performance of students in professional schools decline by 5 points, but their very low score of Disparities in performance by school type are 326 points in 2012 is a significant concern.12 Only small, but professional schools fall significantly 3 percent of students sampled in 2012 were in behind secondary schools and colleges. In professional schools, so fortunately the majority of math, each school type improved by at least 25 points students were in other streams, outperforming their since PISA 2009. Science performance also improved peers in professional schools by a large margin in all across all school types. However, reading gains were subjects. Secondary school students outperformed much smaller (Figure 15), suggesting that reading their TVE professional school peers by the performance remains a challenge for all students, equivalent of at least 1.5 years of schooling. which is especially important for those in professional 26 Strengthening Kazakhstan’s Education System: 3. Drilling down Further into math & reading skills An Analysis of PISA 2009 and 2012 27 PISA offers the opportunity to fully explore one subject area every three years, even though all three subjects are assessed every time PISA is administered. PISA seeks to assess not merely whether students can reproduce knowledge, but also how well they can extrapolate from what they have learned and apply it in unfamiliar settings, both within and outside the school. The detailed test of “subscale” skills of a given subject area is an in-depth assessment with a larger set of questions. The detailed assessment was focused on reading in 2000 and 2009, on math in 2003 and 2012, and on science in 2006. The 2015 round will focus again on science. Math skills in PISA 2012 The PISA math 2012 subscale assessment measured individuals’ abilities to formulate, employ, and interpret mathematics in a variety of contexts and content areas. In PISA, the concept of mathematical literacy includes: (i) mathematical reasoning; (ii) usage of mathematical concepts, procedures, and facts; (iii) tools to describe, explain, and predict phenomena; and (iv) the role that mathematics plays in the world and the need to make well-founded judgments and decisions needed by constructive, engaged, and reflective citizens. Furthermore, mathematic literacy as defined by PISA is not an attribute that an individual has or does not have; rather, it can be acquired to a greater or lesser extent, and it is required in varying degrees in society. The questions faced by students are framed in four real-world context categories: Personal, Societal, Occupational, and Scientific. The PISA math framework is a sophisticated tool for connecting students’ mastery of mathematical processes and contents. The math subscale assessment evaluates capacity in four content categories (Figure 16): quantity (incorporates the quantification of attributes of objects, relationships, situations, and entities); uncertainty and data (understanding messages embedded in data, and appreciating the variability inherent in many real processes); change and relationships (temporary and permanent relations among objects and circumstances); and space and shape (phenomena encountered in patterns, object properties, positions, representations, visual information, navigation, and dynamic interactions). Figure 16 also shows a schematic of the stages faced by a student when solving a real-life problem through the mathematical modelling cycle. The action begins with identifying the problem in context and finishes when the results of the problem are found in a context and again are reflected in the problem context. This process involves four skills that PISA defines as “processes,” assessed in 2012 as: formulate a mathematical situation according to the concepts and relationships identified; employ mathematical facts, procedures, and reasoning to obtain results (usually involving calculation, manipulation, and computation); interpret the results in terms of the original problem to obtain the “results in context”; and finally, evaluate the outcomes and their reasonableness in the context of the problem.13 28 Strengthening Kazakhstan’s Education System: Figure 16 Math contents and process categories in PISA ç problem in context ç formulate mathematical problem ç employ quantity uncertainty and data change and relationships evaluate space and shape results in context Source: OECD 2014. ç interpret mathematical results Kazakhstan’s students performed better in math problems related to space and shape, and not as well in employing math analytical skills. An Analysis of PISA 2009 and 2012 29 Figure 17 Kazakhstan’s PISA 2012 performance on different math subscales compared to the average math performance 20 n OECD n Kazakhstan Performance difference between each content/process 15 subscale and the average mathematics scale 10 Change and Uncertainty Employing Interpreting/ relationships Quantity and Data Evaluating 5 0 -5 Space and Formulating shape -10 -15 Contents Processes -20 Source: PISA 2012. Kazakhstan’s students performed better in literacy as understanding, using, and reflecting on problems related to space and shape, and and engaging with written texts to achieve one’s not as well in employing math analytical skills goals, to develop one’s knowledge and potential, and (Figure 17). Compared with the average math to participate in society. Understanding refers to the performance,14 Kazakhstan’s results showed greater reader’s ability to construct meaning from text; using variation across subscale assessments than that refers to the kind of reading that is directed toward found in OECD countries. Students successfully applying information in a text to an immediate task; solved problems related to space and shape, usually and reflecting on means that readers can relate what related to geometry, algebra, and physics. However, they are reading to their thoughts and experiences. students underperformed when they needed to use their ability to solve data problems or to appreciate The PISA reading framework is built on three variability and uncertainty in real-life problems. major characteristics: texts, aspects, and Moreover, there was room for improvement in the situations. Although texts are differentiated in process of interpreting and evaluating mathematical different characteristics (medium, environment, problems, a key competency crucial for managing type and format), performance on text format is real-life situations that require mathematical skills. the only one reported in PISA, using two types: continuous texts (sentences organized into Reading skills in PISA 2009 paragraphs, which may fit into even larger structures) and non-continuous texts (smaller sentences, usually The PISA 2009 subscale assessment of readings in sample lists, graphs, diagrams, or catalogues), skills measured students’ ability to actively, although there are also mixed and multiple texts. purposefully, and functionally apply reading Aspects are measured as PISA reading subscales in a range of situations. PISA defines reading with three categories: access and retrieve (skills 30 Strengthening Kazakhstan’s Education System: Figure 18 Kazakhstan’s PISA 2012 performance on different reading subscales compared to the average reading performance 10 Performance difference between each content/process 5 Non- Reflect and Continuous Evaluate Texts 0 subscale and the average reading scale -5 Continuous Access and Integrate and Texts retrieve Interpret -10 Texts Aspects -15 n OECD Average n Kazakhstan -20 Source: PISA 2009. associated with finding, selecting and collecting experiences to the text, reflecting a disconnect information); integrate and interpret (which involves between what students learn and their ability to understanding the relationships between different apply their knowledge in real-life situations. parts of a text, or making meaning from something not stated in the text); and reflect and evaluate Teacher practices and student (which involves drawing on knowledge, ideas, or learning strategies values external to the text). Finally, situations intend to maximize the diversity of content included in PISA 2009’s focus on reading assessment allows the PISA reading survey; for example, personal, for a detailed analysis of teacher practices public, educational, and occupational situations are and student learning strategies, as several represented. questions in the student questionnaire relate to these. Teaching practices include discipline, order, Performance in reading reveals that Kazakh time management, management of assignments students have a better understanding of and the extent to which teachers relate knowledge. continuous texts than of non-continuous texts, Student learning strategies include methods that and there is a need to improve their reflection students employ when studying (memorization, and evaluation skills (Figure 18). As with math, control, and elaboration, among others).16 It is the comparison of the reading subscale results with important to note that while learning strategies are the average reading performance15 reveals that employed by students, both teachers and parents Kazakhstan showed much more variation across may have a strong influence over studying methods. subscales than that found in OECD countries. In For this reason, teachers can also be seen as partially particular, students performed better with more responsible for student learning strategies. The traditional texts, and not as well relating their own analysis suggests that: (i) effective classroom An Analysis of PISA 2009 and 2012 31 Figure 19 Kazakhstan reading gap with comparator countries as a function of teacher practices and learning strategies 120 110 100 90 80 70 n Socio-economic (individual) n Socio-economic (peers) 60 n Prechool n School resources 50 n Teaching practices n Learning strategies n Unexplained 40 l Actual gap 30 20 10 0 -10 OECD ECA serbia slovenia bulgaria russia Source: PISA 2009. Note: The gap was computed after an Oaxaca decomposition including the variables above and indexes of learning strategies and teaching practices. Unexplained is the combination of returns to endowments (systemic efficiency) and the other factors (which can include institutional arrangement, capacity of policy makers, quality of curriculum, teachers’ knowledge of the contents and pedagogics, and other social, local, and cultural aspects). management and a teacher’s ability to engage teaching practices and learning strategies. The results show that teaching practices are overall on par students are critical; and (ii) elaboration strategies have a positive impact on reading performance, with comparator countries and the OECD and ECA control strategies have a mixed impact (depending averages. However, Kazakhstan has a significant on the country), and memorization strategies have adisadvantage in learning strategies relative to negative impact.17 the same comparator countries, especially the overreliance on memorization strategies (Figure 19). Compared with other countries, Kazakhstan’s Overall, learning strategies explain 8 points of the teaching practices are relatively strong while difference in scores between Kazakh and Russian student learning strategies present significant students, 9 points relative to the average score of ECA disadvantages. An analysis of the drivers of the students, and 19 points relative to the average score of performance gap with other countries was conducted OECD students.18 to understand the importance of differences in 32 Strengthening Kazakhstan’s Education System: 4. Findings recommendations & The following policy recommendations could be explored to build on the impressive overall gains made in PISA 2012: (i)  nsure that basic reading skills are achieved. Reading performance appears to be the greatest E challenge in Kazakhstan’s education system. Reading is a skill that begins with a solid foundation in early grade literacy. Rather than waiting for national or international large-scale assessments mostly focused on grades 4 and above, teachers should regularly assess early grade literacy in the classroom and then intervene with the appropriate combination of pedagogical approaches. Early grade literacy is a critical stepping stone for further skills acquisition, and a comprehensive approach that brings together teachers’ professional development, learning materials, and assessment tools development is necessary. Kazakhstan should consider implementing a program that includes five important steps to improve reading in the early grades: (i) train teachers how to teach reading; (ii) maximize instruction time in the classroom; (iii) put appropriate books in the hands of children; (iv) implement appropriate language policies and provide mother tongue-based instruction; and (v) measure reading skills. International evidence shows that the quality of teaching is the most important school-based predictor of student learning; thus ensuring teacher effectiveness should be prioritized. Teaching quality is affected by many factors, from teachers’ skills to their practices. In Kazakhstan, teachers appear to be especially challenged in how they manage assignments and relate knowledge taught to students’ lives. In turn, students are disadvantaged by limited knowledge and application of effective learning strategies, such as relating new knowledge to other contexts and applying study methods other than memorization. This study’s results also found large discrepancies in performance between Russian and Kazakh language schools, as well as among regions. While relatively low student performance points to the need to enhance teacher effectiveness at the national level, improving the pedagogical An Analysis of PISA 2009 and 2012 33 skills of teachers in Kazakh language schools and underperforming regions should be a priority. Incentive packages to ensure that the highest performing teachers are assigned to underperforming schools could also be considered. (ii)  Establish a comprehensive Early Childhood Development (ECD) program, with an emphasis on at-risk groups. The analysis of the determinants of quality of education concludes that unlike in most countries, attending preschool in Kazakhstan has no impact on student achievement levels at age 15, controlling for other relevant factors. These results may imply that the quality of the preschool system was poor in the late 1990s and early 2000s. In recent years, Kazakhstan has allocated considerable resources through the Balapan program to expand access to preschools, but the focus has not been on improving quality or ensuring that young children acquire key cognitive and non-cognitive skills that serve as the building block for lifelong skills formation. Kazakhstan therefore should assess the quality of preschools, either by observing teachers and their effectiveness, or by monitoring child development outcomes to ensure that additional investments have the intended impact. Recent evidence also suggests that adequate health, nutrition, and stimulation for children aged 0-3 are critical for brain development, which also affects skills acquisition. Currently, social services for children in the age groups 0-3 and 4-6 are not well coordinated. An effective ECD program should therefore integrate health and social services for children aged 0-3, while ensuring a smooth transition to quality preschool programs and then primary school readiness. The return on investments in ECD is highest for at-risk children and their families, thus in a context of limited resources, a targeted approach is recommended. Develop an integrated assessment system. An integrated assessment system should ensure that (iii)  student assessments reflect the content of the curriculum and are objective and fair, and that results are provided in a way that supports the system to identify areas where further research or remedial action might be needed. While the PISA results present a good starting point to discuss education reform, there are limits to the conclusions that can be drawn. For example, PISA is a competency- based, rather than a curriculum-based, assessment. There are thus limitations to the data and some information gaps are evident in the determinants analysis, especially for understanding the factors driving the improvements in math and science scores. Summative national student assessments, conducted at key milestone grades, could be reformed to collect data on the missing variables. Interim and formative assessments are also necessary to assess student progress with respect to the curriculum or to drill down deeper on mastery of a specific skill. Education policy makers may wish to consider not only results from international student assessments such as PISA but also national assessments at various levels. The results from this report will hopefully contribute to the ongoing debate about student assessment system reform in Kazakhstan. 34 Strengthening Kazakhstan’s Education System: References Amermueller, A. 2004. “PISA: What Makes the Difference? Explaining the Gap in Pisa Test Scores between Finland and Germany.” ZEW Center for European Economic Research Discussion Paper No. 04-004. Barrera-Osorio, F., V. Garcia-Moreno, H.A. Patrinos, and E. Porta. 2011. “Using the Oaxaca-Blinder Decomposition Technique to Analyze Learning Outcomes Changes Over Time: An Application to Indonesia’s Results in PISA Mathematics.” World Bank Working Paper 5584. World Bank, Washington, DC. Engle, P., L. Fernald, H. Alderman, J. Behrman, C. O’Gara, A. Yousafzai, M. Cabral de Mello, M. Hidrobo, N. Ulkuer, I. Ertem, S. Iltus, and Global Child Development Steering Group. 2011. “Strategies for reducing inequalities and improving developmental outcomes for young children in low-income and middle-income countries.” The Lancet- 8 October (Vol. 378, Issue 9799: 1339-1353 ). DOI: 10.1016/S0140-6736(11)60889-1. Ferreira, H.G., and J. Gignoux. 2011. “The Measurement of Educational Inequality: Achievement and Opportunity.” IZA Discussion Paper No. 6161. Firpo, S., N. Fortin, and T. Lemieux. 2009. “Unconditional Quantile Regressions.” Econometrica Vol. 7, No 3:. 953-973. Greenwald, R., L. V. Hedges, and R. Laine. 1996. “The Effect of School Resources on Student Achievement.” Review of Educational Research Vol. 66, No. 3 (Autumn): 361-396. Hanushek, E. 2009. “School policy: Implications of recent research for human capital investments in South Asia and other developing countries.” Education Economics 17(3), 291–313. Hanushek, E. 2010. “The High Cost of Low Educational Performance. The long-run economic impact of improving PISA outcomes.” OECD Publications. OECD: Paris. An Analysis of PISA 2009 and 2012 35 Hanushek, E., and L. Woessmann. 2007. “The Role of Education Quality in Economic Growth.” World Bank Policy Research Working Paper 4122. World Bank, Washington, DC. Heckman, J. 2008. “Schools, skills, and synapses.” Economic Inquiry, 46(3): 289-324.Heckman, J., and P. LaFontaine. 2010. “The American High School Graduation Rate: Trends and Levels.” Review of Economics and Statistics 92(2): 244–262.OECD. 2012. ”PISA 2009 Technical Report.” OECD: Paris. OECD. 2014. ”PISA 2012 Results: What Students Know and Can Do.” OECD: Paris. Sala-i-Martin, X., G. Doppelhofer, and R.I. Miller. 2004. “Determinants of long-term growth: A Bayesian averaging of classical estimates (BACE) approach.” American Economic Review 94 (4): 813-835. World Bank. 2013. “Promoting Excellence in Turkey’s Schools.” World Bank: Washington, DC. 36 Strengthening Kazakhstan’s Education System: Annex: RIF-Regressions: Empirical Strategy and Results The analytical approach used in Section 2 is based on the Firpo, Fortin, and Lemieux (2009) methodology. Usually, the literature of decomposition of student scores in PISA through groups (Amermueller 2004) and years (Barrera et al. 2011) has focused at the mean differences, with little attention to what happens at the tails of the distribution. The Firpo, Fortin, and Lemieux (FFL) allows student gap performance to be decomposed not only for the mean but also for other statistics of the distribution. Traditionally, the problem with quantile regressions has been that the law of iterated expectations does not apply, thus making it impossible to interpret the unconditional marginal effect of each independent variable on the student. However, recent econometric techniques such as the one proposed by FFL have solved this methodological difficulty. The FFL technique is based on the construction of re-centered influence functions (RIF) of a quantile of interest, , as a dependent variable in a regression: ⌧ − D(I  q⌧ )) RIF (I ; q⌧ ) = q⌧ + f I ( q⌧ ) where is an indicator function and is the density of the marginal distribution of scores. A crucial characteristic of this technique is that it provides a simple way of interpreting the marginal impact of an additional unit of a certain factor on a student’s PISA scores. Once the unconditional quantile regression has been computed for different quantiles of the distribution, the results are decomposed following the Oaxaca- Blinder approach. An Analysis of PISA 2009 and 2012 37 Table A 1. OLS regression of determinants of math performance Variables Math Girl -3.823** (1.686) Grade (school average) 22.66*** (5.116) Age -1.862 (3.629) ESCS 8.295*** (0.997) Language at home (Kazakh) -5.071 (3.902) ECD (1 year) -5.550 (3.628) ECD (2+ Years) 1.040 (3.250) Percentage of gilrs at school 0.130 (0.287) ESCS (School average) 29.98*** (7.359) Grade (school average) -11.11 (13.31) Percentage of Minorities at school 0.136 (0.0996) School Medium of Instruction (Kazakh) -20.85*** (5.763) Percentage of Migrants at school -0.146 (0.154) Rural 6.673 (6.777) General Secondary (baseline is basic secondary) -5.140 (6.391) Professional Lyceum (baseline is basic Secondary) -4.141 (18.20) Vocational College (baseline is Basic secondary) -17.74 (15.84) Shortage of teacher in math (Index)) 4.842* (2.890) Student Teacher ratio -1.501 (0.975) Quality of Educational Resources (Index) 1.819 (1.405) Proportion of Certified Teachers 1.755 (11.31) Autonomy of Curriculum and Assessment (index) -2.044 (4.439) Autonomy of Resources (Index) 4.103 (4.959) Constant 362.8** (159.9) Observations 5,310 R-squared 0.206 Robust standard errors in parentheses and clustered at the school level. *** p<0.01, **p<0.05,*p<0.1. 38 Strengthening Kazakhstan’s Education System: Table A 2. Decomposition of 2009-2012 PISA math score gaps by student achievement groups Variables Average Percentile 20 Percentile 50 Percentile 80 Overall Endowments Overall Endowments Overall Endowments Overall Endowments Year 2012 434.1*** 377.1*** 429.5*** 489.7*** (3.129) (2.605) (2.922) (3.272) Year 2009 407.0*** 339.5*** 399.6*** 471.8*** (4.287) (2.461) (2.783) (4.560) Difference 27.14*** 37.55*** 29.81*** 17.92*** (5.307) (3.584) (4.035) (5.613) Endowments 16.41* 8.789** 8.567 15.45 (8.977) (4.285) (6.530) (9.795) Coefficients 16.54*** 29.50*** 21.09*** 7.903* (4.435) (3.564) (3.517) (4.768) Unexplained -5.814 -0.737 0.156 -5.433 (8.449) (4.393) (6.402) (9.277) Individual 3.465*** 2.331*** 2.083*** 2.276* Characteristics (1.022) (0.585) (0.639) (1.208) Peer 7.631** 2.216 3.899** 9.402*** Characteristics (2.976) (1.379) (1.689) (3.316) School 5.317 4.242 2.586 3.772 Resources (8.097) (3.773) (6.024) (8.618) Constant -104.3 -97.00* -112.3* -72.15 (76.15) (49.60) (59.62) (80.54) Observations 10,440 10,440 10,440 10,440 10,440 10,440 10,440 10,440 Note: Robust standard errors in parentheses and clustered at the school level. *** p<0.01, **p<0.05,*p<0.1. Oaxaca decompositi on was conducted including a threefold decomposition, with endowments, coefficients, and an unexplained part. Variable effects are grouped and include individual characteristics (age, gender, grade, language at home, and SES), peer characteristics (SES), and school resources (school location, language of instruction (Kazakh), quality of educational resources, teacher shortage, and proportion of certified teachers). An Analysis of PISA 2009 and 2012 39 Table A 3. Decomposition of 2009-2012 PISA reading score gaps by student achievement groups Variables Average Percentile 20 Percentile 50 Percentile 80 Overall Endowments Overall Endowments Overall Endowments Overall Endowments Year 2012 396.6*** 348.4*** 397.9*** 452.5*** (3.353) (9.882) (3.297) (1.657) Year 2009 393.2*** 319.2*** 388.7*** 470.2*** (4.494) (4.514) (4.613) (4.431) Difference 3.449 29.23*** 9.218 -17.69*** (5.607) (10.86) (5.670) (4.731) Endowments 19.03** 17.48** 16.31* 12.91 (8.295) (8.496) (9.038) (10.01) Coefficients -9.542** -7.471 -3.634 23.04*** (3.714) (10.63) (3.983) (3.707) Unexplained -6.035 19.22* -3.458 -7.552 (6.883) (11.27) (7.890) (9.284) Individual 3.772*** 4.650*** 4.088*** 2.385* Characteristics (1.372) (1.530) (1.286) (1.435) Peer 8.703*** 5.648** 7.377*** 8.968*** Characteristics (2.824) (2.615) (2.671) (2.950) School 6.551 7.184 4.843 1.553 Resources (6.830) (7.133) (7.905) (8.740) Constant -87.59 -183.8 -68.27 -28.14 (64.90) (119.8) (74.94) (74.76) Observations 10,404 10,404 10,404 10,404 10,404 10,404 10,404 10,404 Note: Robust standard errors in parentheses and clustered at the school level. *** p<0.01, **p<0.05,*p<0.1. Oaxaca decompo sition was conducted including a threefold decomposition, with endowments, coefficients, and an unexplained part. Variable effects are grouped and include individual characteristics (age, gender, grade, language at home, and SES), peer characteristics (SES), and school resources (school location, language of instruction (Kazakh), quality of educational resources, teacher shortage, and proportion of certified teachers). 40 Strengthening Kazakhstan’s Education System: Table A 4. Indices of learning strategies and teaching practices Learning Strategies Control How students set clear goals for themselves and monitor their own progress in reaching them Memorization To what extent students try to memorize texts Elaboration How students relate acquired knowledge to other contexts (own life, outside school, and prior knowledge) Metacognition: Compares students’ strategies for understanding and understanding and remembering with what remembering experts rate as the most appropriate strategies Metacognition: Compares students’ strategies for summarizing summarizing with what experts rate as the most appropriate strategies Teaching Practices Discipline, order, What is the disciplinary climate in the and time classroom (noise, and time taken for students management to quiet down)? Discussion & debate Extent to which teachers engage students in discussion Relating knowledge Whether teachers help students relate knowledge to different contexts (prior knowledge, and personal experiences) Clarifying expectations Whether teachers outline how student- teacher interaction will be from the beginning Managing Whether teachers mark assignments, check if assignments students understood the lesson, and motivate students Quality of Educational Shortage or Science laboratory equipment, instructional Resources inadequacy of the materials (including textbooks), computers for following factors (as instruction, internet connectivity, computer reported by school software for instruction, library materials, and principals) audio-visual resources Note: Indices were constructed by World Bank staff based on PISA 2009. See OECD 2014, “PISA 2009 Results: Learning to Learn – Student Engagement, Strategies and Practices (Volume 3)” for more details on the indices. An Analysis of PISA 2009 and 2012 41 Table A.5. Decomposition of PISA 2009 reading score gaps between countries Variables OECD ECA Serbia Greece Slovenia Bulgaria Russia Country Score 500.8*** 468.4*** 453.0*** 491.5*** 493.7*** 453.2*** 467.6*** (1.208) (1.548) (3.405) (3.279) (4.854) (5.807) (4.285) Albania Score 395.8*** 395.8*** 395.8*** 395.8*** 395.8*** 395.8*** 395.8*** (4.404) (4.632) (5.371) (5.029) (6.219) (5.118) (3.190) Difference 105.1*** 72.61*** 57.25*** 95.77*** 97.96*** 57.45*** 71.81*** (4.599) (4.808) (6.265) (6.079) (7.853) (8.017) (5.217) Explained 52.15*** 18.50*** 52.49*** 57.60*** 59.90*** 37.99*** 30.39*** (5.987) (3.624) (7.228) (6.927) (11.52) (7.673) (4.099) Entrance Age 2.009 -0.611 -1.079 1.084 -0.168 -1.050 -0.424*** (1.793) (0.602) (1.339) (1.218) (0.268) (1.212) (0.141) Age -0.222* -0.0255 0.449 -0.792* -0.680 0.167 0.135 (0.126) (0.0772) (0.303) (0.470) (0.545) (0.132) (0.0931) Grade 2.051* 0.987 1.373 2.136 0.999 1.435 -0.675* (1.046) (0.600) (0.976) (1.363) (0.883) (0.951) (0.365) Girl -0.0244 0.159 0.335 0.527 0.376 -0.105 0.324 (0.288) (0.313) (0.598) (0.432) (0.643) (0.680) (0.372) ESCS (Individual) 5.009*** 1.177** 8.131*** 6.673*** 8.101*** 6.504*** 4.191*** (0.795) (0.553) (1.527) (1.254) (1.804) (1.333) (0.666) ESCS (Peers) 18.76*** 4.303** 30.69*** 25.47*** 30.91*** 24.09*** 15.48*** (3.017) (2.067) (5.889) (4.816) (7.022) (5.013) (2.491) Teaching Practices 4.479*** 1.485*** 2.372*** 3.540*** 3.908*** 0.688* 0.969*** (0.748) (0.305) (0.599) (0.768) (1.142) (0.365) (0.198) Learning Strategies 19.69*** 9.810*** 10.71*** 18.28*** 17.44*** 4.890*** 7.893*** (1.435) (1.206) (1.765) (1.712) (2.382) (1.805) (1.373) Quality of Resources -2.620 -0.570 -1.180 -2.085 -3.918 -2.028 -0.411 (4.045) (0.997) (2.545) (4.051) (10.29) (4.049) (0.367) Quality of -0.00737 -0.00582 -0.0161 -0.0142 -0.0151 -0.0199 -0.00721 Resources^2 (0.861) (0.745) (2.609) (2.085) (2.999) (2.997) (0.244) Two or more years of 3.020** 1.792* 0.698 2.785 2.946 3.425 2.908*** Pre-Primary (1.487) (0.972) (0.494) (1.724) (2.469) (2.177) (0.439) Systemic Efficiency 52.94*** 54.11*** 4.767 38.17*** 38.06*** 19.46*** 41.42*** (6.900) (4.224) (7.505) (7.978) (13.07) (7.247) (3.570) Constant 281.2*** 151.5* 238.2** 174.6* 222.4* -39.76 106.6 (75.07) (82.96) (100.8) (92.93) (120.6) (92.69) (120.2) Observations 247,936 85,850 9,426 9,184 9,831 8,313 9,651 Note: Robust standard errors in parentheses. Twofold Oaxaca decomposition. Systemic efficiency accounts for the aggregation of the returns to endowments and unexplained part of the Oaxaca decomposition. Villages are settlements with fewer than 3,000 people; small towns have between 3,000 and 15,000 people; towns have between 15,000 and 100,000 people; and cities have between 100,000 and 1,000,000 people. 42 Strengthening Kazakhstan’s Education System: Table A.6. Oaxaca-Blinder decomposition of gender reading gap, PISA 2009 Variables Overall Endowments Coefficients Girls 431.1*** (4.549) Boys 379.0*** (5.758) Difference 52.05*** (4.586) Endowments 10.35*** (3.329) Coefficients 42.57*** (3.519) Unexplained -0.859 (1.816) Entrance 1.545* 50.28 (0.801) (46.62) Age 0.0209 -379.0* (0.0896) (228.5) Grade -0.373 6.549* (0.777) (3.638) ESCS -0.599 -5.106 (0.582) (3.133) ESCS (School) -0.191 6.491 (1.250) (7.223) Teaching Practices 1.462** 1.958 (0.674) (1.931) Learning Strategies 8.551*** 0.969 (1.528) (0.664) Student-Teacher Ratio 0.0311 23.63** (0.213) (11.27) Quality of Educational Resources (Index) -0.844 2.565 (0.830) (3.650) Attended at least two years of pre-primary school 0.746 -0.852 (0.601) (2.755) Constant 335.0 (216.7) Observations 2,903 2,903 2,903 Note: Robust standard errors in parentheses. Threefold Oaxaca decomposition. An Analysis of PISA 2009 and 2012 43 Endnotes 1 Programme for International life situations, including 9 See Table A.1 in the Annex. Student Assessment. “understanding, using, reflecting 10 A similar analysis conducted 2 Differences in school types on and engaging with written with multilevel models yielded and language spoken at texts, in order to achieve similar results. home are largely explained one’s goals, to develop one’s 11 See World Bank (2013). by socioeconomic and other knowledge and potential, and 13 It is also true that these factors. However, for language to participate in society.” For differences are not significant of instruction, Kazakh language mathematics literacy, it is the after controlling for schools consistently score extent to which students can socioeconomic and other factors around 20 points below Russian reproduce mathematical content (see the Annex). language schools even after knowledge, but also how well they 14 The last two processes are controlling for other factors (see can extrapolate from what they summarized into a single skill in the Annex). know and apply their knowledge the PISA student database. 3 See Sala I Martin, Doppelhofer of mathematics in both new and 15 “Average math performance” and Miller (2004). unfamiliar situations. In both refers to the average score 4 See Hanushek and Woessmann subjects, those who are below across all math subscale scores. (2007) and Hanushek (2010). level 2 can only answer questions 16 “Average reading performance” Using these tests as measures of involving familiar contexts refers to the average score across cognitive skills of the population, where all relevant information all reading subscale scores. they show that countries that is available and the questions 17 For details on the variables had better quality of education are very clearly defined. Such included and methodology, in the 1960s experienced faster students can only solve problems please refer to Table A.4 in the economic growth during the that are almost always obvious Annex. years 1960-2000, controlling for and can be immediately 18 Two indices were constructed other factors. extrapolated from the given to summarize these effects 5 PISA 2009 Technical Report information. for each student, using the (OECD 2012). 8 Ferreira and Gignoux (2011) coefficients resulting from 6 Note: Countries that participated propose a measure of educational a pooled (with all PISA only once in PISA between 2000 opportunity using the share data) linear regression of and 2012 were not considered of variance in test scores that reading scores and each set of for the ECA average, in order to is explained by individual components for each index. strengthen the relevance. predetermined circumstances. 19 Detailed results can be seen in 7 PISA defines functional If a significant share of the Table A.5 in the Annex. literacy in reading as the results is explained by these ability of students to use characteristics, then the equality written information in real- of opportunities is low. Education Global Practice Europe and Central Asia Region