n ANALYSIS OF LEARNING IN ARMENIA Table of Contents List of figures, tables, boxes, and maps............................................................................................................... 2 Acknowledgments ................................................................................................................................................ 3 Abbreviations ....................................................................................................................................................... 4 1. Introduction ...................................................................................................................................................... 5 2. Background ....................................................................................................................................................... 7 2.1 Structure of General Education .................................................................................................................. 7 2.2 Education Expenditure ................................................................................................................................ 8 3. Data and Methodology ................................................................................................................................... 11 4. Quality of Learning in Armenia ....................................................................................................................... 12 4.1 Student Performance across population groups ...................................................................................... 14 4.2. Student Performance across Regions ...................................................................................................... 16 5. Factors associated with learning outcomes in Armenia ................................................................................. 19 5.1 Variance explained by students, schools and regions .............................................................................. 19 5.2 School-Level Factors and Student Performance ....................................................................................... 20 6. Policy Recommendations................................................................................................................................ 25 ANNEX ................................................................................................................................................................. 29 1 List of figures, tables, boxes, and maps FIGURES FIGURE 1. GROSS ENROLLMENT RATE, 2017 OR LATEST AVAILABLE DATA ............................................................................. 8 FIGURE 2. INITIAL GOVERNMENT FUNDING PER STUDENT AS A PERCENTAGE OF GDP PER CAPITA, 2017 OR LATEST AVAILABLE DATA .......................................................................................................................................................... 8 FIGURE 3. STUDENT-TEACHER RATIO, 2018 OR LATEST AVAILABLE DATA .............................................................................. 9 FIGURE 4. LEARNING POVERTY IN ARMENIA AND COMPARATORS, 2016 OR LATEST AVAILABLE DATA ....................................... 13 FIGURE 5. TIMSS MATH AND SCIENCE AVERAGE SCORE FOR ARMENIA ............................................................................... 13 FIGURE 6. TIMSS MATH SCORES IN 4TH GRADE BY PERCENTILE, 2011, 2015 AND 2019 ..................................................... 14 FIGURE 7. TIMSS 4TH GRADE MATH SCORE DIFFERENCES BY GENDER, LOCATION, AND SOCIOECONOMIC LEVEL, 2019 .............. 16 FIGURE 8. VARIATION IN GRADE 4 MATH SCORES ATTRIBUTABLE TO SCHOOLS IN ARMENIA AND COMPARATOR COUNTRIES, TIMSS 2011, 2015 AND 2019 .............................................................................................................................. 19 FIGURES 9. PERCENTAGE OF VARIANCE IN MATH EXPLAINED BY SCHOOLS AND PERCENTAGE OF LOW LEARNERS, 2019 ................. 20 FIGURE 10. DISTRIBUTION OF AVERAGE TEACH SCORES BY AREA AND OVERALL ...................................................................... 22 FIGURE 11. DISTRIBUTION OF AVERAGE TEACH SCORES BY ELEMENT..................................................................................... 22 FIGURE 12. CONDITION OF THE SCHOOL BUILDING BY SCHOOL LOCATION, 2019 .................................................................... 23 FIGURE 13. PERCENTAGE OF LOW LEARNERS AND PERCENTAGE OF SCHOOLS WITH SATISFACTORY CONDITION, 2019 .................... 24 TABLES TABLE 1. PERCENTAGE OF LOW LEARNERS IN 9TH GRADE BY SUBJECT AREAS IN YEREVAN, 2019 ............................................ 18 TABLE 2. PERCENTAGE OF LOW LEARNERS BY SUBJECT, 2019 .......................................................................................... 18 BOXES BOX 1. MAIN ROLES OF CENTRAL AND REGIONAL BODIES IN THE FIELD OF GENERAL EDUCATION........................................ 10 BOX 2. COVID-19 PANDEMIC’S IMPACT ON LEARNING .............................................................................................. 15 BOX 3. TEACH ARMENIA FINDINGS…………………………………………………………………………………………………………………….22 MAPS MAP 1. PERCENTAGE OF LOW LEARNERS IN 9TH GRADE, AVERAGE ACROSS ALL SUBJECT AREAS, 2019 ............................16-17 MAP 2. DISTRICT VARIATIONS ACROSS YEREVAN IN PERCENTAGE OF LOW LEARNERS, GRADE 9, MATHEMATICS, 2019 ............. 17 2 Acknowledgments This study was conducted by Isil Oral Savonitto in close collaboration with Martin Moreno and Karina Acevedo Gonzalez. The team has greatly benefited from the comments provided by our peer reviewers Saamira Halabi, Alina Sava, and Monica Yanez Pagans. The team is equally grateful for the insightful suggestions and guidance shared by Harry Anthony Patrinos, Carolin Geginat, Katia Marina Herrera Sosa, Renata Freitas Lemos, Shizuka Kunimoto, and Anush Shahverdyan. In addition, the Deputy Minister of Education, Science, Culture and Sports of Armenia Ms. Zhanna Andreasyan and her team, Ms. Victoria Aydinyan from the Ministry of Labor and Social Affairs, Mr. Artak Poghosyan from the National Center for Education Technologies, Mr. Artashes Torosyan from the National Center for Education Development and Innovation, and other education development partners from UNICEF Armenia, Teach for Armenia, Textbook Revolving Fund, Brusov State University, British Council, NEO Armenia, Asian Development Bank, Step-by-Step Foundation, and AYB Foundation have provided comments and suggestions which were all taken into account while finalizing the report. Aya Alphs edited the report. 3 Abbreviations ATC Assessment and Testing Center CIS Commonwealth of Independent States ECA European and Central Asia EU The European Union GDP Gross Domestic Product GLAD Global Learning Assessment Database HCI Human Capital Index ICT Information and Communication Technologies IEA The International Association for the Evaluation of Educational Achievement LAYS Learning-adjusted Years of Schooling LT Long-term MoESCS The Ministry of Science, Education, Culture, and Sport NaCET The National Center for Education Technology NLSA National large-scale standardized assessment OECD The Organization for Economic Co-operation and Development PISA Programme for International Student Assessment READ Russia Education Aid for Development ST Short-term STEM Science, Technology, Engineering and Math STR Student Teacher Ratio TIMSS Trends in International Mathematics and Science Study UIS The UNESCO Institute for Statistics UNESCO The United Nations Educational, Scientific, and Cultural Organization 4 1. Introduction Following several years of high growth and poverty reduction, Armenia was hit hard by the crises of the last decade, and progress on social and economic development slowed. Between 2004 and 2008, poverty fell dramatically—the share of poor people (those with incomes under the national upper poverty line) dropped from 54 percent to 27 percent. However, the effect of the 2008 crisis on the Armenian population was deeper than in other countries in Europe and Central Asia (ECA). Armenia’s economic performance recovered during 2017–2019, but the global economic crisis of 2020 contracted Armenia’s economy by 7.6 percent.1 The education sector in Armenia has challenges with low learning levels and additional pressures imposed by the COVID-19 pandemic. The World Bank’s human capital index (HCI) shows that a child who starts school at age four in Armenia can expect to complete 11.3 years of schooling by the age of 18. An analysis of learning outcomes factoring in what children actually learn, however, shows that expected years of schooling equate to only eight years. This results in a learning gap of 3.3 years. The COVID-19 pandemic may have exerted additional hurdles to improve learning outcomes. Due to the pandemic, Armenia risks losing 0.3 learning- adjusted year of schooling as calculated by World Bank simulations. This translates to an average annual earning loss of US$6,457 per student. Additionally, around 26 percent of children at late primary-school age in Armenia are not proficient in reading.2 This, also known as learning poverty, means being unable to read and understand a short, age-appropriate text by age 10. Learning poverty in Armenia is 17.2 percentage points worse than the average for the ECA region (8.9 percent on average). Firms face problems in recruiting and retaining workers with the required skills, and they see the lack of workforce skills as a major obstacle to their activities. The extent to which education provides practical skills and updated knowledge has emerged as a problem. In addition to technical skills, young workers lack generic skills related to problem solving, critical and creative thinking, teamwork, languages, and leadership (Rutkowski, 2013). While employers report skill constraints, a large share of the labor force is unemployed or inactive. Alleviating the skill constraints of Armenia’s firms is crucial to boost productivity and competitiveness (World Bank, 2018). Armenia does not rank high in the HCI, which is largely attributable to its challenges with access to and quality of education services and this has important implications on productivity levels in the country . HCI suggests that a child born in Armenia today will be only 58 percent as productive when (s)he grows up as (s)he could have been if she enjoyed complete education and full health.3 This is lower than the average for both the ECA region (69 percent) and upper middle-income countries of the Commonwealth of Independent States (CIS) (63 percent). Students from poor and rural families have lower future productivity, worse learning outcomes, and less expected years of schooling. Youth in Armenia do not attain their full development potential because of inadequate access to social services and poor-quality education and health systems.4 1 World Bank, World Development Indicators (2021). 2 The learning poverty number for Armenia is calculated using the Global Learning Assessment Database (GLAD) harmonization based on TIMSS. The Minimum Proficiency Level (MPL) threshold used was low level (400 points). For more details, please consult the GLAD and learning poverty repositories in GitHub. 3 Between 2018 and 2020, the HCI value for Armenia has remained almost unchanged, but learning outcomes have improved over time. 4 Human Development Enhancement Program, Sector Assesment (Summary): Education and Health (n.d), https://www.adb.org/sites/default/files/linked-documents/51129-002-ssa.pdf. 5 Quality education provision and skills development are priorities for the government. Quality of education is a key challenge causing a gap between the formal qualifications of graduates and the skills sought by employers. This mismatch is potentially hampering economic growth, leading to underutilized human capital for the country, and slowing labor market demand and overall productivity. Learning gaps are a significant issue with implications not only on the education systems of countries, but also for the well-being and productivity of individuals over a lifetime, especially in countries with an ageing population as Armenia. The population is ageing rapidly, with the number of elderly people (age 65 and above) increasing and the number of children and youth decreasing. 5 Ensuring that students learn necessary skills at school is the most effective way to help them build a fulfilling future and become competent participants of society that contribute to their country’s economy. An education system that can promote learning has the potential to improve outcomes around employment, productivity, earnings, health, and poverty reduction. Having highly skilled human capital also galvanizes innovation, strengthens institutions, and encourages social cohesion. The government sees the development of education and science as a priority through which to achieve sustainable and inclusive development. This direction is reflected in the strategic objectives of: (i) enhancing human capital through better access to quality services including healthcare, education, culture, and basic infrastructure and (ii) expanding employment through high-productivity and decently paid jobs. Armenia is also currently embarking on an ambitious curriculum reform to address the learning and education quality challenges in line with the government goals listed above. Through new reforms, Armenia is looking to transition into a competency-based education system with more inquiry-based, student-centered, and outcome-oriented teaching, learning, and assessment. This is accompanied by measures aimed at improving teachers' policies and the necessary infrastructure while strengthening the education-science-labor market link since there is a pressing need to improve skills and labor productivity in Armenia. The main motivation of the report is to analyze critical human capital dynamics that play into labor productivity, especially that of learning and its determinants. Armenia’s performance in international assessments have been relatively below average but slightly improving over the last decade. This report is also exploring the overall performance of Armenia in terms of learning, where any improvements have occurred and whether they are timely and sufficient in ensuring sustainable growth and productivity. The report will first look at the background of the education system in Armenia including education expenditures and explain the methodology of the study; then analyze the quality of education; focus on differences in student performance across regions followed by factors associated with overall learning outcomes. Finally, it will present recommended ways forward based on the analysis. 5 The overall population in Armenia declined from 3.07 million in 2000 to 2.88 million in 2011, mainly due to out-migration, and increased slightly to about 2.95 million in 2019. 6 2. Background This section provides an overview of features of the Armenian education system that influence the acquisition of foundational skills, such as the structure of general education, education expenditure and education governance. The section concludes with a brief summary of the methodology used for the analysis. 2.1 Structure of General Education Armenia’s education system has 12 years of compulsory education. This includes primary (grades 1–4), basic school (grades 5–9), and high school (grades 10–12), delivered in 1,400 schools. The country has a student population close to 563,000 in general and professional schools. About 97 percent of schools are public. As of 2017, Armenia had 808 community, 10 public, and 50 private preschool education facilities.6 Access to both pre-primary and higher education is low and inequitable. The net attendance rates in pre- primary education suggest that only 35 percent of 3- to 5-year-old children attended this level in 2018 (UNESCO UIS 2020). The gross enrollment rate for children aged 0–5 years was 30.9 percent—36.6 percent in urban communities and 20.6 percent in rural communities. The Integrated Living Conditions Survey 2017 noted that the main reasons for non-attendance was access and financial constraints. 7 Enrollment rates in tertiary education are similar to those of other ECA countries with the same level of development, but below to that of advanced economies. The gross enrollment rate for tertiary education is 56 percent, with more female students (62 percent) enrolled compared to male students (52 percent). Students from better-off households are more likely to attend tertiary level institutions (53 percent) compared to 29 percent for worse-off households and 0 percent for households living in extreme poverty.8 Enrollment rates suggest that in Armenia, access to primary and secondary education is high, but universal education is not achieved, especially at the high school level. As of 2019, gross enrollment was high at primary (91.3 percent) and secondary (90.1 percent) levels but dropped to 65.5 percent specifically at the high school level. Armenia’s enrollment rates are lower at each level of education compared to the European Union (EU) and Organisation for Economic Co-operation and Development (OECD) averages (figure 1). 6 Statistical Committee of the Republic of Armenia (Armstat) and World Bank, Social Snapshot and Poverty in Armenia: Statistical and Analytical Report (Yerevan, 2018). 7 Armstat. “Household’s Integrated Living Conditions Survey�, Anonymized Microdata Database (by Households) (2017). https://www.armstat.am/en/?nid=205. 8 Ibid. 7 Figure 1. Gross Enrollment Rate, 2017 or Latest Available Data 140 120 114 115 96 101 101 102 99 95 93 100 87 71 74 80 57 55 60 49 38 40 20 0 Pre-primary education Primary education Secondary education Tertiary education Armenia CIS EU27 OECD Source: UNESCO UIS 2020. 2.2 Education Expenditure Government expenditure on education is relatively low. In 2017, the government expenditure on education was only 2.7 percent as a percentage of GDP, which is significantly below the recommended expenditure of at least 4-6 percent of GDP.9 The per-student government expenditure on primary and tertiary education as a share of the GDP per capita is half of that of the benchmark group of countries (figure 2). In secondary education, the per-student expenditure represents 15 percent of the GDP per capita, still below comparators. Ensuring adequate levels of public expenditure on education provides the basis to expand educational opportunities, and in turn, positively impact long-term economic development. Figure 2. Initial Government Funding Per Student as A Percentage of GDP Per Capita, 2017 or Latest Available Data 35 29 30 28 28 25 25 23 21 22 20 20 20 20 17 17 18 % 15 15 10 10 10 5 0 Pre-primary education Primary education Secondary education Tertiary education Armenia CIS EU OECD Source: UNESCO UIS 2020. 9 According to The Third International Conference on Financing for Development (in Addis Ababa, July 2015). 8 Expenditure composition leaves little room for investing in core services to improve learning. Capital expenses are high and the expenditure on recurrent expenses, other than staff compensation, represents 15 percent of the total expenditure, as compared to 23 percent in OECD and EU countries, and 25 percent in CIS countries. It suggests that Armenia allocates proportionally less resources towards critical services to support learning, such as schoolbooks and teaching materials, ancillary services (food, transport, administration), and other support activities. The low student-to-teacher ratio (STR), especially in pre-primary and secondary education, and the comparable share of education expenditure devoted to staff compensation suggests low teacher salaries. In 2018, STRs were low in pre-primary and secondary education. It was only 6 students per teacher in pre-primary education, much lower than in CIS countries (10 students per teacher) and OECD/EU countries (15 students per teacher). In secondary education, the STR was only 8 students per teacher, while it was 13 in OECD/EU and 18 in CIS countries (figure 3). Meanwhile, the expenditure on staff compensation (teaching and non-teaching staff) accounts for about 73 percent of the total education expenditure, only slightly above that of regional peers (70 in both CIS and OECD) suggesting that teacher salaries are relatively low.10 Overall, the average teacher salary represents 63 percent of the GDP per-capita, while in OECD countries, the starting teacher salary represents around 90 percent.11 Figure 3. Student-Teacher Ratio, 2018 or Latest Available Data 20 18 18 15 15 16 14 13 14 14 12 12 12 12 12 12 10 10 8 9 8 6 7 6 4 2 0 Pre-primary education Primary education Secondary education Tertiary education Armenia CIS EU27 OECD Source: UNESCO UIS 2020. Education expenditure is also decentralized. Armenia is composed of ten marzes (regions) and Yerevan city. Marz governors implement the regional policy of the government. They coordinate the activities of local branches of the executive authority, except as otherwise specified by law. Financing for schools under marzes accounted for about 54 percent of the sector budget in 2018, which was then transferred to marzes for the distribution to schools. Regions oversee about 34.1 percent of the expenditure, while the national bodies oversee the remaining 11.9 percent. Roles of the Ministry of Education, Science, Culture and Sports (MoESCS) and the marzes in general education are described in box 2. Box 1. Main Roles of Central and Regional Bodies in the Field of General Education 10 UNESCO UIS. 11 The average teacher salary in Armenia was extracted from the Teacher Survey Questionnaire. 9 Ministry of Education, Science, Culture and Sport 1. To develop, approve and implement state programmes of general education 2. To give consent, as prescribed by law, for the establishment of other types of nonstate education institutions 3. To establish the time periods of and the procedure for holding state final examinations 4. To establish the procedure for enrolling school-age children in general education 5. To carry out the testing or attestation of mastering the educational programmes by learners, as well as the procedure for grade-completion and graduation of a learner 6. To ensure the development, expertise and publishing of the forms of syllabuses, textbooks, manuals and school registers 7. To develop and approve the model procedure for calling competition for the vacant position of a teacher of an educational institution 8. To define the requirements to the content of training of pedagogical workers and director; to offer training programmes for teachers 9. To exercise supervision over and state inspection of educational institutions and carry out assessment of the activities thereof 10. To recommend for allocation, on a competitive basis, of additional targeted financing to an educational institution from the State Budget Territorial Administration Body and of the Mayor of Yerevan 1. To ensure the implementation of state policy in the marzes and in Yerevan 2. To supervise schools’ use of the education legislation as well as the implementation of educational programs in line with the state standard for general education 3. To coordinate record keeping of school-age children; to ensure their enrollment in educational institutions 4. To ensure the construction, use, and maintenance of buildings is properly transferred to educational institutions under the law 5. To hire and manage the contract of the school principals as prescribed by the Government of the Republic of Armenia 6. To assist in the process of evaluating learners and graduates of educational institutions and to assist in the process of teacher attestation 7. To exercise other powers established by laws and other legal acts of the Republic of Armenia. General law on education also prescribes competences in the education sector for local self-government bodies. Source: Law of General Education, 2015. The governance style of education finances may be undermining resource efficiency. The structural arrangement of education finance in Armenia complicates the delegation of authority, responsibility, and accountability by the MoESCS towards schools, making it difficult to implement policies and monitor school performance. For example, only high schools are directly under the jurisdiction of the MoESCS; all other schools are under marzes, some of which report to the Ministry of Territorial Administration and Infrastructure.12 As a result of all these factors, Armenia would greatly benefit from an in-depth analysis of expenditure patterns to improve efficiency, equity, and effectiveness of public spending on education. 12 Human Development Enhancement Program, Sector Assessment (Summary): Education and Health. 10 3. Data and Methodology This study used data from IEA’s Trends in International Mathematics and Science Study (TIMSS) and the National Center for Education Technology (NaCET) - the agency of the MoESCS responsible for the management of educational administrative and performance data. TIMSS is an international large-scale achievement study in mathematics and science for grades 4 and 8 conducted since 1995 in 4-year intervals. The study conducts a two-stage stratified sample. In the first stage, schools are sampled by probability proportional to size. In the second state, grade 4 mathematics classes are selected in each school in an equal probability sample with all the students within each class selected. Sampling weights are used to account for difference in the sampling of subgroups and to adjust for nonresponse. In each country, in addition to the assessments conducted among students, additional surveys were conducted to collect information on their individual and household characteristics. Questionnaires specifically design for teachers were applied to gather information on their individual background and the work experience in the school. Similar instruments were applied to the principals to collect information on the school characteristics. Armenia has participated in TIMSS since 2003 up to 2019, the most recent round. In 2019, Armenian participation were limited to the assessments conducted in grade 4. The overall sample consisted of 5,399 students, 216 classes, 212 teachers and 150 schools. Some comparative analyses were conducted in this report using the samples of Bulgaria, Croatia, Georgia, Kazakhstan, Serbia and Russia. The sample sizes are reported in Table A1. NaCET provided several three sources of data. First is the data on students’ examinations which are used to assess learning and make decisions about student progress. These examinations are conducted by the Assessment and Testing Center (ATC) at the end of the academic year in grades 4, 9, and 12. NaCET provided examination score data by subject, aggregated at school level (counts of number of students receiving a certain score) for grade 4. In grades 9 and 12, the examination scores provided were at student level for different subjects. In the examinations, students received scores on a scale from 0 to 20. Due to privacy/security concerns, no student background information (sex, age or socioeconomic variables) was provided. Datasets provided are for the years 2015 to 2019 and geographic coverage include schools in all regions (marzes) with coverage (number of schools reported) improving in recent years. At the time of writing this report, a national large-scale Standardized Assessment (NLSA) to determine student performance is not available. The last NLSA was conducted in 2010 in Armenian language and literature and Armenian history. The second source of data is a teachers survey conducted in all schools across Armenia by NaCET in December 2019. Teachers’ characteristics collected in the survey were gender, date of birth, area of study, year of graduation, education institutional graduated from, training, position title, workload measure, date appointed as teacher, grades taught, subjects taught, salary and bonuses. School identifiers include the marz the school is located, community, type of school (primary, basic, high, and secondary school). The final source of data was dataset on school quality/school infrastructure prepared by NaCET in December 2020. The dataset includes a subset of school characteristics: location of the school (marz, region, community and area classification as urban (town) or rural (village), grades and education levels coverage provide by the school, legal status (public, private), condition of the school building, access to services (water supply, sewage, power supply, gas supply, heating, hot water, landline phone, internet), number of classrooms per grade, enrollment per grade, and number of teachers (full-time, primary teachers, part-time). Following several iterations, we constructed a dataset with 1229 schools, 31034 students tested in 2019 . Due to lack of harmonized school id/school codes across datasets, merging required to use two common open text 11 fields: name of the region and name of the school. The analytical dataset consisted of 28995 students tested in mathematics grade 9 attending basic (grades 1-9) and secondary (grades 1-12) public schools (state-owned). Analytical procedures In this study we use a hierarchical multilevel analysis. A multilevel approach was adopted to capture the hierarchical structure of students nested within schools making it possible to investigate the association between measures at student level and school characteristics. The TIMSS 2019 data was analyzed with a two- level hierarchical linear model to study the relationship between the school variables and the grade 4 student mathematics achievements. To describe the relationship at level 1, the model includes student level variables, and at level 2, school level variables from both the teacher and the school questionnaires were included in the model. A description of the variables used in the model are provided in Table A2. Analyses are adjusted with the house weight at student level, a normalization of the total student weight so the weighted sample corresponds to the actual sample size in each education system. The national examinations also were analyzed following a multilevel hierarchical linear modelling approach (students nested in schools), however the results of the analysis are limited to the relationship of school characteristics to students performance in the exams due to the no availability of students variables. Variables used in the analysis are reported in Table A3. The analyses were conducted in two steps. First the variance between and within schools in relations to mathematics achievement (TIMSS scores and Examinations scores) was estimated using a null model with no predictors. This model allows to estimate the variance attributed to each level of analysis. Second, the model was adjusted by introducing student-level (only for TIMSS), and school level variables. The analysis of the National Examinations datasets was conducted with all the schools, and for three separate samples based on location of the schools, schools in Yerevan, schools in urban areas, and schools in rural areas. 4. Quality of Learning in Armenia More than a quarter of children in Armenia are not proficient in reading, but this share has been declining. Around 26 percent of children at late primary-school age are not proficient in reading.13 This, also classified as learning poverty, means being unable to read and understand a short, age-appropriate text by age 10. Learning poverty in Armenia is 17.2 percentage points worse than the average for the ECA region (figure 4). 13The learning poverty number for Armenia is calculated using the Global Learning Assessment Database (GLAD) harmonization based on TIMSS. The Minimum Proficiency Level (MPL) threshold used was low level (400 points). For more details, please consult the GLAD and learning poverty repositories in GitHub. 12 Figure 4. Learning Poverty in Armenia and Comparators, 2016 or Latest Available Data 50 43.1 40 30 26.1 20 13.8 15.0 16.5 8.9 9.3 11.7 10 4.3 6.3 2.2 3.3 0 Source: Author’s calculations using World Bank data, 2016–2019. Note: ECA average comprises latest available data between 2016–2019. Armenian students’ learning levels in math and science have been improving since 2011, especially in early grades. A large-scale learning assessment, Trends in International Mathematics and Science Study (TIMSS) indicate that students are fast approaching the international benchmark levels in math and science, especially in 4th grade .14 The highest scores achieved by Armenia in both science and math in the 4th grade were recorded in 2019 (figure 5). Figure 5. TIMSS Math and Science Average Score for Armenia Grade 4th, 2003–2019 Grade 8th, 2003–2015 520 520 498 500 481 500 478 480 466 480 467 471 456 452 461 460 460 444 460 437 437 440 440 416 420 420 400 400 2003 2007 2011 2015 2019 2003 2007 2011 2015 Math Science Math Science Source: Authors’ calculations using International Association for the Evaluation of Educational Achievement (IEA) data, 2004– 2020. Note: TIMSS describes average ‘low’ scores as those between 400 and 475, average ‘intermediate’ scores as those between 475 and 550, and average ‘high’ scores as those above 550. The 2017 results are not comparable. Students at the bottom levels of proficiency improved much more than top performers in 2015 and 2019. Those with the most improved performance have been the low performers, which is a very valuable achievement. Over the last two TIMSS rounds, math performance improved particularly among low 14 TIMSS has been conducted every four years since 1995 and has been a valuable tool for monitoring international trends in mathematics and science achievement at the fourth and eighth grades. 13 performers. Math scores for the 5 percent of students with the lowest proficiency levels increased 72 points, from 316 in 2011 to 388 in 2019. Overall, the improvement was higher between 2011 and 2015 than between 2015 and 2019. Top performers, however, have only slightly improved. This suggests that there is still a long way to go for students that are at higher levels of proficiency in math and science (figure 6). This is significant because the qualifications of students performing at a higher level is an important measure of skills relevance, future productivity, and the health/competitiveness of the labor market. Figure 6. TIMSS Math Scores in 4th Grade by Percentile, 2011, 2015 and 2019 700 TIMSS Math scores 600 500 400 300 200 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 Percentiles 2011 2015 2019 Source: Authors’ calculations using IEA data, 2011–2019. Armenia’s significant improvement in TIMSS could be an indication that reform efforts of the last decade are paying off: Armenia has focused on assessment, supported in part by the Russia Education Aid for Development (READ) trust fund designed to enhance the country’s capacity in student assessment. Specifically, the assessment capacity of the teaching workforce has been improved; the scope of the National Large-Scale Assessments (NLSA) was expanded; qualified assessment specialists were trained, and a sustainable cadre of local assessment specialists were developed.15 Participation in TIMSS assessments led to the introduction of national assessments of different subjects and had an impact on the process of education reform in Armenia. Due to the impact of TIMSS on pedagogical reforms in Armenia, the testing process is used widely for all subjects and includes several methods (e.g., continual assessments, final and unified exams, and national and classroom assessments). The MoESCS has based decisions regarding curricula, textbooks, methods of assessment, and continual assessment on TIMSS results.16 These changes together are believed to have made an impact on improved TIMSS results in Armenia over the last decade. It is important to note that Armenia still have not reached the international average (500) in science or math in either Grade 4 or 8 in TIMSS which signifies being able to apply and demonstrate basic knowledge of concepts and material. However, the improvement of learning, especially in the 4th grade, is a significant achievement that should be enhanced going forward. 4.1 Student Performance across population groups Educational inequalities can have long-lasting effects on educational and economic opportunities. Students from lower socioeconomic backgrounds may perform worse in schools than students from higher 15 World Bank READ Trust Fund: Armenia Brief, 2020. 16 IEA, 2019. 14 socioeconomic backgrounds. They are also more likely to attain lower education levels and earn less in the labor market as adults. In addition, learning levels and quality in Armenia have likely been greatly impacted by the Covid-19 pandemic since it has caused widespread school closures and learning interruptions. Its estimated impact on learning in Armenia is observed in box 3. Box 2. COVID-19 Pandemic’s Impact on Learning The COVID-19 pandemic caused school closures and learning disruptions at unprecedented levels. The World Bank built a simulation model to estimate the learning losses for each country as a result of COVID-19. Based on this model’s assumptions that (i) schools were closed for ten and that (ii) remote teaching in the country is half as effective as face-to-face teaching, it is estimated that learning-adjusted years of schooling (LAYS) in general education in Armenia would fall from their baseline of 7.9 years to 7.6 years in an intermediate scenario. Hence, it is critical to mitigate the potential adverse impacts of COVID-19 by supporting teachers and students (Azevedo et al. 2020). The effect of COVID-19 on education may lead to economic harm unless action to recover learning losses and protect human capital is effectively taken as soon as possible. Armenia, much like other countries, needs to protect education spending, ensure remediation to recover learning losses, and invest in building a resilient education system for other potential disruptions in the future. Supporting teachers is a big part of these efforts as well. The World Bank outlines three key principles in order to ensure teacher effectiveness and prepare teachers for challenges and disruptions related to COVID-19 both now and in the foreseeable future: (a) support teacher resilience (b) support teachers instructionally and (c) support teachers technologically. Source: Beteille et al (2020). Note: LAYS is an indicator that takes into account the average years of schooling in general education while adjusting those years by the actual amount of learning that takes place. There are learning gaps between different groups of students, particularly between the wealthiest and poorest quintiles. On an average, in Armenia, students from families with higher socioeconomic status have higher scores. In the TIMSS 2019 4th grade math assessment, there was a 50-point score difference between the students from the wealthiest and those from the poorest quintiles, which is equivalent to roughly two years of schooling (figure 7).17 This means students at the lowest socioeconomic level (bottom 20 percent) are, on an average, two years behind in learning compared to their peers from the top 20 percent. Even though there is a clear score advantage to being in an urban location (10-point score difference when compared with those in a rural setting), which is especially large and significant for 4th grade math scores, the largest and most significant score differentials are still between the socioeconomic levels of students. Although math scores do not reflect a difference between gender, girls perform slightly better in science than boys (471 vs 462 points, respectively) and learning poverty (measuring reading skills at the end of primary education) is higher for boys (28 percent) than girls (23.8 percent). 17 Thirty to forty points is roughly equivalent to one year of schooling. 15 Figure 7. TIMSS 4th Grade Math Score Differences by Gender, Location, and Socioeconomic Level, 2019 Richest quintile 524 SES Gap: Poorest quintile 475 49.7 * Urban setting 503 Area Gap: Rural setting 490 -12.7 * Boys 497 Sex Gap: Girls 500 2.4 n.s. 400 425 450 475 500 525 550 Source: Authors’ calculations using IEA data, 2019. 4.2. Student Performance across Regions Low learners are widespread across all Armenian marzes (regions). The analysis of learning at the regional level is conducted based on the 9th grade national test for 2019. Low learners are defined as students who are in the lowest quartile of the national distribution of scores per year and subject –proficiency thresholds are not directly provided in the national exams. Across all regions and subject areas, low learners represent at least 20 percent of total students who took the 9th grade national examination in 2019 in each region (map 1). This suggests that in all regions, there is a need to ensure the acquisition of basic competencies and skills. However, differences exist among the marzes with Yerevan having a consistently smaller proportion of low learners than the rest of the marzes. Yerevan has the lowest share of students among low learners. Other regions such as Syunik and Shirak also have relatively low proportion of low learners (below 30 percent in all subject areas). In contrast, Ararat region has the largest proportion of low learners across all subject areas, followed by Armavir (map 1). There is a spatial distribution of learning across regions, with those located in the south (except Syunik) registering the highest proportion of low learners. Map 1. Percentage of Low Learners in 9th Grade, Average across All Subject Areas, 2019 STEM Social Sciences Foreign Languages Armenian Language and Literature 16 Source: Authors’ calculations using national exam data. Note: Science, technology, engineering and mathematics (STEM) subjects include biology, chemistry, physics and mathematics; foreign languages include English and Russian; social sciences include geography and history. Within the city of Yerevan, a spatial clustering of results is observed. Students attending schools in districts located on the west side of the city (Shengavit and Malatia-Sebastia) have math scores that placed them in the lowest quartile of the distribution each year (see map 2). The opposite occurred among students in schools located in the districts of Arabkir, Kentron/Nork-Maras, Erebuni/Nubarashen, and Kanaker-Zeytun; they are consistently ranked as the top scorers. Arabkir, Kentron/Nork-Maras and Kanaker-Zeytun are also the districts with the lowest rate of low learners in almost all subjects (see table 1). The districts with the top scorers are those that are the most affluent, and this may be partly or fully responsible for this trend as families with access to more resources have the ability to procure assistance and tutoring for their children. Map 2. District Variations across Yerevan in percentage of Low Learners, Grade 9, Mathematics, 2019 Share of Low Learners Source: Authors’ calculations using national exam data. 17 Table 1. Percentage of Low Learners in 9th Grade by Subject Areas in Yerevan, 2019 Foreign Armenian Language STEM Social Sciences Languages and Literature Ajapnyak (Davtashen) 24 27 24 22 Arabkir 16 23 17 19 Avan 22 20 25 21 Erebuni (Nubarashen) 29 29 25 25 Kentron, Nork-Maras 18 26 17 16 Malatia-Sebastia 30 32 27 24 Nor Nork 24 27 20 18 Kanaqer-Zeytun 12 27 19 13 Shengavit 27 32 28 21 Minimum 12 20 17 13 Maximum 30 32 28 25 GAP 19 13 11 12 Source: Authors’ calculations using national exam data. Disparities in learning across regions are more marked in STEM subjects. The gap between the regions with the highest and lowest share of low learners is about 20 percentage points in STEM subjects (43 for Ararat and 24 for Yerevan), while the gap is only 14 percentage points in social sciences. Across STEM subjects, the gap is substantially high for chemistry and much lower in biology. The percentage of low learners in chemistry is 61 percent in Vayots Dzor, while it is only 13 percent in Aragatsotn. However, in both STEM and social science subjects, at least one in five students is a low learner across all regions (table 2). Table 2. Percentage of Low Learners by Subject, 2019 Armenian Language STEM Social Sciences Foreign Languages and Literature Armenian English Russian Armenian Armenian Biology Chemistry Mathematics Physics Geography History Language Language Language Literature Aragatsotn 27 13 28 28 22 35 31 32 35 27 Ararat 36 49 38 45 36 44 36 38 41 36 Armavir 35 46 33 38 33 41 34 38 34 35 Yerevan 22 25 20 23 23 32 22 24 18 22 Gegharqunik 28 30 24 33 25 36 36 30 29 27 Kotayk 27 29 26 27 34 36 25 34 28 32 Lori 29 15 28 35 28 36 31 31 33 29 Shirak 28 31 26 37 21 33 27 26 30 24 Syunik 25 20 25 33 24 31 24 31 27 26 Tavush 29 15 34 36 30 30 25 36 38 31 Vayots Dzor 30 61 31 40 27 37 34 36 39 28 Minimum 22 13 20 23 21 30 22 24 18 22 Maximum 36 61 38 45 36 44 36 38 41 36 GAP 14 48 18 22 15 14 14 14 23 14 Source: Authors’ calculations using national exam data. The next section explores the key predictors of learning outcomes in Armenia, analyzing how variations in educational inputs are associated with variations in learning. 18 5. Factors associated with learning outcomes in Armenia This section references national and international assessments to evidence how various inputs relate to learning. The main dimensions of analysis are students’ characteristics, schools’ characteristics, and teachers’ characteristics. Regression analyses, for both TIMSS and national exam, were used to provide evidence on the association between those factors and student learning18. On TIMSS, the results correspond to the 2019 math assessment of 4th grade students for Armenia and some comparator countries such as Bulgaria, Croatia, Georgia, Kazakhstan, Russia, and Serbia (Table A3). On national assessments, results correspond to grade 9 math achievement in 2019, but factors associated to students were not available in the national exam (Table A4). 5.1 Variance explained by students, schools, and regions In Armenia, the largest part of the variance in learning results is explained at the student level, followed by schools, with a very low contribution of regions. In TIMSS 2019, the decomposition of the total variation in grade 4 scores in mathematics shows that 76 percent of the total variation is associated with students and the remaining 24 percent is associated with schools .19 In comparison with other countries, the school variation in Armenia has remained stable compared to the previous rounds of TIMSS assessments (figure 8). The analysis of the results of national exams in mathematics in grade 9 shows a similar share of the variation attributable to schools (20.9 percent). In grade 12, the share of the variation attributable to schools is higher at 32 percent (Table A4). Differences in regional attributes explained a smaller part in performance variation. In the national exams, regions capture a statistically significant but very reduced fraction of the total variation, around to 2 percent (Table A5). Figure 8. Variation in Grade 4 Math Scores Attributable to Schools in Armenia and Comparator Countries, TIMSS 2011, 2015 and 2019 70% 60% 60% 60% 50% 46% 44% 41% 39% 40% 35% 33% 34% 34% 31% 27% 24% 24% 27% 30% 17% 20% 14% 16% 10% 0% 2011 2015 2019 2011 2015 2019 2011 2015 2019 2011 2015 2019 2011 2015 2019 2011 2015 2019 2011 2015 2019 Armenia Bulgaria Croatia Georgia Kazakhstan Serbia Russia Source: Authors' calculations using IEA data for 2015 and 2019 TIMSS rounds. Share in variation is estimated with a null multilevel regression model with students nested in schools. 18 For both TIMSS and national exams, a multilevel regression model (hierarchical linear model or mixed model) of two levels was applied with students nested within schools. 19 The relative distribution of the variability in the scores of the students within each school and the average variability between schools were estimated with a multilevel null regression model without predictor variables. The variability at the regional level was also added for the 9th national test. 19 Socioeconomic background and attitude toward learning are the key drivers of performance at the student level. In TIMSS 2019, the index of socioeconomic level measured as “Home Resources for Learning� is one the strongest factor associated with math performance in grade 4 in Armenia and all other ECA countries in Table A3. It also manifests equity issues. While some degree of variation in education outcomes is to be expected in any school system, equity means that whatever variations there may be in education outcomes, they are not related to students’ background20. At the student level, attitude toward learning mathematics also matters. In TIMSS 4th grade, students who tend to like learning mathematics and are more confident with mathematics were more likely to perform better, not only in Armenia but in other ECA countries (Table A6). Schools play a significant role in equalizing learning opportunities, especially across regions. In Armenia, 21 percent of the variation in learning outcomes in mathematics in national exams is explained by schools, as described before. In regions where schools have higher contribution to students’ learning (as measured by the part of the variance explained by schools) have lower shares of low learners (figure 9). This is also the case for Shirak, in which the proportion of low learners is below expected, given poverty levels which is one of the highest in the country, but where schools explain one of the highest percentages of math performance—a variance explained of about 23 percent. Next section will explore key determinants of learning at the school level. Figure 9. Percentage of Variance in Math Explained by Schools and Percentage of Low Learners, 2019 40 Ararat Tavush 35 Armavir % Low Leaners 30 Vayots Dzor Aragatsotn Lori 25 Kotayk Shirak Syunik 20 Gegharquniq Yerevan 15 10 10 12 14 16 18 20 22 24 26 % Variance Explained by Schools Source: Authors’ calculations using national exam data. 5.2 School-Level Factors and Student Performance Armenia’s teacher training system and teacher policies need improvements as teachers are critical for the development of cognitive and non-cognitive skills in schools. In Armenia, about 31,018 teachers teach in general schools (Grades 1–12). The law in education establishes that teachers should have obtained a pedagogical qualification or have higher education 21 and at least five years of service of pedagogical (or teaching) activities in the last ten years. Nonetheless, the effectiveness of in-service and pre-service training is low, and teachers do not always come into the profession having mastered modern teaching methods and 20 OECD (2019), PISA 2018 Results (Volume II): Where All Students Can Succeed, PISA, OECD Publishing, Paris, https://doi.org/10.1787/b5fd1b8f-en. 21The higher education system in Armenia consists of state and private institutions including universities, academies, institutes, and conservatoires. 20 technologies.22 Teacher qualification was an important driver of learning in TIMSS 2015, but lack of robust data prevents the examination of this relationship in 2019. While TIMSS 2015 showed a positive relationship between the highest level of education attained by teachers (especially those with bachelor’s degrees and higher) and student results in the 4th grade math test, this is not the case in TIMSS 2019. The results in TIMSS 2019 might be attributable to a methodological issue in the data collection for this question. However, this should not mean that higher levels of teacher professionalization in the form of bachelor’s, master’s, or doctoral degrees (accompanied by high quality of the aforementioned programs) do not lead to enhance quality of education among students. In the most successful education systems such as Finland and South Korea, teachers are required to have advanced degrees. In Armenia, teacher professional development matters in student learning, while teachers report that they need training in curriculum, ICT, and working with students with special needs. Teacher professional development allows educators to develop the knowledge and skills to address learning challenges. In Armenia, the number of teachers who participated in professional development in math is positively correlated with TIMSS math scores. Teachers who report that they need training in math in the future positively correlates with learning, suggesting that the open-minded attitude towards new skills acquisition is important. In this regard, the new curriculum that is being piloted in Armenia is a critical intervention as teachers reported new training, ICT skills for teaching and skills for teaching students with special needs as their main challenges.23. 22 Republic of Armenia Ministry of Education and Science, Armenia: Education for All 2015 National Review (2015), https://www.globaldisabilityrightsnow.org/sites/default/files/related-files/275/Education_Report__2015__English.pdf. 23 These findings were based on a teacher training questionnaire developed by the World Bank Armenia team and implemented by NaCET. 21 Box 3. Teach Armenia Findings Teach measures (i) the time teachers spend on learning and the extent to which students are on task, and (ii) the quality of teaching practices that details classroom culture, instruction, and socio-emotional skills. Teach Armenia measured teacher practices of 80 teachers in grades 2, 4, 5, 7 and 8 across several subjects in 20 primary schools in Tavush and Yerevan. Findings from the study are summarized below: • Teachers in Armenia provide a learning activity to students 97 percent of the time and when the learning activity is provided, all students are on task most of the time (78 percent). • Armenian teachers have strong ability in classroom culture and instruction. However, they exhibit weaker ability in socio- emotional skills (76 percent have a score lower than 3 and 11 percent lower than 2) (figure 10). Figure 10. Distribution of Average Teach Scores by Area and Overall Classroom Culture 100% 80% 100% Instruction 75% 75% 60% 50% 50% 19% 29% 25% 1% 25% 11% 0% 0% 0% 0% 1-2 2-3 3-4 4-5 1-2 2-3 3-4 4-5 Distribution of Scores Distribution of Scores Socio-Emotional Skills 100% 76% 75% 50% 25% 11% 13% 0% 0% 1-2 2-3 3-4 4-5 Distribution of Scores • Teachers are relatively skilled at creating a supportive learning environment, setting positive behavioral expectations, facilitating the lesson and checking for understanding. However, they score around the medium range for providing feedback and encouraging students to think critically. They also score around the medium range in promoting student autonomy but are weaker at fostering perseverance and social and collaborative skills (figure 11). Figure 11. Distribution of Average Teach Scores by Element Cultur Classr 1.Supportive Learning Environment 3.8 oom e 2.Positive Behavioral Expectations 3.3 3.Lesson Facilitation 3.6 Instruction 4.Checks for Understanding 3.5 5.Feedback 2.8 6.Critical Thinking 3 Socioemo 7.Autonomy 2.9 tional Skills 8.Perseverance 2.3 9.Social and Collaborative Skills 1.8 1 2 3 4 Source: Teach Armenia (World Bank 2021). 22 Disciplinary environment in the classroom is a good predictor of student learning in Armenia. In Armenia, school discipline was found to be a predictor of math scores in TIMSS 2019, and it is one of the few factors that show a positive association with results in most countries in the Programme for International Student Assessment (PISA). In particular, the index of school discipline and index of safe and orderly schools were both positively correlated with TIMSS 4th grade math scores. This may reflect the fact that teachers with a positive learning environment are more effective in classroom management and can focus their efforts on academic activities. Likewise, while a positive school climate is associated with other socioeconomic factors that are related to performance, TIMSS analysis suggests that once these factors are accounted for, the school environment remains an important determinant of results (table A6). Physical conditions of the school buildings in Armenia need to be improved, especially in schools where basic utilities are not guaranteed. The results of the regression analysis from national examination finds that the average performance of the schools is correlated with by infrastructure (bivariate model), but once other factors are added to the model, this relation become not significant (multivariate model – Table A7). This is consistent with global evidence that shows either mixed or positive correlations on learning, while it is generally accepted that strongest drivers of learning rely on what happen in classroom, which include teachers and learning practices. Despite the high share of the education expenditure devoted to infrastructure, about 28 percent of school facilities need repair and another 32 percent need rehabilitation. These rates are higher in rural schools. In terms of access to basic utilities, 8 percent of rural schools do not have access to water supply, 15 percent do not have sewage systems, and 45 percent do not have gas supply (figure 12). These numbers along with the decline of the population brings an opportunity to enhance physical conditions, while schools are rationalized – maintaining proper class sizes, as large class size were negatively correlated with leaning in both TIMSS and national exams (figure 13). Figure 12. Condition of the School Building by School Location, 2019 100 80 43 38 40 60 % 31 32 40 35 20 31 28 22 0 Urban (Town) Rural (Village) Total Needs ongoing repair Needs rehabilitation Satisfactory Source: Authors’ calculations using administrative data. 23 Figure 13. Percentage of Low Learners and Percentage of Schools with Satisfactory Condition, 2019 45 Vayots Dzor Ararat 40 Percentage of Low Learners Tavush 35 Armavir Aragatsotn 30 Lori Gegharquniq Kotayk 25 Shirak 20 15 Syuniq 10 5 0 20 25 30 35 40 45 50 Percentage of Schools with Satisfactory Condition Source: Authors’ calculations. Note: Average of all subjects in grade 9. The location of a school is significant in understanding the differences in the aggregated performance, which show that schools in Yerevan, followed by other urban areas perform better. Both TIMSS and national exam show that Yerevan perform comparatively better than other urban and rural areas. This finding is consistent after accounting for student’s socioeconomic background. Additionally, other urban areas perform better than rural areas, difference that is positive and significant in national examination for grade 9th, but non-significant in TIMSS 4th grade. It may suggest that disparities increase as student progress through the education system. The multivariate regression analysis of student performance in the national exams reveals that teachers the presence of larger share of teachers older than 40 years old has a positive and statistically significant association in the Math test, but the percentage of those teacher is 9 percentage points in Yerevan than in rural areas (see table A8). There are other potential factors behind the relatively high learning poverty rate and low learning levels in Armenia that could not be tested for due to lack of data. These are (i) assessment data not informing instruction; (ii) inadequate support given to teachers for lesson planning; and (iii) relevant content missing in the reading material and in the curriculum. These potential explanations make it important to consider interventions around curricular reform and more effective teacher policies that align with the needs of learners. It is noteworthy that Armenia is going through policy changes exactly along these lines. 24 6. Policy Recommendations The findings of this study support the following three recommendations, which focus on improving learning across all education levels in Armenia: (i) improving effectiveness of teaching, (ii) ensuring high quality and equitable educational resources across regions, and (iii) measuring learning and implementing relevant programs targeting low learners to fill the learning gaps. (i) Improving effectiveness of teaching. Teachers are the most important drivers of learning. The analytical work shows that both pre-service (qualification) and in-service (continuing professional development) training make substantial contributions to the development of skills and competencies of Armenian students. 24 Based on the findings of the Learning Analysis and Teach Armenia, it is important to focus on ensuring a strong practicum component in pre-service education so that teachers are well-equipped to transition to and perform effectively in the classroom. Teach Armenia findings show that teachers have challenges with providing feedback, encouraging students to think critically, promoting student autonomy, and fostering perseverance and social and collaborative skills. Providing continuous support to teachers in the form of high-quality, in-service training that focuses on these specific issues and the broader technical aspects of teaching would benefit both teachers and students alike and possibly lead to better learning outcomes. Finally, it is important that teachers learn to employ ICT to enhance their ability to reach every student.25 This has been especially relevant in the context of COVID-19, which provided an opportunity for teachers to build technological skills. Once school systems stabilize, teachers should be encouraged to maintain and improve their technological skills. School leadership can support this through investments in hardware, connectivity, and continuing professional development in ICT. These solutions should be tested locally before scaling. (ii) Ensuring high quality and equitable educational resources across and within regions and locations. In Armenia, disparities in learning outcomes across urban and rural areas are significant, but the Learners’ Report shows that schools can compensate for pre-existing socioeconomic disadvantages. As part of the national goal of promoting local development, a more decentralized approach to distribute educational resources can be considered to enhance learning outcomes in all regions. This may include policies related to teachers and infrastructure, and the analysis of how the transfer of public financial resources to regions can contribute to improved learning. The current low expenditure on education offers an opportunity to devote additional resources toward the education sector in an efficient manner. Increasing expenditure can support the transformation of the teaching profession and infrastructure development. More importantly, it can support vulnerable students that struggle to acquire basic skills and competencies. Material and financial resources can be delivered to all schools by: (i) sustaining education budgets and investing where returns are greatest, or more specifically, ensuring that critical expenditures are maintained to keep children enrolled (and minimize dropouts) and to protect the most vulnerable and more disadvantaged students, (ii) using targeted 24 In order to enhance teacher effectiveness, teachers can be trained in areas in which they feel they need training. In Armenia, this includes student assessment for the new curriculum, ICT skills for teaching, and teaching students with special needs. 25 Béteille, Tara and David K. Evans. 2019. Successful Teachers, Successful Students: Recruiting and Supporting Society’s Most Crucial Profession. 25 block grants to ensure that funds reach disadvantaged/vulnerable schools, and (iii) making sure that levels of teacher salaries are maintained and that teachers are paid on time.26 (iii) Measuring learning and implementing relevant programs that target low learners to fill the learning gaps. In the post-pandemic period, measuring learning loss and regularly monitoring learning will be critical in tackling learning inequalities. This period presents an opportunity to implement learning recovery policies as permanent elements into education systems to improve learning equity and reduce learning poverty. The emerging consensus on education quality and learning poverty suggests that policies on learning equity should: (i) identify the issues associated with disparities in learning outcomes between mainstream and disadvantaged groups in the population, such as access to quality teachers, pedagogical resources, compensatory programs, and the need for educational materials in the student’s main language, (ii) clearly define equity goals and the resources needed to improve learning equity, (iii) implement policies and innovations for hybrid instruction in a way that does not increase the workload on teachers, (iv) monitor learning outcomes by tracking learning progress in real time, testing students often, within the context of instructional change, (v) promote a climate of innovation for improving hybrid/blended methods of education delivery using the individual experiences of schools as a means of testing what works and seeing whether it can be scaled up, (vi) invest in digital pedagogy that oversees the provision of training to upgrade teachers’ digital skills and solicit the participation of trained teachers in the development of a new digital pedagogy, and (vii) use impact monitoring and evaluation to track progress in learning, learning equity, and learning performance under the hybrid methods of instruction to ensure the efficiency and accountability of the education system at large. This type of monitoring is crucial for the long-term transformation of education.27 26 Arcia, Gustavo, Rafalel de Hoyos, Harry Patrinos, Alina Sava, Tigran Shmis, and Janssen Teixeira. 2021. Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice. World Bank, Washington, DC. 27 Ibid. 26 Recommendation Actions Responsible Timing: ST (< Fiscal Cost: Priority Entity 1 year); LT (> Small, (highest= 1 year) Medium, 1) Large 1. Improving Focus on ensuring a strong practicum component in pre-service education so MoESCS and Short term Small to 1 effectiveness of that teachers are well-equipped to transition to and perform effectively in the other (building into medium teaching classroom relevant long term Provide continuous support to teachers in the form of high-quality in-service partners results) training. Teach Armenia findings show that teachers have challenges with providing feedback, encouraging students to think critically, promoting student autonomy, and fostering perseverance and social and collaborative skills. Teacher training focused on these specific issues and the broader technical aspects of teaching would benefit both teachers and students alike and possibly lead to better learning outcomes. Use technology wisely to enhance the ability of teachers to reach every student. This is especially relevant in the context of COVID-19, which provided an opportunity for teachers to build technological skills. Once school systems stabilize, teachers should be encouraged to maintain and improve their technological skills. School leadership can support this through investments in hardware, connectivity, and continuing professional development in ICT. These solutions should be tested locally before scaling. 2. Ensuring high A more decentralized approach to distribute educational resources can be MoESCS, Long term Medium 1 quality and considered to enhance learning outcomes in all regions. Resources can be Ministry of equitable delivered to all schools by: (i) sustaining education budgets and investing where Finance educational returns are greatest, (ii) ensuring that critical expenditures are maintained to resources across keep children enrolled (and minimize dropouts) and to protect the most regions vulnerable and more disadvantaged students, (iii) using targeted block grants to ensure that funds reach disadvantaged/vulnerable schools, and (iv) making sure that levels of teacher salaries are maintained and that teachers are paid on time. 3. Measuring In the post-pandemic period, measuring learning loss and regularly monitoring MoESCS Long term Large 2 learning and learning will be critical in tackling learning inequalities. Policies on learning equity implementing should: (i) identify the issues associated with disparities in learning outcomes relevant programs between mainstream and disadvantaged groups in the population, such as targeting low access to quality teachers, pedagogical resources, and compensatory programs learners to address and the need for educational materials in the student’s main language , (ii) clearly learning gaps and define equity goals and the resources needed to improve learning equity, (iii) 27 further improve implement policies and innovations for hybrid instruction in a way that does not learning increase the workload on teachers, (iv) monitor learning outcomes by tracking learning progress in real time, testing students often, within the context of instructional change, (v) promote a climate of innovation for improving hybrid/blended methods of education delivery using the individual experiences of schools as a means of testing what works and seeing whether it can be scaled up, (vi) invest in digital pedagogy that oversees the provision of training to upgrade teachers’ digital skills and solicit the participation of trained teachers in the development of a new digital pedagogy, and (vii) use impact monitoring and evaluation to track progress in learning, learning equity, and learning performance under the hybrid methods of instruction to ensure the efficiency and accountability of the education system at large. 28 REFERENCES Arcia, Gustavo, Rafalel de Hoyos, Harry Patrinos, Alina Sava, Tigran Shmis, and Janssen Teixeira (2021). Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice. World Bank, Washington, DC. Armstat (2017). Households’ Integrated Living Conditions Survey, Anonymized Microdata Database (by Households) (2017). Asian Development Bank (2019). Human Development Enhancement Program, Sector Assesment (Summary): Education and Health. Retrieved from https://www.adb.org/sites/default/files/linked-documents/51129-002- ssa.pdf Azevedo, João Pedro, Amer Hasan, Diana Goldemberg, Syedah Aroob Iqbal, and Koen Geven (2020). Simulating the Potential Impacts of Covid-19 School Closures on Schooling and Learning Outcomes: A Set of Global Estimates. Washington, DC: World Bank. Beteille, Tara, and David K. Evans (2019). Successful Teachers, Successful Students: Recruiting and Supporting Society’s Most Crucial Profession. Washington, DC: World Bank. Beteille, Tara; Ding, Elaine; Molina, Ezequiel; Pushparatnam, Adelle; Wilichowski, Tracy (2020). Three Principles to Support Teacher Effectiveness During COVID-19. World Bank, Washington, DC. International Association for the Evaluation of Educational Achievement (IEA). (2019). TIMSS 2019 Encyclopedia: Education Policy and Curriculum in Mathematics and Science in Armenia. OECD (2019), PISA 2018 Results (Volume II): Where All Students Can Succeed, PISA, OECD Publishing, Paris, https://doi.org/10.1787/b5fd1b8f-en. Republic of Armenia Ministry of Education and Science (2015). Armenia: Education for All 2015 National Review, https://www.globaldisabilityrightsnow.org/sites/default/files/related- files/275/Education_Report__2015__English.pdf Rutkowski, Jan. 2013. Skills Employers Seek. Results of the Armenia STEP Employer Skills Survey . Report no. 98904. Washington, DC: World Bank. http://microdata.worldbank.org/index.php/catalog/2567 Statistical Committee of the Republic of Armenia (Armstat) and World Bank, Social Snapshot and Poverty in Armenia: Statistical and Analytical Report (Yerevan, 2018). https://www.armstat.am/en/?nid=205. UNESCO Institute for Statistics (UIS), 2020. http://uis.unesco.org/country/AM World Bank (2018). World Development Report 2018: Learning to Realize Education’s Promise . Washington, D.C.: World Bank. 29 World Bank (2020). READ Trust Fund: Armenia Brief, 2020. https://documents1.worldbank.org/curated/en/552741619466400084/pdf/READ-Trust-Fund-Armenia.pdf 30 ANNEX Table A1. Sample sizes of selected participant countries in TIMSS 2019 Grade 4 Mathematics and Science Assessment ISO Code Country Students Classes Teachers Schools ARM Armenia 5399 216 212 150 BGR Bulgaria 4268 211 209 151 HRV Croatia 3785 263 263 153 GEO Georgia 3787 223 220 154 KAZ Kazakhstan 4791 224 224 168 SRB Serbia 4380 214 214 165 RUS Russia 4022 200 200 200 31 Table A2. Description of variables used in the analysis of TIMSS 2019 Grade 4 Mathematics Assessment Variable Definition Measurement Math Achievement in Grade 4 Based on 5 plausible values Continuous Student Characteristics Male Student’s sex is male (ITSEX). Indicator (dummy 0/1) Students Age Student’s biological age. Continuous Socioeconomic Status (Home SES scale. This scale measures the availability of Continuous Resources for Learning) resources suitable for learning at the student’s home. In the TIMSS International Report this scale is referred as the Home Resources for Learning Scale (ASBGHRL). Students Like Learning Mathematics Scale of nine questions concerning student’s Continuous Scale. interest and attitude toward mathematics Standardized within country. (ASBGSLM). Students Confident in Mathematics Scale based on the responses to seven questions Continuous Scale. concerning student’s confidence on learning Standardized within country. mathematics (ASBGSCM). Student Bullying Scale based on the responses to seven questions Continuous Scale. concerning student’s self-report to experiences Standardized within country. related to bullying (ASBGSB). Teacher Characteristics Sex Teacher’s sex is female (ATBG02). Indicator (dummy 0/1) Age Teacher’s age measured in intervals (ATBG03). Indicator (dummy 0/1) Transform into binary indicator for teachers of 40 years or more. Education Attained Highest level of education of the teacher: Categorical Secondary Bachelor and Master/PhD (ATBG04). Majored in Ed and Math Derived as a binary indicator when the teacher’s Indicator (dummy 0/1) major was in Education and the area of specialization was Mathematics (ATDMMEM). Professional Development in the past Have participated in Professional Development in Indicator (dummy 0/1) Math in the past two years (ATBM09A) Professional Development in the Will need Professional Development in Math in the Indicator (dummy 0/1) future future (ATBM09B) Job Satisfaction Teacher’s responses to five items on the level of Continuous Scale. satisfaction in the current job (ATBGTJS). Standardized within country. Safe and Orderly Schools Scale based on teacher’s responses to eight items Continuous Scale. (standardized) related to security and orderly behavior of the Standardized within country. students in the school (ATBGSOS). Class size Number of students in the class the student Continuous Scale attends (ATBG10A). Weekly time spent teaching Number of weekly hours dedicated to teaching Continuous Scale mathematics in the class mathematics School Characteristics School location Area where the school is located and consist of Categorical three categories: 1) schools are located in a large city, corresponding to schools located in areas with a population of 500 thousand persons or more and extracted from 32 Variable Definition Measurement the question “how many people live in the area� (ACBG05A)1/; 2) schools located in urban areas corresponds to schools in urban–densely populated areas, suburban–on fringe or outskirts of urban area and medium size city or large town per responses in the question “immediate area of the school� (ACBG05B); 3) schools located in rural areas based correspond to the following responses in question ACBG05B: small town or village or remote rural. School has a problem of teachers Principal’s report on teachers arriving late as a Indicator (dummy 0/1) arriving late moderate or serious problem in the school. (ACBG16A). School has a problem of teachers’ Principal’s report on teachers’ absenteeism as Indicator (dummy 0/1) absenteeism moderate or serious problem in the school. (ACBG16B). School Discipline Problems Direction of the scale inverted. Higher values Continuous Scale. indicate better discipline climate in the school Standardized within country. (ACBGDAS). Instruction Affected by Math Resource Scale based on Principal’s responses to 13 items to Continuous Scale. Shortage measure school capacity to provide general and Standardize within each subject-specific resources (ACBGMRS). country. Total Instructional Time in a typical Total number of hours in a year dedicated to Continuous day (hours) providing instruction (ATBM01) 33 Table A3. School characteristics associated to Student Scores in Mathematics in grade 9. Variable Definition Source Student level Student Score Scale from 0 to 20 Examinations School level Location of the school (Town_Village): Urban or Rural NaCET school level dataset (2020) School type Education level provided by the school categorization as Basic (grades 1-9) or Secondary (1-12 grades) PTR across all school grades Total number of students across all grades enrolled divided by the total number of full-time teachers in the school Pupil-Classroom Ratio (PCR) Total number of students across in grade 9 enrolled divided by in Grade 9 the total number of classrooms in the school in grade 9 Infrastructure Index An index built using Principal Components Analysis (PCA) and a polychoric correlation matrix. Results indicate that retaining one factor about 51% of the shared variance is explained. Items included in the index are: gas supply is available, water Supply is centralized, sewerage is centralized, heating is centralized, hot water supply is available, landline phone is available Teacher Characteristics (aggregated at school level) Share of Female Teachers Average proportion of female teachers at school level Teachers Survey (2019) Share of Teachers ages 40 or Average proportion of teachers of age 40 or more at school more level. Share of Teachers trained in Average proportion of teachers who received training in 2015 or 2015 or after after. 34 Table A4. Variance Decomposition in Mathematics (Two-Level Analysis: Students and Schools) Grade: 9 2017 2018 2019 Schools (L2) Variance Estimate 24.2% 22.5% 20.9% Lower Bound 22.4% 20.7% 19.2% Upper Bound 26.1% 24.4% 22.7% Schools (L2) 1277 1196 1214 Students (L1) 29083 29187 30985 Grade: 12 2017 2018 2019 Schools (L2) Variance Estimate 31.3% 33.7% 32.3% Lower Bound 28.8% 30.6% 29.7% Upper Bound 34.0% 36.9% 35.1% Schools (L2) 863 754 839 Students (L1) 15656 7289 14249 Note: Share of variance using a hierarchical two-level null regression model with students (L1), nested in schools (L2). Lower and upper bounds of the variance estimate calculate at a 95% confidence interval. Source: World Bank staff based on National Exam 2017-2019 Table A5. Variance Decomposition in Mathematics (Three-Level Analysis Students, Schools and Regions) Grade: 9 2017 2018 2019 Marzes (L3) Variance Estimate 1.5% 1.8% 1.6% Lower Bound 0.6% 0.7% 0.6% Upper Bound 3.9% 4.5% 4.1% Schools (L2) Variance Estimate 23.9% 22.3% 20.5% Lower Bound 21.9% 20.2% 18.5% Upper Bound 26.0% 24.5% 22.5% Marzes/Regions (L3) 11 11 11 Schools (L2) 1260 1190 1209 Students (L1) 28790 29021 30827 Grade: 12 2017 2018 2019 Marzes (L3) Variance Estimate 2.1% 1.8% 1.6% Lower Bound 0.8% 0.6% 0.5% Upper Bound 5.7% 5.4% 4.6% Schools (L2) Variance Estimate 31.7% 33.6% 32.8% Lower Bound 28.9% 30.4% 30.0% Upper Bound 34.6% 37.0% 35.7% Marzes/Regions (L3) 11 11 11 35 Schools (L2) 848 749 834 Students (L1) 15139 7158 13970 Note: Share of variance using a hierarchical three-level null regression model with students (L1), nested in schools (L2), nested in marzes/regions (L3). Lower and upper bounds of the variance estimate calculate at a 95% confidence interval. Source: World Bank staff based on National Exam 2017-2019 36 Table A6. Multilevel model of the factors associated to Math TIMSS performance, grade 4. Armenia Bulgaria Croatia Georgia Kazakhstan Serbia Russia Student characteristics Female -0.62 -4.28* -8.07*** -8.45*** -3.00 1.59 -3.57** (2.23) (2.20) (2.32) (2.88) (1.89) (2.92) (1.79) Students Age (standardized) 2.78*** 3.35** 2.04** 1.74 1.82 0.85 0.11 (1.07) (1.31) (1.00) (1.60) (1.15) (1.54) (1.07) Home Resources for Learning/SCL (standardized) 10.69*** 25.14*** 16.63*** 13.38*** 5.89*** 27.99*** 9.91*** (1.18) (1.98) (1.38) (1.51) (1.15) (1.48) (1.25) Students Like Learning Mathematics/SCL (standardized) 6.38*** -6.16*** -6.97*** 1.92 1.06 -10.74*** -3.91*** (1.11) (1.49) (1.43) (1.76) (1.33) (1.92) (1.17) Students Confident in Mathematics/SCL (standardized) 18.15*** 30.72*** 32.41*** 25.84*** 16.33*** 39.31*** 24.61*** (1.25) (1.69) (1.76) (2.07) (1.23) (1.73) (1.39) Student Bullying/SCL (standardized) -0.47 -3.33** 1.25 -7.52*** -2.30** -0.89 -3.46*** (1.11) (1.31) (1.12) (1.75) (1.07) (1.26) (0.99) Teacher characteristics Teacher Majored in Ed and Math (dummy) -4.40 14.53* -5.24 7.35 -3.90 -1.93 1.67 (4.47) (8.66) (4.60) (10.69) (8.68) (4.31) (5.13) Teacher age is 40+ (dummy) 4.85 -13.51* 9.03** 6.79 6.94 5.16 -5.64 (5.16) (8.00) (4.23) (10.25) (5.85) (5.63) (6.49) Have participated in Professional Development in Math in the past two years 12.15** 7.69* 6.43 5.28 -8.81 9.21* 14.99** (5.10) (4.15) (4.66) (11.68) (8.67) (5.39) (6.03) Will need Professional Development in Math in the future 13.58** 1.67 -2.48 5.39 13.37 3.24 -7.06 (5.33) (6.46) (10.68) (8.38) (10.31) (6.46) (6.55) Teachers Job Satisfaction/SCL (standardized) 2.01 4.65*** -0.81 -2.08 -0.10 -1.70 -1.35 (2.50) (1.79) (1.76) (3.80) (3.19) (2.17) (2.46) Safe and Orderly Schools- Teacher/SCL (standardized) 3.95** -0.24 0.16 4.46 -3.68 -2.57 0.87 (1.95) (2.11) (1.61) (4.09) (3.89) (2.56) (2.43) Class Size (per 10 students) -9.15** 17.94* 9.03** -10.83 -13.00* 3.63 0.87 (4.45) (9.77) (4.11) (6.65) (6.84) (4.11) (5.74) Weekly time spent teaching mathematics in the class (in hours) 5.58 2.04 -5.04** -0.85 0.25 -0.89 -3.37 (6.34) (3.61) (1.98) (3.20) (2.07) (2.68) (3.95) School characteristics Large city (500k+ pop) 1/ 11.34* 29.53*** 1.87 18.41* 18.09 13.03 18.42** (6.42) (10.04) (5.28) (9.99) (11.21) (8.27) (9.03) Other urban area 5.96 12.51 -2.90 -3.68 24.91*** 3.60 6.09 37 Armenia Bulgaria Croatia Georgia Kazakhstan Serbia Russia (6.30) (8.36) (4.42) (8.93) (8.81) (6.52) (9.07) Teachers arriving late Is a moderate or serious problem -0.02 -82.79*** 13.94** 0.30 13.01 9.94 15.29 (20.12) (25.88) (6.13) (14.94) (27.54) (6.92) (11.54) Teacher absenteeism Is a moderate or serious problem 17.77 92.01*** 0.46 6.04 -3.84 -0.13 0.00 (20.65) (25.99) (6.19) (12.47) (20.98) (7.70) (.) School Discipline (Index) (standardized) 5.96** 7.16 -0.37 0.45 9.97** 3.70 4.60 (2.84) (4.70) (2.33) (4.87) (5.08) (3.68) (4.42) Math School capacity to provide instruction affected by shortage (standardized) 0.47 1.53 0.07 -2.00 -4.30 1.79 2.60 (2.33) (3.55) (1.87) (3.05) (3.29) (2.22) (2.95) Total Instructional Time in a typical day (hours) -3.25 -0.54 2.09* 6.54** -0.26 2.63 -3.02 (6.67) (2.32) (1.14) (2.95) (2.74) (1.81) (4.68) Constant 494.08*** 469.71*** 495.10*** 456.99*** 526.14*** 475.13*** 579.26*** (40.43) (32.21) (17.10) (27.39) (20.74) (20.94) (24.94) Cases Schools (Level 2) 133 124 141 104 141 147 178 Students (Level 1) 3883 3134 3098 2145 3576 3656 3476 Variance Total Variance 3033 3591 2650 4289 3548 3628 3155 Student Level (Residual) 2391 2645 2220 3338 2407 3007 2144 School Level 643 947 430 951 1141 622 1011 correlation ICC at school level Estimate 0.21 0.26 0.16 0.22 0.32 0.17 0.32 Standard Error 0.04 0.04 0.03 0.04 0.03 0.03 0.03 Model converged 1 1 1 1 1 1 1 Standard errors in parentheses * p<0.10, ** p<0.05, *** p<0.01 Note: 1/ In Armenia, Yerevan is the only largest city with a population of more than 500 people. In other countries also one city has a population over 500k: in Bulgaria, Sofia (1.1m); in Croatia, Zagreb (792k); in Georgia, Tbilisi (1.1m); and in Serbia Belgrade (1.7m). In Kazakhstan three cities meet the criteria (Almaty 1.8m, Nur-sultan 1.1, and Sshymkent 1.04m). In Russia more than 38 cities have population over 500k+. 38 Table A7. Predictors of average student score in Grade 9 Mathematics National Exam in 2019 Armenian Schools (1) (2) Bivariate Multivariate Student Score School Resources Pupil Teacher Ratio across all school grades 0.007* 0.000 (0.00) (0.00) Pupil-Classroom Ratio (PCR) in Grade 9 0.018*** -0.019** (0.01) (0.01) School Infrastructure Infrastructure Index 0.265*** 0.034 (0.04) (0.06) Teacher Characteristics (aggregated at school level) Share of Female Teachers 2.610*** 0.407 (0.52) (0.62) Share of Teachers ages 40 or more 1.440*** 0.941** (0.36) (0.35) Share of Teachers trained in 2015 or after -0.132 -0.126 (0.22) (0.21) Location of the school Yerevan 1.190*** 1.214*** (0.13) (0.16) Other Urban Areas 0.859*** 0.923*** (0.12) (0.14) Intercept 10.686*** (0.55) Cases Students (Level 1) 28707 Schools (Level 2) 1103 Decomposition Total Variance 9.91 Student Level (L1 Residual) 8.23 School Level (L2) 1.68 (ICC) atschool level ICC Estimate 0.17 ICC Standard Error 0.01 Explained_r2_from_Full_Model r-squared Schools (L2) 0.16 Notes: The results reported in this table are estimated using a two-level hierarchical regression estimated using random intercepts with students nested within schools. Coefficients in column (1) are estimated separately with one regression per predictor and then stacked. Column (2) reports the results of a model with all predictors estimated simultaneously. Significance levels: + p<0.10 * p<0.05 ** p<0.01 *** p<0.001 Note: This analysis is restricted to students in basic (grades 1 to 9) and secondary (grades 1 to 12) public schools 39 Table A8. Average Teacher Characteristics in Schools Located in Urban and Rural Areas Other urban Yerevan areas Rural areas Share of Female Teachers 92% 89% 80% Share of Teachers ages 40 or more 73% 68% 64% Share of Teachers with tenure 15+ years 49% 51% 54% Share of Teachers trained in 2015 or after 24% 30% 27% Share of Teachers graduated before 2000 52% 49% 40% Share of Teachers graduated from the Armenian State Pedagogical University 34% 25% 25% Share of Teachers teaching in Middle Grades (5-9) 72% 73% 79% Share of Teachers with salaries over 100k (LCU) 49% 55% 55% Share of Teachers classified as Teaching Staff 96% 96% 94% 40