REPORT NO: ACS13261 MYANMAR EARLY GRADE READING ASSESSMENT 2014 RESULTS REPORT FOR THE JUNE 4, 2015 YANGON REGION GEDDR EAST ASIA AND PACIFIC Standard Disclaimer: This volume is a product of the staff of the International Bank for Reconstruction and Development/ The World Bank. The findings, interpretations, and conclusions expressed in this paper do not necessarily reflect the views of the Executive Directors of The World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. MYANMAR EARLY GRADE READING ASSESSMENT FOR THE YANGON REGION JUNE 4, 2015 GEDDR EAST ASIA AND PACIFIC PREFACE In the 14 years since the Republic of Myanmar committed to the Millennium Development Goals (MDGs), access to and enrollment in primary school has increased significantly. This is a notable achievement. The next important steps for the Government of the Union of Myanmar are to establish learning achievement baselines for its primary school students and to build an internal capacity in the Ministry of Education to monitor and report on learning improvements over time. This snapshot (from schools within the Department of Basic Education, DBE 3) of students’ current levels of reading ability in early grades—and factors associated with reading acquisition—is part of this effort to establish measurable learning standards and build institutional capacity to continuously monitor learning achievement. This document briefly summarizes the state of education in Myanmar, the information needed by stakeholders, and the way the Early Grade Reading Assessment (EGRA), administered in Yangon, Myanmar (DEB 3) addresses these needs. It reviews the most relevant education theories on reading acquisition in the early grades of primary education and how they inform the development of the Myanmar EGRA tool. This document describes the results of this study and the insights they offer about the characteristics of schools and teachers that can affect students’ success in learning to read. The final section proposes recommendations to the Ministry of Education on how to link EGRA results to current policy discussions on language and literacy in the early grades of primary education. TABLE OF CONTENTS Preface v Acknowledgments viii Abbreviations ix Executive Summary 1 1. Introduction 5 2. Overview of the Education Sector in Myanmar 7 2.1 Educational reforms in Myanmar 7 2.2 Education sector indicators 8 3. Theory and measurement of reading development 11 in the early grades 3.1 Theory of early grade reading 11 3.2 Measuring early grade reading skills 14 4. Survey Design and Implementation 17 4.1 Implementation arrangements 17 4.2 Guiding questions 17 4.3 Development of the EGRA tool in the Myanmar language 17 4.4 Sampling 18 4.5 Field work 19 5. Analysis and Results 21 5.1 Who are the students and teachers in Yangon? 21 Remoteness and wealth 21 Gender and class size 21 Language 21 Teacher demographics, class types, qualifications, training, 22 and years of experience Students’ prior enrollment in preschool 22 Student absenteeism 22 Teaching and learning materials 22 Homework, family involvement, and outside tutoring 23 Assessments 23 Activities and pedagogical approaches to teach reading 24 5.2 Can EGRA be adapted for the Myanmar Language? 24 Internal consistency in reliability of scales 24 Pair-wise correlations across subtasks 26 Predictive power of lower level skills to higher level skills 26 5.3 What does EGRA results indicate about early reading skills in Yangon? 27 Mean scores and percentage of zero scores per grade 27 International comparison 28 Curricular expectations and reading patterns of good and poor readers 28 Performance on literal versus inferential comprehension 31 5.4 What factors explain differences observed in early reading skills? 32 Overview of the methodology 32 Bivariate analysis 33 Multivariate modelling 34 Analysis of the variance decomposition 35 Coefficient analysis 35 Interpretation 35 6. Suggested recommendations for policy-makers interested 39 in improving early grade reading skills Bibliography 44 Annex 1: Overview of EGRA tools and stimuli 46 A1.1 Presentation of the eight subtasks in the Myanmar EGRA 46 tool for grades 1−3 A1.2 Actual subtask stimuli in the Myanmar EGRA tool 47 Annex 2: Detailed results for the reliability of the EGRA tool 51 A2.1 Internal consistency in reliability of scales 51 A2.2 Pair-wise correlations across subtasks 52 A2.3 Predictive power of lower level skills to higher level skills 53 Annex 3: Descriptive statistics for EGRA subtasks (Grades 1, 2, 3) 54 for section 5.3 Annex 4: Methodology and detailed results table for subsection 5.4 57 A4.1 Overview of the methodology 57 A4.2 Bivariate analysis 58 A4.3 Selecting variables for multiariate analysis 64 A4.4 Analysis of variance decomposition 67 A4.5 Coefficient analysis 68 ACKNOWLEDGMENTS The Myanmar EGRA study is the result of the government’s commitment to broaden its knowledge base on the state of reading in early grades. It also identifies the characteristics of students, teachers, and schools that can affect how well young students in Myanmar learn to read, which can serve as a guide in identifying the most effective interventions to help students acquire reading skills. This study was possible thanks to the support of officials and staff in the Ministry of Education, as well as the school principals, teachers, and the 1,681 students who enthusiastically participated in the survey. This report was written by a team led by Marie-Helene Cloutier (Economist, World Bank) and Souhila Messaoud-Galusi (Language and Literacy Specialist, World Bank), and included Fernando Cartwright (psychometrician, consultant), Stephanie de Silva (econometrician, consultant) and Myrna Machuca-Sierra (Education Specialist, World Bank). They were deftly guided by Luis Benveniste (Education Practice Manager, World Bank), James A. Stevens (Task Team Leader, World Bank), and Lars M. Sondergaard (Program Leader, World Bank). Save the Children (Myanmar) organized and managed the field work for the study under the guidance of Mar Mar Thwin (Education Program Manager, Save the Children). The authors thank Mar Mar and her dedicated team, the members of the EGRA Steering Committee and Technical Group, and—last, but not least—Dr. Thein Lwin and Daw Ohnmar Tin, the two dedicated local consultants without whom none of this would have been possible. The team would also like to extend its appreciation to the peer reviewers at the World Bank for their thoughtful guidance and comments: Cornelia Jesse (Senior Education Specialist), Nathalie Lahire (Senior Economist), and Marguerite Clarke (Senior Education Specialist). Finally, the authors are grateful to Prateek Tandon (Senior Economist), Abdoulaye Seck (Country Manager for Myanmar), and Xiaoyan Liang (Senior Education Specialist) at the World Bank for the additional benefits their comments provided. This first Early Grade Reading Assessment in Myanmar and the production of this report would not have been possible without the financial support of the Australian Government. ABBREVIATIONS CESR Comprehensive Education Sector Review CFWPM Correct Familiar Words Per Minute CIS Correctly Identified Sounds CIWPM Correct Invented Words Read Per Minute CLCQ Correct Listening Comprehension Questions CLNPM Correct Letter Names Identified Per Minute CLSPM Correct Letter Sounds Identified Per Minute CPR Continuous Personal Record CRCQ Correct Reading Comprehension Questions DBE 3 Department of Basic Education 3 DEPT Department of Education Planning and Training DFAT Department of Foreign Affairs and Trade DMER Department of Myanmar Education Research EFA Education for All National Action Plan, 2003–2015 EGRA Early Grade Reading Assessment GOM General Outcome Measures MDGs Millennium Development Goals OECD Organization for Economic Cooperation and Development ORF Oral Reading Fluency TEO Township Education Officer WSC Words Spelled Correctly “ACCESS, ENROLLMENT, AND NOW QUALITY STANDS AT THE CENTER OF THE EDUCATION POLICY DISCUSSION IN MYANMAR.” EXECUTIVE SUMMARY Access, enrollment, and now quality stands at the center The completion of the 2014 EGRA (data collection of the education policy discussion in Myanmar. As a and analysis) is a significant accomplishment in this measure of its commitment to the Millennium Development phase of Myanmar’s CESR initiative and on-going Goals (MDGs), the Republic of Myanmar has significantly and future research activities, building upon this increased access to and enrollment in primary school1—a work will continue to nourish thinking around the notable achievement. The Myanmar government continues issue of quality. The Ministry of Education continues its to work to improve the quality of education through its resolve to collect data on student reading levels via the (just Country Education Sector Reform (CESR) initiative. completed) 2015 EGRA to four more states/regions. This, and further research to come, will deepen and improve the Recently, the Myanmar Ministry of Education insights provided by this initial study. conducted a rapid education assessment, as part of the CESR initiative, which identified the lack of This report does not assume expert statistical knowledge reliable information on how well students are learning by the reader, although the annexes rigorously detail as a significant obstacle to improving the quality of the methodology and statistical analyses. It attempts education. It recognized that problems with learning in to present, in plain language, the work to adapt the many schools—whether in other developing countries or EGRA tool to the Myanmar language, administer it (and in high income countries—often begin during the first year its questionnaires) in the Yangon region, and analyze of a child’s schooling. Consequently, it supported an Early the collected data. It focuses on answering these four Grade Reading Assessment (EGRA) in 2014, adapted to questions. the Myanmar context and language, to measure how well primary school children are acquiring fundamental literacy 1. Can EGRA be adapted and used to skills (such as learning to read) as one of its priorities. In measure reading skills in the Myanmar addition to assessing reading ability (initially in Myanmar’s language? Yangon region), the survey should also provide the Ministry with crucial information needed to develop benchmarks to Yes, although some changes should be considered measure and compare students’ performance. that can further strengthen the usefulness of the tool. The EGRA tool comprises eight subtasks, which The EGRA project had four main objectives: design were specifically developed for the Myanmar language. The a valid EGRA tool for Myanmar, building capacity findings presented here suggest that the tool, as designed in assessment, defining core learning reading and tested in the Yangon region, worked for six of the eight standards for early primary grades, and identifying subtasks. Considerations should be given to redesigning factors associated with different levels of reading (or dropping) Subtask 2 (“initial sound identification”) and skills in order to guide policy design. The EGRA Subtask 3 (“letter sound knowledge”) in future versions of project successfully adapted and tested the EGRA tool in the EGRA tool. The data from the six successful subtasks, Myanmar,2 although it revealed two test areas that may however, still presents a reliable picture of early grade need to be redesigned, plus some resolvable data and reading skills. sampling issues. The data gained from EGRA offers clear directions for building domestic capacity to track, report, and analyze students’ progress in reading. The Ministry 2. What is the status of early grade reading of Education’s goal of defining core learning (reading) skills in a sample of schools in the Yangon standards for early primary grades is concretely supported region? by the EGRA findings. Despite some caveats, the study data identified potential explanatory factors for differences Despite performing relatively well, compared to other in EGRA scores among schools and students. And countries (in terms of zero scores on the oral reading tactically, the Ministry of Education will be able to use the fluency subtask), the Myanmar EGRA findings are EGRA findings to guide it in prioritizing future interventions worrying and indicate that too many students have that best help primary grade students improve their not learned to read or read well. Although the mean reading skills. scores on all the subtasks improved with grade level, the 1 Ministry of Education, Union of Myanmar, 2013, “Millennium Development Goal Report” (Nay Pyi Taw, Myanmar: Ministry of Education). 2 The 2014 EGRA study was conducted in the Yangon region, with 176 teachers and 1,681 students participating. 1 Overview of EGRA scores per grade Percent 90 77 80 70 60 50 37 40 30 27 20 10 12 9 10 0 Grade 1 Grade 2 Grade 3 % of students who could not read a single word % of students who could not answer a single question correctly about a text they just read THE MINISTRY OF EDUCATION’S “ substantial proportion of students who could not answer one item correctly, and the low mean scores on various GOAL OF DEFINING CORE subtasks by the end of Grades 2 and 3, remain worrying. For example, 10 percent of second graders cannot read LEARNING (READING) STANDARDS a single word. And, of the second graders who can read, 27 percent did not understand what they had just read FOR EARLY PRIMARY GRADES IS (see figure). Furthermore, analyzing the proportion and performance of good and poor readers indicates that CONCRETELY SUPPORTED BY students are not meeting curriculum expectations in all three grades. Finally, the reading comprehension results THE EGRA FINDINGS.” suggest that primary students in Myanmar are capable of understanding a text when it is read aloud to them, but find it more difficult to understand the inferences in the text when they read it themselves. 3. Is it possible to identify factors that affect students’ performance and explain the differences in EGRA scores? If so, what are they? The results of this analysis not only indicate that progress in learning to read may be affected by factors outside the school but also that some schools are better than others at offsetting or ameliorating these effects. Based on the data, a considerable proportion of the differences in students’ early reading skills across schools is determined by non-school factors, such as the socio–economic status of students, help with homework, etc.3 These unfortunately have little to do with educational practice and cannot easily be addressed by 3 As shown by the differences in average EGRA scores between schools. 2 schools, but this does not mean that such factors cannot complementary research and analysis. This can be overcome. Indeed, the Myanmar data indicates that the include improving the EGRA tool, research protocols, and effects of student characteristics, despite being relatively reporting efforts. Also recommended is building more consistent, vary significantly from school to school. Some capacity within the Ministry to carry out surveys, analyze schools are able to provide environments that offset or findings, expand coverage of the research, adapt EGRA ameliorate the effects of gender, family background, and and its administration to more ethnic languages, broadly home educational support. share EGRA findings, and add a math component to the assessment. This study does suggest that Myanmar’s Ministry of Education may do well to prioritize a few specific Second, the Ministry of Education could use the interventions, targeting characteristics of school findings to begin establishing measurable standards environments which are statistically correlated to assess reading in the early primary grades. Much with differences in test scores while controlling for value can be added—and knowledge increased—by students’ demographics and/or initial skill levels. developing short diagnostic tests and training teachers to Interventions with the potential to make a difference include measure and report progress toward the standard. helping teachers, students, and even parents or other adults use the Myanmar exercise book more effectively (by Finally, the Ministry of Education could consider developing and disseminating a set of best practices for using the findings to help design and select specific it), addressing the need for teachers to take on other jobs, interventions aimed at improving reading outcomes. and investing in school libraries or book corners.4 Further For example, the Ministry of Education and other insights into these potential interventions could be obtained government agencies could capitalize on EGRA for a from the analysis of the second round of EGRA in 2015 and number of initiatives: in the context of future studies such as impact evaluations of such interventions. • Launch a public awareness campaign to promote reading, for example, that encourages parents (or But the results of this 2014 Myanmar EGRA are not adults) to read to children every day and gives authors conclusive in many areas, so further research is incentives to write more children’s books. needed. One particularly puzzling—counter-intuitive— • Make more reading materials available to students (to finding was the inverse relationship between the use complement the basic textbooks) by helping schools of both summative (e.g., use of chapter- or term-end add or enlarge a library, or set up reading corners in tests) and formative (e.g., feedback to teachers on classrooms via education grant. effectiveness of their instruction) assessments. The use of assessments is expected to help teachers pinpoint where • Improve the quality of instruction by helping teachers (and which) students need help and/or to fine-tune their make better use of Myanmar exercise books, instruction methods and so should be positively related to developing best practices for teaching with textbooks, performance. The negative correlation between the use and establishing pre- and in-service programs for of assessment and EGRA scores should be investigated teachers on early reading. further. In addition, the study was unable to conclusively • Start the process of defining standards (or distinguish the effectiveness of specific educational competencies) that can be linked to schools’ curricula, techniques used in the schools. beginning with reading performance standards for students in early grades. 4. What are the recommendations for • Integrate standards into pre- and in-service teacher policy-makers who are interested in training programs. improving early grade reading skills? • Improve the monitoring and evaluation system for Potential actions for the Ministry of Education, policy- how well schools (and students) meet performance makers, and development partners to consider can be standards and compare to other schools. organized into three main categories. All of the above initiatives could also be rigorously evaluated First, the Ministry of Education could consider using impact evaluation methodology combined to pre- and expanding the research and linking it to post-EGRA. 4 The potential of these interventions is suggested by the observed statistically significant correlation between the related variables and the EGRA test scores in multivariate hierarchical model (see subsection 5.4). 3 WHEN CHILDREN DO “ NOT LEARN TO LEARN TO READ AND UNDERSTAND A SIMPLE TEXT IN THE EARLY GRADES, THEY ARE MORE LIKELY TO FAIL IN SCHOOL, REPEAT MORE GRADES, AND DROP OUT OF SCHOOL.” 1. INTRODUCTION Recently, the Myanmar Ministry of Education This report presents the work to adapt the EGRA to conducted a rapid education assessment, as part of the Myanmar language and to administer it (and its its Country Education Sector Reform (CESR) initiative, questionnaires) in the Yangon region. The results of this that identified the lack of reliable information on exercise were analyzed to assess how valid and reliable learning outcomes as a significant obstacle. In this adapted tool is for the Myanmar language. Other aims addition, education authorities in Myanmar are working in of the analysis were to define core learning standards in a fragmented system, where regionally-based ethnic and early primary grades, and identify potential explanatory language groups are not cooperating (or not cooperating factors for differences in EGRA scores among schools. The effectively) with the Ministry of Education or other regional specific information from EGRA can help guide the Ministry counterparts. Government authorities and donor partners of Education in prioritizing future interventions that best help are attempting to promote national reconciliation and students improve their reading skills in early grades, as well collaboration in many fields, including education. They as support the development of measurable, comparable are proposing that these different regional groups work benchmarks for students’ progress. together to develop standards for reading in the early primary grades—and the means to measure them—as a The report is organized into six sections, including vehicle for collaboration in the education system. the introduction. Section 2 briefly reviews the reforms and goals of the education sector in Myanmar and the major Problems with learning in many schools—whether in challenges that helped define the research questions for developing countries or in high income countries— the EGRA. Section 3 describes the most relevant education often begin during the first year of a child’s schooling. theories about learning to read in the early grades and the When children do not learn to learn to read and understand way these are reflected in the design of the EGRA. Section a simple text in the early grades, they are more likely to 4 outlines the design and implementation of the study fail in school, repeat more grades, and drop out of school. and tools, and how they took into account the information Ultimately, it reduces their opportunities in life and increases needs of education stakeholders and the specificities of the incidence of social ills. Yet few developing countries give the language and education sector in Myanmar. Section priority to measuring how well children acquire reading skills 5 presents the methodology and results of the study for in the early grades. The Early Grade Reading Assessment non-experts, with the methodology and statistical analyses (EGRA) measures a child’s progress in learning to read. detailed in annex 5. Section 6 concludes with a summary of While the general framework and approach of the EGRA is recommendations. similar across countries, its specific components can vary. 5 “RECENT EDUCATION REFORMS HAVE SET THE LONG-TERM VISION, DIRECTION, AND GOALS FOR THE COUNTRY’S DEVELOPMENT.” 2. OVERVIEW OF THE EDUCATION SECTOR IN MYANMAR In the last decade, the government of the Union of Goals (MGDs) for education, the action plan incorporates Myanmar has made great efforts to improve access to the six EFA goals: basic education, but gaps in knowledge and strategies remain around issues of learning. Recent education • Expand early childhood care and education. reforms have set the long-term vision, direction, and • Provide free and compulsory primary education. goals for the country’s development. The government has also defined specific sector development plans (for • Promote learning and life skills for young people and immediate deployment) and key areas for immediate adults. attention. Planning efforts recognize that some of these • Increase adult literacy. areas represent new challenges and questions. Solutions and answers require more education research, which also • Achieve gender parity. need greater technical and financial support. One of these • Improve the quality of education. questions—the focus here—asks about the current state of literacy, or how well students are reading, in early grades in As it started to implement reforms, the Ministry Myanmar and the factors associated with it. of Education recognized that it needed a deeper understanding of the current status of education, 2.1 Educational reforms in Myanmar especially students’ performance.8 In February 2012, it launched a new initiative, the CESR. The CESR is a The government has made access to quality participatory process led by the Ministry of Education, education at all levels a central point of education which also relies on contributions from a wide range of reform discussions in Myanmar. The framework for education stakeholders, including other ministries and Myanmar’s most recent plans for educational reform is development partners (namely, the United Nations and detailed in its 30-year Long-Term Education Development other multi-lateral, bilateral, non-governmental, and civil Plan, 2013-2031.5 The 10 objectives of the plan assign society organizations). Although the government had to the education sector the important role of leading the provisionally identified a number of priority areas,9 the country’s development and modernization. Issues of access CESR’s mission started with improving the knowledge to quality education occupy most of the reform agenda, base of the strengths and challenges of the Myanmar with stated goals of improving basic education,6 increasing education system. opportunities for pre-vocational and vocational education, and expanding non-formal education at all levels. The initial rapid assessment and first phase of the review establishes urgent priorities and issues, One key target in Myanmar’s plan is to achieve specifically a quantitative analysis of the access, universal basic education by 2031. Furthermore, to equity, quality, efficiency, and effectiveness of the ensure progress toward the plan’s goals, it adopted the school system in Myanmar. Its results provided a Education for All (EFA) National Action Plan, 2003-2015.7 baseline for the in-depth education analysis and phase 2 In addition to the objectives of the Millennium Development of the CESR, and also identified sources of additional data. 5 Ministry of Education, Union of Myanmar, 2008, “Country Report: Myanmar Education Development Strategy Focusing on Inclusive Education” (Nay Pyi Taw, Myanmar: Ministry of Education). http://www.ibe.unesco.org/National_Reports/ICE_2008/myanmar_NR08.pdf 6 Basic education encompasses primary school (Grades 1-5), lower secondary school (Grades 6-9), and upper secondary school (Grades 10-11). Ministry of Education, Union of Myanmar, 2004, “Development of Education in Myanmar 2004” (Nay Pyi Taw, Myanmar: Ministry of Education). 7 Ministry of Education, Union of Myanmar, 2012, “Education for All: Access to and Quality of Education in Myanmar,” Conference on Development Policy Options with Special Reference to Education and Health in Myanmar, 13-16 February 2012, Nay Pyi Taw, Myanmar. http://yangon.sites. unicnetwork.org/files/2013/05/Education-for-All-in-Myanmar-Final-2012-FEB-2.pdf. 8 CESR.org, 2012, “Terms of Reference for the Myanmar Comprehensive Education Sector Review, 2012,” http://www.cesrmm.org/index.php/en/. 9 These focus areas aim to review legislation and enact new laws, reorganize departments in accordance with decentralization, restructure the basic education sector and review the basic education curriculum, improve the quality of teacher education and strengthen the capacity of education personnel, and reform quality assurance systems that assess students. 7 Among the needs identified, the lack of reliable data on Given the low primary completion rate (54 percent), students’ performance was a particular limitation for phase data on students’ achievements in early grades is 2 and the education sector plan for 2014-2016 (phase 3). crucial for decision-makers in the education sector. In Consequently, the CESR Steering Committee expressed Myanmar, about 8.5 million students—with roughly 5 million great interest in this study in order to collect valid and in primary school—are receiving basic education. The latest representative data on the state of early-grade reading in national statistics point to a net primary enrollment rate the Yangon region of Myanmar. of 88 percent.12 Among them, 75 percent of first graders reach the fifth (and last) primary grade, but only 54 percent complete primary education.13 Further analysis reveals 2.2 Education sector indicators that most students leave school in the first two years of Reading, writing, and mathematics are the most primary schooling.14 Because children in the early grades important skills for primary students to learn, and are likely to quit before completing their basic education, it represent key indicators and standards of quality is vital to determine the actual level of education and skills for primary schools and teachers. The international they achieve. But, such data is rarely available to decision- standard for classification of education defines primary makers. education as providing students with fundamental skills The availability of data on students in early grades— in reading, writing, and mathematics.10 In Grades 1-3, in particular on learning—is even more important, Myanmar teachers are advised to devote 25 percent of given that the low completion rate may be strongly classroom time to oral and written language instruction, influenced by the quality of schools. The information close to the OECD average of 30 percent.11 In Myanmar above suggests that one of the major obstacles to and in other countries across the world, available statistics universal basic education in Myanmar is indeed the drop- indicate that reading is the subject that students in early out rate that occurs early in the educational process. grades spend the most time studying. This phenomenon is most often viewed as an issue of How well children learn to read in Grades 1-3 has a access and enrollment,15 but student retention is also wide reaching and critical impact on their education: strongly influenced by school quality.16 Robust research early reading skills are correlated with children’s has shown that a student is much less likely to remain in overall learning outcomes, including other subjects. school if that school is of lower quality.17 Because such little Thus, learning to read is not just an indicator of how information exists on school quality, it can only be assumed effectively schools teach these fundamental skills but can that Myanmar’s low completion rate18 results in part from be a proxy for the general quality of education that primary school students having difficulty learning grade- children receive. appropriate skills at their school. 10 UNESCO Institute for Statistics, 2011, International Standard for Classification of Education (ISCED). http://www.uis.unesco.org/Education/ Pages/international-standard-classification-of-education.aspx. 11 Ministry of Education, Union of Myanmar, 1999, “Instruction and Guide for Teachers,” in Myanmar Language Textbook (Nay Pyi Taw, Myanmar: Ministry of Education); and Japanese International Cooperation Agency, 2013, “Data Collection Survey on Education Sector in Myanmar, February 2013” (Tokyo: JICA). 12 Net primary enrollment rate is defined as the percentage of students aged 5−9 years who attend primary school. From World Bank staff calculations based on the Integrated Households Living Conditions Survey (IHLCA) 2009-2010. UNDP, 2011, “Integrated Households Living Conditions Survey in Myanmar (2009-2010): Poverty Profile” (Yangon, Myanmar: IHLCA Project Technical Unit). 13 Ministry of National Planning and Economic Development, and the Ministry of Health, Union of Myanmar, 2011, “Multiple Indicator Cluster Survey 2009-2010: Final Report” (Nay Pyi Taw, Myanmar: Ministry of National Planning and Economic Development and the Ministry of Health), http://www.childinfo.org/files/MICS3_Myanmar_FinalReport.pdf. 14 M. Hayden and R. Martin, 2013, “Recovery of the Education System in Myanmar,” Journal of International and Comparative Education 2 (2): 47-57. 15 Ministry of Education, “Millennium Development Goal Report 2013.” 16 S.R. Khandker, V. Lavy, and D. Filmer, 1994, “Schooling and Cognitive Achievements of Children in Morocco: Can the Government Improve Outcomes?” World Bank Discussion Paper, no. 264 (Washington, DC: World Bank). 17 E.A. Hanushek and V. Lavy, 1994, “School Quality, Achievement Bias, and Dropout Behavior in Egypt,” Living Standards Measurement Study (LSMS) Working Paper, no. LSM 107 (Washington, DC: World Bank), http://documents.worldbank.org/curated/en/1994/12/697806/school- quality-achievement-bias-dropout-behavior-egypt. 18 C.B. Lloyd, B. S. Mensch, and W.S. Clark, 2000, “The Effects of Primary School Quality on School Dropout among Kenyan Girls and Boys,” Comparative Education Review 44 (2): 113-47 8 The main challenge is that Myanmar does not have Completion of basic education and access to tertiary a way of reliably measuring and tracking students’ education depends on passing the Basic Education progress in reading, writing, and mathematics. High School matriculation examination at the end of Currently Myanmar has no regular assessments of student upper secondary school19 and the success rate on learning at the end of each cycle. There is, therefore, no the matriculation examination and the tests leading information about whether students are mastering the to it varies widely across Myanmar.20 In primary school, most important skills (e.g., reading and math) at the end of despite a promotion rate of 93.3 percent from Grade 1 to primary or middle school. The only regularly administered Grade 5,21 students’ progress has been regularly assessed examination is the Basic Education High School exam at since the reform of 2008. Starting in Grade 4, schools the end of high school. But it is not clear that these exam review scores on chapter-end tests, in combination with the results are comparable year to year. Comprehensive Personal Record (CPR), to determine which students can progress to the next grade.22 In Grades 1 and 2, however, the students are given no test or examination, a practice that is sometimes assumed to reduce student absenteeism and drop-out rates.23 19 Ministry of Education, “Development of Education in Myanmar 2004.” 20 Hayden and Martin, “Recovery of the Education System in Myanmar.” 21 Ministry of National Planning and Economic Development, and the Ministry of Health, “Multiple Indicator Cluster Survey, 2009−2010.” 22 The CPR for primary school tracks nine components. The student must: 1) have 75 percent school attendance record; 2) regularly take the chapter tests; 3) obey school rules and regulations with no history of social crimes; 4) fulfill obligations to the school, teachers, parents, and the community, and help take care of younger students; 5) Participate in making the school yard verdant and green by helping grow trees and plants; 6) assist in parent’s livelihood; 7) participate in sports and physical activities; 8) participate in arts education, such as literary activities, music, singing, dancing, and painting; and 9) keep neat and tidy. The components are divided into six areas, which combined have a highest potential score of 100. In addition, students can earn up to a mark of 100 for each subject in the chapter test. To be promoted to the next level, students must obtain at least 40 percent in both academic and school activities. 23 Ministry of Education, “Development of Education in Myanmar.” 9 THE RATE OF LEARNING “ TO READ IN THE PRIMARY GRADES IS A STRONG DETERMINANT OF LATER LITERACY: GOOD READERS BECOME MORE FLUENT, WHILE POOR READERS GET WORSE OVER TIME.” 3. THEORY AND MEASUREMENT OF READING DEVELOPMENT IN THE EARLY GRADES 3.1 Theory of early grade reading 1. Demonstrate the ability to hear and manipulate the sounds of the language (phonemic awareness). Research shows that students who experience 2. Be able to recognize and sound out letters (and difficulties learning to read in the early grades never letter groups) of the alphabet (alphabetic knowledge really catch up with their peers.24 The rate of learning and awareness), and blend the sounds into words to read in the primary grades is a strong determinant of (phonological decoding). later literacy: good readers become more fluent, while poor readers get worse over time.25 Longitudinal data 3. Accurately and fluently identify and connect written on reading in early grades in the United States indicates words into text without effort (reading fluency). that gaps in achievement in early grades tend to be 4. Understand both oral language and written texts—when magnified over time. Children not only lose out on learning text is read out loud to students or they read written text opportunities early but they are set on a trajectory of on a page (comprehension strategies).27 underperformance that worsens over time. Figure 1 (reproduced from a 1998 study in the United States26) plots Phonemic awareness is typically defined as the the trajectories of children with varying literacy skills. At the ability to produce and manipulate the sounds of a end of the first grade, good readers begin to separate from language (phonemes). Phonemic awareness is believed children having more difficulty and the gap continues to to help with reading. Before young children learn to read, widen through to the end of second grade. they are not “aware” that words can be broken down into their component sounds (or are capable of doing so). The scientific literature identified a fixed number Thus, the ability to match letters and sounds (referred to as of skills and strategies deemed fundamental to grapheme-phoneme correspondence) depends on a child’s becoming a good reader. In alphabetic languages, the ability to become aware of phonemes. This, in turn, helps initial stages of learning to read include four basic skills them build language decoding skills.28 (described in more detail as follows): 24 K.E. Stanovich, 1986, “Matthew Effects in Reading: Some Consequences of Individual Differences in the Acquisition of Literacy,” Reading Research Quarterly 21: 360-406. 25 The study called this phenomenon the “Matthew effect,” referring to the observation in the Gospel of Matthew (in the Christian Bible) that the rich get richer and the poor get poorer. 26 R.H. Good, D.H. Simmons, and S. Smith, 1998, “Effective Academic Intervention in the United States: Evaluating and Enhancing the Acquisition of Early Reading Skills,” School Psychology Review 27: 45-56. 27 G.H. Good, D.C. Simmons, and E.J. Kame’enui, 2001, “The Importance and Decision-Making Utility of a Continuum of Fluency-Based Indicators of Foundational Reading Skills for Third-Grade High-Stakes Outcomes,” Scientific Studies of Reading 5: 257-88; M.J. Adams, 1990, Beginning to Read: Thinking and Learning about Print (Cambridge, MA, USA: MIT Press); National Reading Panel, 2000, Teaching Children to Read: An Evidence-Based Assessment of the Scientific Research Literature on Reading and Its Implications for Reading Instruction—Reports of the Subgroups (Bethesda, MD, USA: National Institute of Child Health and Human Development); and D.C. Simmons and E.J. Kame’enui, eds., 1998, What Reading Research Tells Us about Children with Diverse Learning Needs: Bases and Basics (Mahwah, NJ, USA: Lawrence Erlbaum Associates, Inc.) 28 L. Sprenger-Charolles, P. Colé, and W. Serniclaes, 2006, “Reading Acquisition and Developmental Dyslexia” in Essays in Developmental Psychology (Hove, UK, and New York: Psychology Press. 11 Figure 1: Early grade reading progression Source: Good, Simons, and Smith (1998). “GRAMMATICAL KNOWLEDGE, Phonemic awareness and knowledge of the alphabet must be acquired together to learn to read. They are VOCABULARY, AND WORKING reliable predictors and diagnostic measures of reading readiness29 because young children learning to read cannot MEMORY (LINGUISTIC connect the letters making up a written word with the sounds underlying the spoken word unless 1) they are FACTORS) ALL CONTRIBUTE TO consciously aware of both and 2) have the intent to learn the relationship between the two (the alphabetic principle). COMPREHENSION.” Thus, if children know the letters and know there is some relation between the letters and the spoken word, but they do not know the sounds underlying the spoken word, then they will not be able to figure out the relationship between oral and written words.30 Alphabetic knowledge is also crucial because it emphasizes the interconnection between oral and written language. It helps students understand that when they read, the letters in written words represent the sounds in spoken words. Thus, in the same way, when they write, the sounds in spoken words can be turned into written words. The purpose of alphabetic knowledge is to help children understand that written words are “sounded out,” and not memorized.31 Knowledge that spoken words are made up of sounds, and written words are made up of letters is not sufficient for developing good decoding skills, or the ability to correctly pronounce words (in the child’s language). 29 D.L. Share, 1999, “Phonological Recoding and Orthographic Learning: A Direct Test of the Self-Teaching Hypothesis,” Journal of Experimental Child Psychology 72: 95-129. 30 W.A. Hoover, 2002, “The Importance of Phonemic Awareness in Learning to Read,” SEDL Letter, Putting Reading First 14 (3), http://www.sedl. org/pubs/sedl-letter/v14n03/3.html. 31 S. Wren et al., 2000, “Cognitive Elements of Reading” in Cognitive Foundations of Learning to Read: A Framework (Austin, TX, USA: SEDL), http://www.sedl.org/reading/framework/elements.html. 12 Decoding, the ability to translate text into speech, occurs by reading itself). However, these skills alone are is considered a powerful strategy that helps not sufficient to support and further develop good reading beginning readers learn to read words effortlessly comprehension. When a child does not know the meaning and automatically. It involves matching letters to sounds of words and sentences in a text, it becomes a bottleneck to form syllables and words until this process becomes and slows down progress in reading comprehension. automatic. The importance of decoding in early reading Grammatical knowledge, vocabulary, and working memory has been formalized in the self-teaching theory.32 This (linguistic factors) all contribute to comprehension, and states that each successful decoding of a new word gives their contribution increases with age.37 An essential aspect children an opportunity to gather word-specific information of language comprehension hinges on the ability to draw that makes it easier for them to efficiently and automatically inferences and appreciate implications; thus it is important recognize words. Decoding skills are particularly important to understand both the explicit and implicit messages and effective for beginning readers, who are the most likely contained in language. to meet unknown or new words in the texts they read and need to develop fluent and accurate readings. The ultimate goal of learning to read is to understand the meaning of text. Thus, a large body of research has Reading fluency is an important part of being a examined the particular strengths and difficulties of students proficient reader.33 When reading aloud, fluent readers with different levels of comprehension skills to establish that sound natural and read at a conversational pace. Fluent the quality of the mental representation built by the reader readers are accurate and quick, and use proper expression, (i.e., thoughts about things, events, and surroundings that while non-fluent readers may read slowly, make errors, help people create meaning or significance for something) and sound monotonic or unnatural.34 Being able to matters significantly. Deep comprehension relies on the recognize words easily (word identification fluency, a task construction of a situation model, based on the reader’s that measures automatic word recognition) has been ability to understand literal information, as well as the ability repeatedly shown to be an excellent index of early reading to draw inferences that bridge elements in the text and development and an early precursor of fluent text reading, support its coherence. which is necessary for good reading comprehension.35 Among all possible inferences that can be drawn Reading and listening comprehension is composed from a text, both text-based and non-text-based of two equally important components: decoding inferences are most necessary. The reader must make (or translating text into speech) and language text-based inferences in order to make a text coherent, comprehension (or the ability to understand spoken but often the reader uses non-text-based inferences, such language). Each component is necessary, but is not as causal inferences, to connect actions in a story. For sufficient on its own to ensure that children understand example, in this text—“Michael took the drink out of the fully the text they are reading. (Multiple studies have bag. The orange juice was very refreshing.”—the reader demonstrated that all struggling readers have difficulty makes text-based inferences that the drink in the bag (first with either language comprehension or decoding, or both.) sentence) was an orange juice (second sentence). In this However, what these component skills of reading contribute example—“Martha and Julius are playing in the sand and to reading comprehension differs, depending on the swimming.”—the reader must make a non-text-based reader’s expertise. inference that they are at the beach. The reader uses knowledge about the outside world to make the connection When children begin to learn to read, automatic word that a place with sand and water to swim in is the beach. In recognition, decoding and phonological awareness36 addition to processing text literally and through inference, all help them develop efficient word-identification children with good comprehension skills are sensitive to skills and lead to fluent reading (which in large part story structure and are able to monitor comprehension. 32 D.L. Share and A.F. Jorm, 1987, “Segmental Analysis: Co-requisite to Reading, Vital for Self-Teaching, Requiring Phonological Memory,” Cahiers de Psychologie Cognitive/European Bulletin of Cognitive Psychology 7: 509-513. 33 S.L. Deno, 2003, “Curriculum-Based Measures: Development and Perspectives,” Assessment for Effective Intervention 28 (3-4): 3-12. 34 Sprenger-Charolles, Colé, and Serniclaes, “Reading Acquisition and Developmental Dyslexia”; and L. Sprenger-Charolles et al., 2003, “Development of Phonological and Orthographic Processing in Reading Aloud, in Silent Reading, and in Spelling: A Four-Year Longitudinal Study,” Journal of Experimental Child Psychology 84 (3): 194-217. 35 S.L. Deno, P.K. Mirkinand, and B. Chiang, 1982, “Identifying Valid Measures of Reading,” Exceptional Children 49: 36-45. 36 Phonological awareness is a general term for meta-linguistic awareness of any of the phonological characteristics of language, including phonemic units, syllables, rimes, and words. This is different from the term phonemic awareness, which refers to the ability to consciously manipulate language at the level of phonemes. Hoover, “The Importance of Phonemic Awareness.” 37 V. Muter et al., 2004, “Phonemes, Rimes and Language Skills as Foundations of Early Reading Development: Evidence from a Longitudinal Study,” Developmental Psychology 40: 665-81. 13 Writing and spelling skills also play a role in stakes standard-based measures (that can determine future developing reading skills in early grades. Writing course), GOM has been recognized as a useful tool in preschool and kindergarten has significant predictive for diagnosing reading difficulties in early grades. It relations with later reading ability;38 there is also a direct incorporates a continuum of fluency measures for causal relationship to reading from spelling. In the early reading, which gives teachers essential feedback on the stages of learning to read, children are still mastering the effectiveness of their instruction. relations between letters and sounds, which explains why multiple studies show that spelling contributes greatly to Over the last 25 years, the GOM framework has been decoding. Additional evidence suggests that writing also used in a variety of ways, such as: provides the opportunity for children to build word-specific orthographic (letters and spelling) knowledge. In writing, • establishing norms for identifying students who need they must select the correct word-specific grapheme (the special-education services, letter or letters that “spell” a sound in a word) associated • evaluating the effectiveness of educational programs, with each phoneme (sound in a word) and thereby gain a deeper exposure to words.39 • reintegrating students with disabilities into general- education classrooms, In languages with an alphabetic script—like the • monitoring students’ progress and helping teachers plan Myanmar language—research has identified certain instructions in general-education classrooms, and basic mechanisms. If these are present, they help students learn to read skillfully; if absent, they make it • identifying potential candidates for special-education harder for students to read.40 These indicators, which can evaluation using a dual-discrepancy model.43 predict reading success in early grades, include phonemic awareness and letter knowledge—both necessary to The GOM framework also helped develop the EGRA, establish good decoding skills. Decoding skills are a sine a system diagnosis tool to advance the efforts qua non and a self-teaching mechanism that enables of developing countries to improve the quality of students to automatically recognize words and read text reading outcomes in the early grades of primary fluently. Linguistic abilities, along with fluent word- and education. The structure of the EGRA tool focuses on text-reading abilities are key components of reading assessing the degree of automaticity44 students have comprehension. developed, on average. In other words, it shows how accurately and quickly students are learning basic reading skills in the first grades of primary school. Applied in more 3.2 Measuring early grade reading skills than 40 countries and in 70 languages, the EGRA tool has effectively provided useful baseline data to countries Among different research tools used to evaluate that are planning preventive measures for early-reading performance and skills, General Outcome Measures difficulties, establishing benchmarks to track development (GOM), a tool created in special-needs research, is targets, and measuring the impact of reforms and literacy widely used to improve educational decision-making programs.45 in a variety of contexts.41 GOM was created to be a simple, reliable, and valid set of measurement procedures The full set of the EGRA test includes nine modules that teachers could use frequently and repeatedly to (or nine subtasks); eight are used in this study.46 These measure their students’ progress in the basic skills of are the eight subtasks used for this study: reading, spelling, and writing.42 As a complement to high- 38 D. Shahar-Yames and D.L. Share, 2008, “Spelling as a Self-Teaching Mechanism in Orthographic Learning,” Journal of Research in Reading 31: 22-39. 39 Ibid. 40 M.S. Seidenberg, 2013, “The Science of Reading and Its Educational Implications,” Language Learning and Development 9: 331-60. 41 S.L. Deno, 2003, “Curriculum-Based Measures: Development and Perspectives,” Assessment for Effective Intervention 28 (3−4): 3-12. 42 Ibid. 43 P.M. Stecker, L.S. Fuchs, and D. Fuchs, 2005, “Using Curriculum-Based Measurement to Improve Student Achievement: Review of Research,” Psychology in the Schools 42: 795-819. 44 RTI International, 2009, Early Grade Reading Assessment Toolkit, prepared for the World Bank Office of Human Development (Research Triangle Park, NC, USA: RTI International; and Washington, DC: USAID). https://www.eddataglobal.org/documents/index. cfm?fuseaction=pubDetail&id=149. 45 USAID and Eddata, “Early Grade Reading,” https://www.eddataglobal.org/reading/. 46 The subtask that was not used in this study is “orientation to print.” It measures students’ ability to orient themselves in the text, direct the reading, and understand where a new line begins. 14 1. Letter name knowledge Table 1 below describes each subtask in the full set of the EGRA tool used in Myanmar, the skill associated with it, 2. Identification of initial sounds in words (a measure of and its measure and indicator. (See Annex 1 for how the phonemic awareness) eight subtasks are presented in the EGRA tool, plus the 3. Letter sound knowledge actual subtask stimuli.) It is important to note that the use of the EGRA tool and EGRA-like assessments do not require 4. Familiar word reading (a measure of automaticity in word that countries administer the full set of subtasks. Each identification) country should determine which subtasks are most relevant 5. Invented word reading (a measure of decoding) for their particular purpose and context. 6. Oral reading fluency with comprehension 7. Listening comprehension 8. Dictation Table 1: EGRA instrument structure and early skills tested in Myanmar Ability demonstrated by students Subtask Skill Measure and indicator mastery of skill Provide the name of upper- and 1. Letter name The correct letter names Letter recognition lower-case letters distributed in knowledge identified per minute (CLNPM) random order. Segment words into 2-5 phonemes. 2. Identification of Phonemic The number of correctly Identify words with different initial sounds awareness identified sounds (CIS) beginning or ending phonemes. Provide the sound of upper- and 3. Letter sound The correct letter sounds Phonics lowercase letters distributed in knowledge identified per minute (CLSPM) random order. 4. Familiar word Read simple and common one- and The correct familiar words read Word reading reading two-syllable words. per minute (CFWPM) Make grapheme-phoneme 5. Invented word correspondences (GPCs) by reading The correct invented words read Alphabetic principle decoding simple invented words to test per minute (CIWPM) decoding skills. The correct words read orally Read a text with little effort and at a per minute in a narrative Oral reading fluency sufficient rate. passage (ORF) 6. Oral reading fluency with Respond correctly to different types The percentage of correct comprehension Reading of questions, including literal and answers to reading comprehension inferential questions about the text comprehension questions they have read. (CRCQ) Respond correctly to different types The percentage of correct 7. Listening Listening of questions, including literal and answers to listening comprehension comprehension inferential questions about the text comprehension questions read to them. (CLCQ) The percentage of overall early writing skills (spelling and Alphabetic Write, spell, and use grammar 8. Dictation basic conventions)—words principle properly in a dictation exercise. spelled correctly (WSC)—with a weighted score. Note: Adapted by the authors, based on RTI International, 2009, Early Grade Reading Assessment Toolkit, prepared for the World Bank Office of Human Development (Research Triangle Park, NC, USA: RTI International; and Washington, DC: USAID). 15 MYANMAR IS COMMITTED “ TO PROMOTING GENDER EQUALITY AND HAS TARGETED THE ELIMINATION OF GENDER DISPARITY IN ALL LEVELS OF EDUCATION BY 2015.” 4. SURVEY DESIGN AND IMPLEMENTATION 4.1 Implementation arrangements 1. Is reading performance different with gender? Myanmar is committed to promoting gender equality Implementation of the EGRA in Myanmar was the and has targeted the elimination of gender disparity result of a collaborative effort between the Ministry of in all levels of education by 2015. Available rates of Education, the World Bank, Australia (Department of enrollment and literacy among young people 15-24 Foreign Affairs and Trade, DFAT), Save the Children, years old indicate that there is no gender disparity in and representatives of DBE 3 and teacher training Myanmar. Girls and boys are expected to be treated colleges. The Ministry of Education led the work financed equally within the education system; therefore, no by the World Bank and DFAT. The World Bank provided significant difference in reading performance should be complementary technical assistance, Save the Children observed between boys and girls in DBE 3. was contracted to carry out the field work and data entry, representatives of DBE 3 supplied data for the sampling 2. What effect does a student’s language background and facilitated logistics, and teacher training colleges have on reading performance? The Myanmar assigned teacher trainees to be EGRA enumerators. language is the official language of instruction in classrooms and in all textbooks for Grades 1-3, To oversee the whole process, the Ministry of including the reading texts. However, as many as 100 Education appointed a steering committee, languages are spoken by the people of Myanmar. As in and to work out the technical details and the many multilingual countries in Southeast Asia, students implementation, the EGRA Technical Working Group entering primary schools are not always familiar with was formed. The Steering Committee comprised a the official language used in schools and textbooks. representative from each Department of Basic Education, The Steering Committee wished to investigate whether the Department of Education Planning and Training students speaking mostly another language are reading (DEPT), the Department of Myanmar Education Research below the level of their peers who are fluent in the Bureau (DMER), and the CESR Task Team. The EGRA Myanmar language. Technical Working Group involved the participation of Myanmar language scholars and the representatives from 3. Does a school’s remote location affect reading education colleges. Finally, the organizations advising performance? The study looked at whether the the Steering Committee—DFAT (AusAID at the time), the remoteness of a school, as measured by the official World Bank, and Save the Children—were invited to send administrative isolation classification, affected student a representative in a funding and implementing capacity. literacy. In other words, do hard-to-reach schools These organizations gave progress reports at each key perform less well than schools within easy commuting step in the study during regularly held steering committee distance of the Township Education Office (TEO)? meetings. Education experts from the Steering Committee and the EGRA Technical Working Group suggested that 4. Do other student characteristics and teachers’ the EGRA framework be used to measure early grade behaviors affect reading performance? The Steering reading skills. Committee suggested that the study identify and investigate whether other factors influence students’ reading performance. 4.2 Guiding questions Prior to designing survey and assessment tools, the 4.3 Development of the EGRA tool in the Steering Committee defined the scope and research Myanmar language questions to guide the study, which focused on the relation between early skills and gender, language, If an EGRA tool does not exist in the language of a school remoteness, and the characteristics and country, a new tool (never a translation of another behaviors of other students and teachers. More tool) must be developed, following a process precisely, the Steering Committee defined these four that depends on the availability of fundamental core research questions, which best supported the statistics on the written language of assessment. current reform: The official language in Myanmar and its educational 17 system is the Myanmar language (historically sometimes instruments were designed by modifying the standard referred to as the Burmese language); consequently, it is EGRA questionnaires and adapting them to the Myanmar also the language selected for this EGRA exercise. The context. The information collected was analyzed to solid research base supporting the EGRA tool provides determine how different environmental and behavioral authoritative guidelines on the selection and development factors could influence students’ reading performance. of the test items in each subtask.47 International languages, such as English or French, are well researched and 4.4 Sampling statistical information can be readily accessed in scientific literature and scholarly databases from academic research The study subjects were a random sample of the institutions.48 However, to date, the Myanmar language and target population. The target population of the survey its orthographic system have not received the same degree consisted of students in Grades 1, 2, and 3 at schools in of research attention and lack the statistical data needed to the Yangon region of Myanmar.50 The focus on Yangon was develop an EGRA tool. not justified on the basis of its representativeness of the rest of the country but, given that this is the first round of The development of test items for EGRA Myanmar EGRA ever carried out, on the basis of cost and easiness of therefore started with the research and development logistics. Furthermore, it was estimated that the reliability of of basic linguistics data and statistics on the writing the EGRA tool for the Myanmar language could be tested system. Two statistics were needed: (i) The frequency of the even if the sample was not nationally representative. A graphemes in literature for children in early grades, which two-stage probability sample, drawn from all 2,609 schools is used to design subtasks for letter and phoneme reading in the Yangon region (which make up the DBE 3), was or analysis; and (ii) A list of some of the most frequently intended to support the reporting on reading outcomes by used words in children’s literature, which is used to design gender, school remoteness, class size, and first (or main) word-level reading. To establish relevant statistics, the team language. first identified the most prevalent books used by students in Grades 1-3. The Ministry of Education and educational First, the schools were ranked by five classifications experts confirmed that the official classroom reading of remoteness (an administrative isolation textbooks were the main reading sources for students in classification) and assigned equal probabilities of Myanmar.49 selection within each stratum. These are defined below: The statistical analysis of the results was treated A—Schools that are located in the same place or within separately for each grade level. This is particularly one hour of the Township Education Office (TEO); important because the year-end learning goals for some of the most basic reading skills differ in early primary grades. B—Schools that are farther from the TEO (more than one Thus, two different forms of the EGRA were designed, hour traveling time) and have easy access by roads and based on specific grade textbook statistics and curriculum railways; expectations: one for Grade 1 and another for Grades C—Schools that are located 1-2 miles from roads and 2-3. Consequently, Grade 3 students are expected to railways, where travel is somewhat difficult or requires perform better than Grade 2 students on each subtask. walking, but still within one day’s travel of the TEO; In Grade 1, the children learn a portion of the letters of the Myanmar alphabet. Thus, the EGRA subtasks for the D—Schools that can be reached from the TEO within a Grade 1 assessment only contain those letters covered in day, but are more difficult to travel to than group C schools, the curriculum. In Grades 2 and 3, students learn and know have higher traveling costs, or are remote enough that a all letters (and possible combinations), so the complete round trip from the TEO occurs on school holidays; alphabet is included in the EGRA subtasks for both grades. E—Schools that are in difficult locations, where travel from the TEO can occur only once (maybe more) per month. After reviewing the draft version of the EGRA tool tailored to the Myanmar language (the language of Second, 10 students from each grade in each reading instruction), the Steering Committee agreed school were sampled with equal probability, using that all designed subtasks would be applied in Grades a systematic sampling technique. They were selected 1 through 3. from the roster of students in attendance on the day of data collection. In other words, enumerators obtained the roster Additional instruments (questionnaires) were for all classes in Grades 1, 2, and 3. For a given grade, also designed to collect information on student the number of students were counted and divided by 10 and teacher characteristics and behaviors. These (target sample size) to obtain the sampling interval (“skip 47 Deno, “Curriculum-Based Measures.” 48 J.C. Ziegler, A.M. Jacobs, and G.O. Stone, 1996, “Statistical Analysis of the Bidirectional Inconsistency of Spelling and Sound in French,” Behavior Research Methods, Instruments, and Computers 28: 504-515; and R. Peereman, B. Lété, and L. Sprenger-Charolles, 2007, “Manulex-Infra: Distributional Characteristics of Grapheme-Phoneme Mappings, Infra-Lexical and Lexical Units in Child-Directed Written Material,” Behavior Research Methods 39: 579-89. 49 Pali words (used in the study of Buddhist scriptures), which are sometimes used in writing, were excluded from this analysis and the assessment. 50 Monastic (Buddhist) schools, which constitute around five percent of the school population in Myanmar, were not included in this sample. 18 identifying” the Nth student). Starting from the top of the six supervisors from DBE 3, and 42 enumerators from three roster, every Nth student among those present was selected education colleges (Thingankyun, Yankin, and Hleku) in for participation in EGRA. The final sample includes 176 the Yangon region were trained in early January 2014 by teachers and 1,681 students. the World Bank Specialist, local consultants, and Save the Children. The training lasted eight days and included the The main issues with the sampling strategy is that its following activities: (i) Practice presenting each survey to implementation limits the validity of the analysis and the participants, one by one and question by question; (ii) the interpretation of results with regards to gender In-classroom practice with peers (role-play); and (iii) 3-days and language. This caveat should be kept in mind when of in-school practice for inter-rater reliability tests. reading the results section below. With gender, because the typical class roster in Myanmar lists all boys first and All the questionnaires were pilot-tested in a sample of then the girls, in small classes the probability of selecting a 18 schools in three townships (Dagon North, Tharkayta, girl is lower than for selecting a boy. This bias was further and Mingaladon) with 600 respondents (540 students and exacerbated because the field team rounded up the skip, 60 teachers). which meant that often the last girls on the roster had zero probability of being selected. For the field work, there were six data-collection teams, each with one data-quality supervisor from For language, despite the intention of reporting Save the Children, one supervisor from DBE 3, and on students’ language background as a factor in seven enumerators from the education colleges. learning to read, this information on students was Each team was assigned one school per day and was not ultimately available to apply to the sample frame. responsible for 10 schools. At the schools, the team Only the information collected in the student interviews, interviewed all Grade 1, 2, and 3 teachers, assessed the particularly on the languages the students speak at home, reading skills of a randomly selected subsample of the can be used to explore the effect of language, but the students, and then interviewed these same students. An results cannot be considered highly robust. enumerator from the team administered the EGRA task to each sample student individually, away from the distraction of the on-going classes and other assessment activities. 4.5 Field work Save the Children carried out data processing During the field work, two types of data were with oversight by the World Bank Specialist, using collected: 1) information on students, teachers and a rigorous process for entry and verification of schools through questionnaires, and 2) students’ data. Excel templates were developed prior to the data- reading performance through the administration processing phase and included efficient checks with clear of the EGRA tool. Both the questionnaires and EGRA error messages for inconsistent or out-of-range data. tool were first field-tested in November 2013 and then Data clerks trained for three days at the Save the Children administered in January 2014. office. Data was entered once and random verification Save the Children oversaw training, pilot-testing and checks were carried out twice a day. Save the Children collection of the field data by specifically trained staff conducted spot checks and an extensive process teams, comprised of supervisors and enumerators of revising datasets occurred over two months. The final from Save the Children and education colleges. A datasets were submitted in May 2014 and are described group of six data quality supervisors from Save the Children, in Table 2. Table 2: Description of datasets No. of Dataset Unit Description of key information participants Demographics, pedagogical practices, years of teaching, pre- and in-service training, materials, assessments and homework, class Teacher 176 Teacher enrollment, and absenteeism, etc. (Information came from 53 Grade 1 questionnaire teachers, 50 Grade 2 teachers, 46 Grade 3 teachers, and 27 multi- grade teachers.) Reading assessment results on the different subtasks (Information EGRA results 1,681 Student came from 560 Grade 1 students, 577 Grade 2 students, and 544 Grade 3 students) Socioeconomic status, preschool, homework, absenteeism, learning Student 1,681 Student materials, and outside tutoring, etc. (Information came from 560 questionnaires Grade 1 students, 577 Grade 2 students, and 544 Grade 3 students) 19 IN ALL GRADES, AS “ EXPECTED, ORAL READING FLUENCY WAS SHOWN TO BE A STRONG CONTRIBUTOR TO READING COMPREHENSION.” 5. ANALYSIS AND RESULTS The analysis of the assessment and survey results Remoteness and wealth has three main objectives related to gauging the validity of the EGRA tool, describing student reading Most schools are classified as non-remote and performance, and explaining differences in EGRA the remoteness index is correlated with wealth. scores. First, since this is the first occasion that the The distribution of schools according to the remoteness EGRA tool was adapted and administered in the Myanmar classification (see subsection 4.4, Sampling) is: 54 percent language, subsection 5.1 describes the characteristics in A (least remote), 23 percent in B, 12 percent in C, of teachers and students in the Yangon region, and seven percent in D, and 3.5 percent in E (most remote).51 subsection 5.2 assesses the appropriateness of this Regarding wealth, the student questionnaires collected specific EGRA tool for this study measuring early reading data on several durables owned by the student’s household skills in the Myanmar language. Second, subsection 5.3 (e.g., a radio or bicycle). On the basis of the answers to gives a picture of student performance in reading by looking these questions, wealth variables—one continuous, one at means, zero scores, and performance groupings. Finally, categorical—were constructed using principal component subsection 5.4 presents the investigation into potential analysis (PCA).52 Not surprisingly, the analysis shows explanatory factors for differences observed in early reading that wealth is significantly different according to school skills (as requested by the Steering Committee). Note remoteness: students in non-remote schools have more that, as described in subsection 4.3, because the EGRA wealth than those in remote schools. subtasks differ for Grade 1, the analysis presented below is sometimes split between Grade 1 and Grades 2 and 3. Gender and class size Gender parity is not an issue and average class size 5.1 Who are the students and teachers (47 students) is relatively high, especially in non- in Yangon? remote schools. Classes are significantly larger in non- remote schools (49 students) than in remote ones (24 The first objective of the analysis is to provide an students), but class size does not vary significantly across overview of the characteristics of students and grades. On the day the EGRA study cohort visited, about teachers in Grades 1, 2, and 3 in the Yangon region 50 percent of the students were girls (the computed parity (DBE 3). It attempts to answer who are the students ratio is 1.03). Among students tested, 46 percent were and teachers in early grades in DBE 3. The main findings girls, but this is potentially a consequence of the bias in indicate that the target population possesses the following the random selection (as described in subsection 4.4, general characteristics: (i) A majority of teachers and Sampling). students are in non-remote schools; (ii) Students from the poorest households attend the most remote schools; (iii) Language Most students and teachers speak and use the Myanmar language in the classroom; (iv) Student absenteeism is Variation in the several language variables collected high; (v) Availability of basic teaching and learning materials was extremely low: most students, parents, and is adequate. teachers speak and use the Myanmar language. Although the proportion of students speaking the Myanmar The discussion below presents the main language in school is slightly lower in remote schools characteristics of students and teachers (weighted, (89 percent), overall in DBE 3 almost all students use according to the sampling strategy, to be the Myanmar language (94 percent). The second most representative of the Yangon region) as reported on frequently reported home language was Kayin (eight the interview questionnaires. (See Annex 4 for the detailed percent of remote school students and two percent overall). results.) It focuses first on the characteristics related to A similar pattern applies for the ability of family members the primary research questions selected by the Steering to read the Myanmar language: 85+ percent for mothers, Committee, followed by other characteristics on which fathers, and siblings; and 69 percent for grandparents information was collected. When (and only if) statistically overall. (Percentages are slightly lower in remote schools.) significant, the difference across grades, gender, or school The same pattern emerges with teachers too. Teachers of remoteness is reported. 89 percent of students speak the Myanmar language as To simplify the classification for this discussion, a remote school is defined as being in group D or E, as opposed to A, B, or C. 51 The absolute level of this wealth variable has no concrete meaning and is therefore not reported. Its only purpose is to order students according 52 to household wealth. 21 their first language; the rest of the teachers are equally split Most teachers (67 percent) live within 30 minutes of the between speaking Kayin and Rakine languages.53 school where they teach. Significantly more teachers live closer in remote schools (83 percent) than in non-remote Interestingly, languages other than Myanmar are schools (66 percent). Possibly this is because remote more rarely used in Grade 1 than in Grades 2 and 3. In areas require that teachers live there or the teacher’s home other words, more Grade 1 teachers reported never using a village may be near the school. Worryingly, 18 percent of language other than Myanmar in their classes. There are no students are taught by a teacher who has another income- significant differences regarding the remoteness of generating job. This situation was more than twice as likely the school. in remote schools (42 percent) than in non-remote schools (15 percent). Teacher demographics, class types, qualifications, training, and years of experience Students’ prior enrollment in preschool The representative teacher is a woman, relatively A significant proportion of students declared they inexperienced, who reports using the Myanmar had been to preschool (72 percent overall), but this language almost exclusively for teaching. As expected, opportunity was more rarely available to (or taken almost all students (98 percent) have a female teacher. The advantage of) students in remote schools, relative to majority of students are taught by a class teacher, rather non-remote schools. Students in non-remote schools than by subject. Thus, only 24 percent of students have a were almost twice as likely to have attended preschool teacher who teaches the Myanmar language exclusively as (74 versus 46 percent). Conditional on reporting preschool a subject. Overall only 14 percent of students in the DBE 3 attendance, 40 percent of students said they went to sample are in multi-grade classes. Multi-grade classrooms, preschool for one year and 13 percent for at least two however, are significantly more common in remote schools years. Field reports, however, indicated that students had (43 percent of students) than in non-remote schools (10 trouble recalling how long they attended preschool, so this percent). Most students have relatively inexperienced information is unlikely to be reliable. teachers: 68 percent of teachers have 0-2 years of experience teaching, 20 percent have 3-5 years, and 11 Student absenteeism percent have six years or more experience. Reported student absenteeism is high. When asked The academic qualifications of teachers are relatively whether they were absent from school at least once in the high, but the types of teaching certifications are quite past week, a majority of students (55 percent) responded diverse. In less remote schools, 94 percent of students affirmatively. The most frequent reason was illness, which are taught by teachers with a bachelor’s degree; in remote explained 59 percent of absences. Furthermore, according schools, 80 percent are. The most common teacher to school registers (which are kept universally), on the day certifications are the Certificate for Junior Assistant Teacher of the survey, the student absenteeism rate averaged (30 percent), Diploma in Teacher Education Competency 14 percent. (26 percent), Certificate in Teacher Education (24 percent), and Post-graduate Diploma in Teaching (15 percent). Teaching and learning materials Training in methods and pedagogical approaches Ownership of the Myanmar reader textbook and specific to teaching reading is uncommon. Teachers a Myanmar exercise book is almost universal, but of 93 percent (pre-service) and 67 percent (in-service) use of this material and the availability of other of students report they have never had specific training reading resources varied. Almost all students (90 to teach students to read. Grade 1 students seem to percent) owned a Myanmar reader textbook, with slightly be slightly more likely to have a teacher who had such higher ownership in non-remote schools. All teachers also pre-service training: 10 percent in Grade 1, five percent reported using the Myanmar reader textbook and finding in Grade 2, and none in Grade 3. (In-service training the material useful. Almost all students owned a Myanmar does not show such a pattern.) Remote schools appear exercise book (96 percent overall, with no difference disadvantaged regarding teachers with in-service training in according to grade, gender, remoteness, or language) reading: only 65 percent had training versus 84 percent in and most had used between one-quarter and three- less remote schools. quarters of it at the time of the survey (as observed by the enumerators). According to their responses, teachers live in proximity to the school where they teach and a sizeable number has another income-generating job. 53 Further, 89 percent of students are taught by a teacher who reports never using a language other than Myanmar in class, while 2 percent of students are in classes where only another language is spoken. 22 Figure 2: Proportion of assessment types Oral test 0.78 Lesson recitation 0.41 Chapter end test 0.38 Other 0.10 End of term evaluation 0.04 Homework 0.03 0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 Enumerators were also asked to check the proportion A majority of students (71 percent) reported of pages in the exercise books that had markings benefiting from some support with their studies at by teachers, as a proxy for how much the teachers home, mostly from parents/guardians or siblings (54 monitored the students’ work. Although teachers’ and 40 percent, respectively). However, according to markings were visible throughout the exercise book (30 teachers, the review of homework by parents is infrequent. percent of pages), it was not unusual to see no markings In more than 50 percent of cases, teachers reported (12 percent) in students’ books). Finally, enumerators also that only some homework is reviewed by parents and asked students to show them original work (not copied that involvement of parents is lower in remote than in from a book or blackboard, or dictated by the teachers), non-remote schools: six percent of parents never review but this pedagogical exercise was never used. homework in non-remote schools versus 16 percent in remote schools. Surprisingly, 65 percent of students reported having other reading materials at home (more so Getting outside tutoring (paid or not) in reading in non-remote schools). Furthermore, 90 percent skills or in the Myanmar language is quite common of students went to a school or had classroom with for students and more so in non-remote schools: a library, a reading corner, a book box, or book shelf, 59 percent of all students reported seeking academic especially in non-remote schools (92 percent versus help outside school (62 percent in non-remote schools 74 percent in remote). In most cases (85 percent), they compared to 37 percent in remote schools). used these facilities more than once a month. Only 23 percent of students were taught by teachers using teaching Assessments and learning materials other than the Myanmar reader: 23 percent used Du Won, 25 percent used Pan Taing,54 and 77 The most frequent methods used by teachers in the percent intriguingly used other unspecified material. study to measure students’ progress in reading and writing are oral (77 percent) and written tests (54 Homework, family involvement, and percent), shown in Figure 2. Homework or end-of-term outside tutoring tests are almost never used for this purpose (less than 4 percent). Teachers use oral tests significantly more often The frequency of homework assignments varied in non-remote schools (80 percent versus 56 percent), significantly from the students’ versus the teachers’ homework is used more often in remote schools (13 perspectives. Only 63 percent of the students reported percent versus 2 percent), and chapter-end tests are used having homework assignments in the week prior to the less often in Grade 1 (19 percent) than in Grades 2 (40 survey. On the other hand, only 20 percent of students percent) and 3 (56 percent). are taught by teachers who said they assigned homework less than once per week. The frequency of homework, Assessment results (all types grouped together) are according to teachers, varied according to the remoteness not often used to inform parents or administrators of a school: 40 percent of non-remote schools reported about student performance (28 percent) or to daily homework being assigned versus only 18 percent of categorize students and create instructional groups remote schools. (28 percent)—as shown in Figure 3. Assessments are mainly used to identify children who need help (47 percent), grade students (45 percent), and adapt teaching methods to better suit students’ needs (41 percent). 54 Du Won and Pan Taing (and Pimya) are privately published exercise books, work books, and guide books, based on the standard (government- issued) text book lessons. 23 Figure 3: Proportion of uses of assessment results Identify children who need help 0.47 Grade students 0.45 Adapt teaching to better suit students’ needs 0.41 Evaluate students’ understandihg of subject matter 0.30 Categorize students (create instructional groups) 0.28 Inform parents/administration of pupils performance 0.28 Other 0.05 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50 Activities and pedagogical approaches The same is true for “sound-based reading,” although to to teach reading a lesser extent. “Playing with words” is significantly more common in Grade 1 (64 percent) than in Grade 2 (42 Repetition is the most common activity used in percent) and Grade 3 (43 percent). Non-remote schools reading lessons. Figure 4 shows the frequency of the (versus remote schools) favor “sound-based reading,” “look different activities reported by the teachers surveyed. and say,” “questioning,” “role playing,” and “playing with Activities involving repetition are relatively common, while words” methods. sounding out letters; reading for comprehension; reading new words, sentences, or texts; or constructing new words 5.2 Can EGRA be adapted for the Myanmar are relatively rare. Language? There are few differences in reading methods across Since the EGRA tool has not been designed to assess grades and degree of school remoteness: early reading skills in the Myanmar language, the second objective of the analysis is to assess the • Some comprehension activities are encountered less appropriateness of the tool as developed for this frequently in remote schools than in non-remote schools study. In other words, the analysis of this subsection aims (30 percent versus 50 percent). For example, the to answer whether the adapted EGRA tool is appropriate to teacher provides questions on text comprehension and measure early reading skills in the Myanmar language. the answers first, and students answer questions on the text after. The short answer is yes, but some changes could • Letter-related activities are more frequent in lower further improve the usefulness of the tool. In fairly grades than in higher grades. For example, the teacher technical terms, this subsection discusses how the models and shows a letter first, and then the students reliability of the tool was assessed and what the specific repeat the sound of the letter or give the name of a findings are. The subsection is structured according to letter without modeling. the three main characteristics of an appropriate EGRA tool: 1) internal consistency in reliability of scales, 2) high • Constructing new sentences by using newly learned correlations across subtasks, and 3) predictive power from words is reportedly more frequent in Grade 3 than in lower level to higher level skills. Grades 1 and 2. Internal consistency in reliability of scales The most common pedagogical methods for teaching students to read are “look and say,” “questioning,” The EGRA tool and six of eight subtasks (subtask and “pictorial story telling”; least frequently employed 2, initial sound segmentation; and subtask 3, are “sound-based,” “role-play,” and “playing with letter sound knowledge excepted) appear to have words” activities. In addition to describing the activities sufficiently high internal consistency in the reliability they use to teach reading (Figure 4), teachers were asked of scales to justify its use in Myanmar. Annex 2.1 to identify their pedagogical methods (Figure 5). The presents the estimates for Grade 1 and Grades 2-3 for the traditional method of “thin bone gyi” (reading by spelling three statistics supporting this assessment: word by word) is used significantly less often in Grade 3 (40 percent) than in Grades 1 (97 percent) and 2 (85 percent). 24 Figure 4: Proportion of reading activities Teacher explains the meaning of new words 0.96 Students repeat isolated words after teacher models 0.95 Students repeat new words 0.93 Students engage in whole class reading 0.93 Teacher reads first, student(s) read/repeat after 0.92 Students repeat the name of a letter after the teacher models 0.90 Teacher tells a story orally, students retell the story on their own 0.85 Shown a letter, students give the name of a letter without model 0.76 Students read isolated words without model 0.68 Student(s) read a new, unseen text or passage without teacher 0.64 Students write compound words using known stem words 0.60 Teacher provides text comprehension questions and answers 0.55 Students repeat the sound of a letter after the teacher models 0.54 Students answer questions on text after 0.50 Students construct new sentences by using newly learned words 0.50 Teacher tells a story orally, students repeat the story verbatim 0.46 Shown a letter, students give the sound of a letter without model 0.38 Students engage in silent reading 0.35 Don’t know/Refuse 0.00 0.00 0.20 0.40 0.60 0.80 1.00 1.20 Figure 5: Proportion of pedagogical approaches Look and say method 0.93 Questioning method 0.89 Pictorial story telling method 0.84 Traditional method of “Thin bone gyi” (Reading by spelling) 0.75 Lecture and discussion method 0.74 Playing with words 0.50 Role play method 0.41 Sound based reading 0.35 Others 0.03 0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00 25 Figure 6: Predictive relationship between lower and higher level reading skills Skills or cognitive Higher level skills/ Lower level skills/subtasks concept subtasks (predictors) - Letter name per minute Text reading Oral reading fluency - Non-word per minute - Familiar-word per minute - Letter name per minute Spelling Dictation - Non-word per minute - Familiar-word per minute Reading Correct reading - Oral reading fluency comprehension comprehension - Dictation 1. Cronbach’s alpha: High average correlation of scores in sound fluency, which show undersized correlations with all subtasks, which is estimated using Cronbach’s alpha, other subtasks. Reading and listening comprehension also indicates high internal consistency in the reliability of are significantly correlated with all the other subtasks. several subtasks, which are assumed to measure the same construct. Predictive power of lower level skills to higher level skills 2. Item-test correlation: High correlation between scores on each subtask and the overall score indicates high The general expectations derived from the reliability. construction of the EGRA tool set were that 3. Item-rest correlation: High correlation between the score performances on subtasks measuring lower level on a given subtask and the global score, computed skills should be able to predict (or, at least, not be using only the other subtasks, indicates high reliability. negatively correlated) performance on subtasks measuring higher level skills. Given this, the analysis The overall Cronbach’s alpha (task scale) is 0.77 of the collected data indicates that the EGRA tool set for Grade 1 and 0.82 for Grades 2 and 3, which exhibits the appropriate relationship between lower indicates that the overall task is sufficiently reliable. and higher level reading skills. Figure 6 shows the main At the subtask level, the results indicate that scores on expected predictive relationships between lower level skills all subtasks have high item-rest and item-test correlation, and higher level skills, in text reading, spelling, and reading except for letter-sound fluency (subtask 3), initial sound comprehension. (See Annex 2.3 for the hierarchical multiple segmentation (subtask 2), and, to a lesser extent, listening regressions run to assess these relationships.) comprehension (subtask 7). Looking at the values of Cronbach’s alpha overall (column [v] in Tables A2.1 and In all grades, as expected, oral reading fluency A2.2 in Annex 2.1), removing letter sound fluency and initial was shown to be a strong contributor to reading sound segmentation would lead to substantial increases comprehension. Therefore, the unique contributions of 1) in Cronbach’s alpha. Therefore, these two subtasks, as phonetic awareness to reading fluency and spelling, and 2) designed and as administered, are not sufficiently reliable oral language skills to comprehension, illustrated in A2.5 measures of early reading skills in the Myanmar language. Annex 2.3, confirms the alignment of the data collected in the Yangon region of Myanmar with the global theoretical Pair-wise correlations across subtasks and experimental data on reading comprehension. Similar to the observations for internal consistency Based on the statistical analyses performed, it is and reliability, on the basis of pair-wise correlations, possible to conclude that, overall, the EGRA tool most subtasks appear appropriate, except for developed for this study appears appropriate for the initial sound identification and letter sound measuring early reading skills in the Myanmar knowledge (subtasks 2 and 3). Because each EGRA language. However, two of the subtasks—initial sound subtask is meant to measure the same overall skill— identification (subtask 2) and letter sound knowledge namely, early reading—appropriate subtasks should be (subtask 3)—are either irrelevant to explain reading skills in highly correlated to each other. Pair-wise correlations of the Myanmar language or their designs or administration scores among most of the subtasks of the EGRA battery were inappropriate and should be revised for future rounds are significant and positive (see Tables A2.3 and A2.4 in of assessment.55 Consequently, the discussion that follows Annex 2.2), except for initial sound identification and letter does not refer much to these two subtasks. 53 There are only hypotheses, but no convincing evidence as to why subtasks 2 and 3 appear inappropriate. These subtasks may be particularly difficult for students in Myanmar because of 1) the structure of the Myanmar language, which makes sounding the letter particularly difficult; 2) the pedagogical approaches used in surveyed schools (i.e., very little focus on phoneme awareness and letter sounds); and 3) the administration protocol, which does not provide enough clues as to the expected answer —and which could help soften the challenges as a result of 1 and 2. 26 Figure 7: Mean scores per subtask, per grade 60 G1 G2 G3 47 48 50 43 40 36 Initial sound 40 identification 30 Oral reading 50 Letter 30 26 comp. 40 sounding 30 18 20 13 20 13 Letter 10 Dictation 8 7 naming 0 10 3 4 3 3 3 1 2 2 0 0 0 0 0 0 0 0 Familiar Listenting words comp. ds on g n on n g ds on tio tio in in Non-words or si or si si Reading am nd ca ta en en en rw -w ic ou comp. rn tifi eh eh eh on ilia D rs en tte pr pr pr N m tte m id m m Le Fa co co co Le d un g g ng so tin in di ad en al ea Re iti st lr In Li ra O 5.3 What does EGRA results indicate about Mean scores and percentage of zero scores early reading skills in Yangon? per grade The third objective of the analysis is to assess the Although mean scores on all subtasks improve with appropriateness of the EGRA tool developed for this the grade, the substantial proportion of students who study to measure early reading skills. In other words, could not answer one item correctly and low mean the analysis here aims to answer how, based on the EGRA scores on various subtasks by the end of Grades 2 results, Myanmar students fare in general and, relative to and 3 remain worrying. Figures 7 and 8 illustrate the curricular expectations, in terms of early reading skills. mean scores and percentages of students who could not answer one item correctly (called “zero scores”) on each of Overall, the study portrays a situation where, despite the EGRA subtasks in each grade. (See Annex 3 for details having a relatively low proportion of non-readers statistics, such as standard error, number of observations, compared to other countries, the low mean scores etc.) Across grades and for the rest of the skills tested, indicate reading results that fall short of Myanmar Grade 1 students had a larger proportion of zero scores, curriculum expectations and point to particular especially in reading comprehension and dictation. Grades difficulties with phonemic awareness, decoding, and 2 and 3 students showed significant improvement on these inferential comprehension of text. This subsection skills, but the proportion of zero scores remains worrying. examines students’ performance on the basis of Across subtasks and grades, the largest proportions • mean scores and percentage of zero scores per grade, of zero scores appeared in the two problematic subtasks (initial sound identification and letter sound • international comparisons in the proportion of non- knowledge), even in Grades 2 and 3, which again readers, suggest that these tasks are either too complex • curricular expectations and reading patterns of good or too unfamiliar for students. Such a pattern points and poor readers, and unambiguously to floor effects, which seem to indicate that the tasks fail to capture reliably the cognitive skills targeted. • performance on questions about inferential versus literal As a consequence, both tasks are excluded from further comprehension. analyses. 27 Figure 8: Zero scores per subtask, per grade 120 G1 G2 G3 100 99 98 99 98 97 100 Letter 77 naming 80 68 Initial sound 100 Listenting identification comp. 60 50 Letter Familiar 37 35 sounding 0 words 40 32 28 27 23 20 14 11 12 9 12 Dictation Oral 10 9 4 1 7 6 5 reading comp. 0 0 Reading Non-words comp. ds on g n on n g ds on tio tio in in or si or si si am nd ca ta en en en rw -w ic ou rn tifi eh eh eh on ilia D rs en tte pr pr pr N m tte m id m m Le Fa co co co Le d un g g ng so tin in di ad en al ea Re iti st lr In Li ra O International comparison Curricular expectations and reading patterns of good and poor readers When comparing the proportion of non-readers in Yangon with those in other countries, the picture is Relative to the Myanmar curriculum in Grades 1, 2, somewhat encouraging. As a general rule, differences and 3, analyzing the proportion and performance of in language structure and complexity (and differences in good and poor readers indicate results well below sampling frame and implementation protocol) introduce expectations in all three grades and high correlation variations in the EGRA results that preclude direct across subtasks. Given that no norms or references are comparisons. In particular, students tend to learn to read available to evaluate and review EGRA scores, the following at a faster rate in languages with shallow or transparent analysis breaks down and examines students’ performance orthographies than in languages with deeper or opaque against national curriculum standards. It anchors the orthographies.56 However, because it can be assumed interpretation of the scores to Myanmar’s local curricular that the inability to read even a single word is relatively expectations. comparable across languages, zero scores on oral reading fluency can be compared and reported. On the basis of curriculum expectations, students were assigned to four groups: non-readers, poor Figure 9 illustrates scores for students in Grades 2 and 3 readers, readers with limited comprehension, and in a number of countries. It shows that, although it is in fluent readers. Currently, the primary education curriculum the best quartile, Yangon region is performing worse in Myanmar expects that students acquire these abilities than the Philippines, Indonesia, Vietnam, Guatemala, and skills in language and reading: Nicaragua, part of Honduras, the central region of Kenya and part of DRC. • For Grade 1, students successfully read and understand a short connected text and correctly answer at least three questions. • For Grade 2, students successfully read and understand a short connected text and correctly answer most of the six reading comprehension questions. • For Grade 3, although curriculum expectations are different, students are benchmarked against the grade 2 curriculum. 56 Transparent orthographies have more direct, regular correspondences between their sounds and letters. With deep or opaque orthographies, the correspondence between sounds and letters is less consistent, and letters or certain groups of letters often have different sounds in different words. See P.H.K. Seymour, L. Aro, and J.M. Erskine, 2003, “Foundation Literacy Acquisition in European Orthographies,” British Journal of Psychology 94: 143-74. 28 Figure 9: International comparison of zero scores in oral reading fluency Phillipines - Manilla/Filipino - G3 Vietnam - G3 Phillipines - Manilla/English - G3 Guatemala - Spanish - G3 DRC - Bandundu/French - G4 Indonesia - G3 Nicaragua - Spanish - G2 Honduras - Urban/Spanish - G3 DRC - Equateur/French - G4 Myanmar - Yangon - G3 Kenya - Central - English - G3 Myanmar - Yangon - G2 DRC - Equateur/orientale - G4 Laos - G3 Kenya - Central - Kiswahili - G3 Kenya - Central - Gikuyu - G3 Iraq - G3 Senegal - French - G3 Kenya - Luo-Nyanza - English - G3 Kenya - Luo-Nyanza - Dholuo - G3 Afghanistan - Dari and pashtu - G2 Phillipines - Mindanao/Filipino - G3 Yemen - G3 Kenya - Luo-Nyanza - Kiswahili - G3 Honduras - Rural/Spanish - G2 Guyana - English - G2 Phillipines - Minsdanao/English - G3 Laos - G2 Nicaragua - Acoast/Miskito - G2 Liberia - English - G2 Ethiopia - Orromifa - G3 Vanuatu - G2 Timor Leste - G2 Haiti - Creole - G2 Haiti - French - G2 Uganda, Central/Luganda - G2 Uganda Central/English - G2 Gambia -English - G2 Mozambique - Portuguese - G3 Pakistan - Urdu - G2 Nigeria - Hausa - G3 Nepal - G2 Uganda, Lango/Lango - G2 Mali - Bamanakan - G2 Mali - Songhoi - G2 Uganda, Lango/English - G2 Pakistan - Pashtu - G2 Mali - Fulfulde - G2 Mali - Bomu - G2 Mali - French - G2 Malawi - Chichewa - G2 Malawi - English - G2 0% 20% 40% 60% 80% 100% Sources: Authors’ calculations for Myanmar; World Bank, 2013, “Report on Early Grade Reading Assessment: Grades 1 and 3 in SEQAP Schools (Vietnam), unreleased PowerPoint (Washington, DC: World Bank); M. Machuca-Sierra and S. de Silva, 2014, “Draft Baseline Report for Laos” and 2012, “Lao P.D.R. EGRA Survey Report” (Washington, DC: World Bank, East Asia and the Pacific Education Unit); eddata//, https:// www.eddataglobal.org/reading/; and A. Gove and P. Cvelich, 2010, “Early Reading: Igniting Education for All—A Report by the Early Grade Learning Community of Practice” (Research Triangle Park, NC, USA: Research Triangle Institute). 29 Figure 10: Proportion of students per reading ability grouping Percent 100 2% 90 12% 32% 80 55% 70 60 49% 50 41% 40 30 33% 20 37% 17% 10 3% 10% 9% 0 G1 G2 G3 Non-readers Poor readers Limited comprehension Fluent readers Given these curriculum expectations, the study LIBRARIES HAVE A SIGNIFICANT, “ examined the continuum of students’ reading skills assessed by EGRA and assigned students to four INDEPENDENT POSITIVE groups: INFLUENCE ON STUDENTS’ 1. Non-readers: Students who do not read a single word of the connected text (ORF = 0). READING PERFORMANCE, 2. Poor readers: Students who read a portion of the ACCORDING TO THE STUDY.” connected text (ORF > 0), but cannot correctly answer a single question of the reading comprehension task. 3. Readers with limited comprehension: Students who read a portion of the text (ORF > 0), but only answer correctly a maximum of two questions (Grade 1), four questions (Grades 2 and 3), or fewer questions of the reading comprehension task. 4. Fluent readers: Students who read (a portion of) the text (ORF > 0) and answer correctly at least three questions (Grade 1) or five questions (Grades 2 and 3) of the reading comprehension task. The proportion of fluent readers increased across the three grades, but a significant proportion of students remained in the weak reading ability groups, even in Grade 3. Figure 10 illustrates the percentage of students in each of the reading ability groups. Among first graders, hardly any students met the curricular expectations of successfully reading and understanding the short connected text used in EGRA. Only 0.7 percent of Grade 1 students were capable of reading the grade-appropriate text presented and responding correctly to at least three questions. In Grades 2 and 3, only 23 percent and 48 percent of the students, respectively, reached curricular expectations and could be considered good readers. 30 Figure 11: EGRA scores for good and poor performers in Grades 1, 2, and 3 Grades 1 Grades 2 Grades 3 50 50 70 40 40 60 50 30 30 40 20 20 30 10 10 20 0 0 10 0 r- e ic ist ng n- g r- e g n- g e ic Lis ing n- ng Re ext Re ext ad t ra d ra d ra d d d d on rd on rd -W d Re Tex ilia m ilia m ilia m tio nin Li din in O or O or O or or or or on r L di tio ni N Wo N Wo N -Wo lT lT ta ten m Na m Na m Na -W -W W W W e ta te a a l s Fa ter Fa ter Fa ter r tio t t t Le Le Le ta ic D D D Poor Good It is also notable that the prevalence of fluent readers Performance on literal versus inferential versus limited readers versus weak readers (which comprehension includes non-readers and poor readers) exhibits different patterns according to grade level. In Grade 1, Deepening the analysis by looking into scores on there were far fewer fluent and limited readers (14 percent) literal and inferential comprehension suggests that than weak readers (86 percent). In Grade 2, only a minority students in Myanmar are capable of processing of students (32 percent) met the criteria for fluent reading and understanding inferences when language is performance and 27 percent of students were weak. While presented orally, but they find it more difficult when all third-graders were expected to perform well against the the inferences are text-based. The comprehension Grade 2 curriculum, only 55 percent of Grade 3 students subtasks (reading and listening) of EGRA include questions met this standard, and 12 percent were weak readers. that assess two important processes of comprehension: the capacity to understand and respond to literal questions Performance of good and weak readers is relatively and the capacity to understand and respond to inferential consistent across the EGRA subtasks, with weak questions. Table 3 summarizes the students’ scores for readers performing worse than good readers. With retrieving literal and inferential information when 1) reading the students classified by reading ability, it is interesting to the text themselves and 2) when hearing the text read explore the performance of the average student in these to them. groups on the different subtasks. Figure 11 provides an overview of good readers’ performance relative to poor Students in Grades 2 and 3 made more errors when readers and non-readers. (Fluent readers in Grades 2 drawing inferences from printed text than with and 3 were considered as a single category; in Grade 1, retrieving literal information. However, listening however, fluent readers were combined with readers with comprehension showed a reverse pattern: students limited comprehension, since there were too few fluent were better able to answer inferential questions than literal readers for a meaningful analysis.) In all three grades, ones. This result suggests that students in Myanmar are students categorized as good readers scored substantially capable of processing and understanding inferences when higher than any of the weak reading groups. language is presented orally, but they find it more difficult when the inferences are text-based. Although no inference or causal link between tasks can be made at this stage of the analysis, these This pattern of results was also observed in results indicate that successful readers who reach students who were presented with all six questions or approach curricular standards for their grades and five of the six questions (see Tables A3.4−3.6 performed well on all component skills of reading in Annex 3). This suggests that the results are not an assessed by EGRA. On the other hand, students artifact from the administration of the EGRA’s reading identified as the furthest from meeting the standards— comprehension task. (Only questions about the portion of because they could not read or could not comprehend the the text read were asked.) text—scored in the lowest range on EGRA and exhibited difficulties performing the tasks. 31 Table 3: Percent of correct responses in reading and listening comprehension for Grades 1, 2, and 3 (literal and inferential questions presented separately) Reading comprehension Listening comprehension Literal question Inferential question Literal question Inferential question Grade 1 5% 1% 58% 30% Grade 2 51% 37% 38% 56% Grade 3 72% 55% 49% 70% Total N (%) of questions 4 (100%) 2 (100%) 4 (100%) 2 (100%) 5.4 What factors explain differences observed Overview of the methodology in early reading skills? In order to identify potential factors that explain the The fourth and final objective of the analysis differences across schools, grades, and students investigates the potential explanatory factors for in early reading skills (as measured through EGRA differences in EGRA scores in the Yangon region (DBE scores), the correlation between EGRA scores and 3). It attempts to answer what can potentially explain the student and teacher characteristics are analyzed observed difference in early reading skills. first in a bivariate context (as a preliminary and preparatory stage) and second in a multivariate The results suggest that the Ministry of Education context, using a set of three-level hierarchical should prioritize interventions for a few specific mixed models (students nested within grades and factors—regardless of the students’ original or initial grades nested within schools). The first step combined skill levels—when aiming to improve reading skills. the results of the subtasks for early reading skills into a Interventions with potential for adding value could include; composite EGRA score. Then, in order to identify potential factors that might explain the differences in early reading • more efficient use of Myanmar exercise books by skills (measured by EGRA scores) across schools, grades, teachers, students, and even parents; and students, the correlation between EGRA scores and student/teacher characteristics were analyzed in two ways. • development and dissemination of best practices for First, a bivariate analysis examined school and non-school use with the exercise books; variables against the EGRA scores. Second, a multivariate • the issue of teachers taking second jobs; and analysis looked at several variables together, using groups of variables (A-G) and a set of successive mixed models in • the presence of a library in schools or a book corner in a three-level hierarchy (students nested within grades and classrooms. grades nested within schools). One of the main puzzles requiring further research The bivariate correlation analysis looks at the one- is the persistent and strong counter-intuitive to-one relationship (or correlation) of the variables relationships between the use of both summative captured in the study to higher/lower overall reading (e.g., use of chapter- or year-end tests) and formative performance—the composite EGRA score. However, (e.g., use of tests for feedback to teachers on those interventions that can help students’ performance or effectiveness of their instruction) assessments of school/teacher effectiveness cannot be directly determined student performance. It is unclear from the findings from just the bivariate results. Indeed, interpretation of whether tests students reflect on what they have learned or bivariate results does not take into account that some of whether teachers use test scores to guide their curriculum the study variables, for example, associated with a school or understand the progress of students. Finally, regarding or teaching, may also reflect (and are therefore correlated the general discourse on teaching methods in Myanmar, the with) non-school factors, such as students’ socioeconomic data suggests that the research base may not be sufficient status or the skills that students have before they start to advocate for or against specific techniques. primary school. In other words, bivariate relationships The text that follows is necessarily shortened and distilled may be the result of many factors and may results from a to highlight the results of the bivariate and multivariate reversal of causality—a cause and effect that may be the analyses for the non-expert reader. Please note that Annex opposite of a commonly held presumption. For example, 5 contains the rigorous discussions of the methodology and schools that use chapter- or year-end tests may have bivariate and multivariate analyses. introduced them to check student performance. But a common factor could underlie both variables. For another 32 example, low-performing schools might use a different Remoteness and wealth means of assessing students (e.g., as an intervention to help them monitor students) that not all schools use. Unsurprisingly, being enrolled in a remote or However, a bivariate analysis is a crucial step because it non-remote school is significantly and positively helps identify the variables that are consistently correlated correlated with the EGRA composite score overall, with the EGRA scores; and thus are necessary to—and as well as for each grade. Students in less remote included in—the multivariate analysis that follows. schools are more likely to read better and have higher EGRA composite scores than are students going to In order to analyze the correlation among multiple schools in more remote areas. Wealth is also positively potential explanatory factors (and ensure some and significantly, although weakly, correlated with scores. control for the relationship), a multivariate analysis The less well off a student’s family is, the less likely they with a three-level hierarchical mixed model is will read better. also necessary. This included grouping variables into sets (A−G), related to areas important to the study, and Gender and class size establishing five “successive” models to help remove less relevant variables. Bearing in mind the caveats regarding the sample— more male students are in the study than female The reason for using this hierarchical structure students, due to the fact that boys are listed in class modeling technique is to uncover which groups rosters before the girls—being a female student of “intervenable” variables (e.g., education inputs appears to be positively and significantly correlated and instructional practices) can improve the with a higher EGRA composite score. However, this differences in students’ reading performance (high/ difference disappears in Grade 3, possibly indicating that low EGRA scores) that may be due to socioeconomic the boys are catching up or that the gender/ability mix has characteristics and non-school factors. Identifying changed as a result of differentiated drop-outs. Class size is such variables can help identify and design interventions uncorrelated with performance. to improve early reading skills in Myanmar classrooms. This approach can also help explain 1) why schools differ Language from each other in average reading performance, and 2) why students within the same schools differ from each Despite the low variation in the language spoken other in reading performance. The goal of this study and by students, parents, and teachers in the study, the this analysis is primarily concerned with understanding results point to some possible positive correlation differences between schools to aid the Ministry of between EGRA scores and speaking the Myanmar Education in identifying productive interventions. language at home or being taught by a native Myanmar speaker. The high number of households Bivariate analysis speaking the Myanmar language (and fewer households using a different primary language) implies that it is difficult Many of the relationships between the characteristics to robustly estimate the correlation between students’ surveyed (identified in the bivariate analysis) are language and EGRA scores. This is because the proportion intuitive; in other words, the relationships seem of non-Myanmar languages in the sample was quite small, obvious and based on reasonable presumption. but included numerous languages.57 This is also the case The analysis uncovers intuitive positive correlations (both with language-related variables for teachers. variables increase or decrease) for female status, wealth, adult support (for reading and homework), prior enrollment Remembering that these results are only indicative in preschool, use of a Myanmar language exercise book, and explorative, the bivariate analysis points to some and availability of library. Expected negative correlations positive correlation between speaking the Myanmar (one variable increases as the other decreases—an inverse language at home and being a stronger reader. relationship) appear for teacher travel time and teachers Living with an adult (parents, guardian, and grandparents) with other jobs. However, the analysis also reveals several who can read the Myanmar language fluently also seems non-intuitive (or counter-intuitive) relationships, such as to positively relate to higher EGRA scores. In addition, the positive relationship for absence due to sickness teachers whose first language is Myanmar have higher and the negative correlations with the use of summative performing students, which may indicate that these assessment and the use of assessment results for teachers are better at teaching this specific language. modifying instruction. The characteristics (or variables) discussed below offer the clearest findings in the bivariate Class types, teacher demographics, qualifications, analysis. These findings are both positive and negative in training, and years of experience relation to the composite EGRA scores. However, some of the results are puzzling and need further investigation. 57 In order of frequency, the non-Myanmar languages were Kayin (2%), Shan (0.6%), Chin (0.5%), Rakine (0.4%), Kachin (0.2%), Mon (0.2%), and others (0.2%). 33 As often seen in education literature, few of the Reading activities, pedagogical approaches for basic teacher characteristics—such as gender, teaching reading, and assessments qualifications, experience, etc.—are significantly correlated with student EGRA scores. Pre-service None of the activities or instructional methods used training on how to teach reading is the notable exception: by teachers is significantly correlated with EGRA it is positively correlated with EGRA scores, but at a low scores, which is troubling on several levels, but needs significance level. In-service training on reading instruction more in-depth investigation before drawing conclusions. is oddly not correlated with scores, possibly indicating the Using chapter- and year-end tests (summative low quality or relevance of this training. Concern over this assessments) is, intriguingly, negatively correlated finding is heightened when seen in conjunction with another with EGRA scores. Use of other assessments is not result showing that being taught by a teacher who has significantly correlated with EGRA scores. Every purpose another income-generating job is significantly correlated for using the assessments is also negatively correlated with with lower EGRA scores. scores, but only a few are significant. Preschool and absenteeism Multivariate modelling The results indicate a positive correlation between For brevity and relevance, the multivariate analysis EGRA scores and preschool (as expected), as well includes only a subsample of the student, teacher, as with student absenteeism (unexpected). Going to and school characteristics. The full discussion of the preschool and the duration of preschool attendance are multivariate analysis is in annex 5. Here only the clearest both significantly correlated with higher EGRA scores. results are included, plus a cursory introduction of the Absenteeism, as reported by students, is surprisingly not elements of the multivariate analysis. To make selection significantly correlated with the EGRA score. However, the easier, the explanatory variables were grouped into specific absenteeism rate on the day of the study enumerators’ visits sets, namely, areas important to the study: showed—counter-intuitively—a positive correlation with performance, although at a very low level of confidence. Group A—explanatory variables that lie outside the This positive correlation may be an artifact of teachers scope of schools or government to affect asking the weakest students to stay at home on the day of Group B—family and home variables that have an effect the assessment, which was announced ahead of time. on school outcomes or variables where the Ministry of Education could have some influence (e.g., through Teaching and learning material communication campaigns) Owning a Myanmar reader, having access to other Group C—variables that can account for students’ reading material at home, and having an exercise initial skill levels when they enter primary education book are all significantly and positively correlated (which can also serve as a proxy for how ready the with EGRA scores, as are the extent of teachers’ students are for primary school) markings in the exercise books. The use of teaching Group D—variables that represent student-teacher materials (privately published) other than the Myanmar interactions in a classroom setting (such as tests, reader (reported by teachers) is negatively correlated with marking exercise books, homework, reading activities, EGRA scores with Pimya materials and positive with Pan etc.) Taing materials. This possibly indicates different values of these complementary resources. The presence of reading Group E—school variables that represent education resources in the school (or library, etc.) is also positively inputs (use of textbooks and types of textbooks) and correlated with scores, but not the frequency of use. other elements affected by education policy Group F—variables that represent different methods for Homework, family involvement, and outside tutoring teaching early reading skills, such as letter naming and repetition Getting outside tutoring and help at home was predictably positively correlated with scores, but the Group G—variables that represent teaching methods frequency of homework does not seem to matter. appropriate for more advanced reading (at primary Given the students’ reading skills in the sample, it is quite level), such as answering comprehension questions and troubling that the frequency of homework, as reported by creating new words students or teachers, is not significantly correlated with scores. If students get help with homework, the result is The correlations for variable sets A to E are almost positively correlated with EGRA scores, but the proportion identical for all grades, even though these variable of parents reviewing the homework (as perceived by the groups include a wide range of household, personal, teacher) is not. The variable for students who get outside and contextual conditions. This consistency of tutoring (paid or not) in reading or in the Myanmar language correlations suggests the presence of commonly observable was also significantly and strongly correlated with EGRA phenomena. Indeed, as seen with the bivariate analysis, scores. The possible usefulness of such tutoring sessions many of the same variable relationships are intuitive, and the hints that they compensate for what may not be adequately same counter-intuitive relationships also appeared, which taught in school. emphasizes the need to include multivariate analysis. 34 When analyzing the results for sets F and G (variables be understood as the lower bounds of estimates that related to early versus advanced reading instruction, indicate the direction of association between the real world respectively), it is unclear whether education conditions they represent and the outcome of student practices are responding to students’ needs, rather reading performance. than driving student performance (reverse causality). One pattern in the analysis of set F—teaching methods For the most part, the results of this multivariate and activities for students learning to read—shows analysis reflect the same observations as did the positive correlations for Grade 1 students (as expected), bivariate analysis, although with some differences— but negative (surprisingly) for Grade 3 students, with namely the lack of correlation between scores and Grade 2 students generally in the middle. This pattern adult help with homework, teacher travel time, and is consistent with teachers using instructional methods enrollment in preschool. The effects on EGRA scores of appropriate to younger students when they are teaching all the student-level variables also vary significantly across older students with poor reading skills. On the other hand, schools. This indicates that, despite consistency of the though, the pattern of results (although less strong) seen effects across grades, there may still be ways for schools with techniques for more advanced readers (set G) have to ameliorate the effects of students’ background. Being a positive correlation for Grade 3 students and weaker a female and degree of wealth are still relatively strong (or negative) correlations for students in Grades 1 and 2. predictors of higher reading scores. Interestingly, the In either case, these patterns serve as a warning against effects of adults reading and outside tutoring are relatively simplified interpretations of the findings and support the strong and positive, which may explain why adult help on necessity for more conclusive follow-up studies or more homework shows little effect. Teacher travel time, which focused experimental research. also had a strong (but negative) bivariate relationship with EGRA scores, had little effect when controlling for other Analysis of the variance decomposition variables. The most likely reason is that the effects of travel time are explained by socioeconomic factors related Deeper investigation of the multivariate analysis to school surroundings (already captured by student/ (variance decomposition between the different household characteristics). successive models; see Annex 5) indicates that the variation between test scores is explained mostly by Of particular note in all models is the large effect of student-level variables, a proportion by school-level teachers having another job, which has a strongly variables and almost none by grade. Knowing which negative relation to student performance. The school a student attends is five times better at predicting negligible effects of attending preschool indicate that a student’s reading performance than knowing what grade preschool itself is likely strongly correlated with other strong a student is in. Other results may also be from mixes of predictors of performance, such as family background. In school and non-school variables, such as whether the the model with student, household/family, and education potential benefit of attending preschool is already reflected variables, there are relatively strong effects related to use in student/household characteristics, why the use of a of Myanmar exercise books. Although owning an exercise Myanmar exercise book plays a key role, and why teaching book has a significant relationship by itself with reading methods and reading activities seem to have little effect performance, each additional variable describing the use on scores. of the book (e.g., quantity of use, teacher feedback, and original stories written by students) is associated with Coefficient analysis additional performance increases. With both bivariate and multivariate analyses, Interpretation caution is required when interpreting the results. Despite providing statistical explanation, they do not The results of this analysis indicate that most of what necessarily explain the underlying phenomena. The is considered school effects58—or how well a school explanatory variables in the model may simply be highly teaches reading skills to students in early grades—is correlated with a phenomenon that happens to be highly likely influenced by non-school factors, such as the correlated with school effectiveness, for example, without socioeconomic status of students. These factors have actually having any relationship to school effectiveness. For little to do with educational practice and can hardly be this reason, it is useful to examine the variable coefficients affected by the Ministry of Education. Indeed, non-school (a numerical measure of the variables) themselves, as the factors account for half of the variation in average school patterns may suggest what some of the true underlying performance. However, this does not mean that these causes may be. However, in many cases, the variables factors cannot be countered. Indeed, the Myanmar data represent proxies for conditions that the study cannot indicates that the effects of student characteristics, despite directly measure, despite a wish to. Accordingly, the being relatively consistent, do vary significantly from school magnitude of the effect (size of the coefficient) in the results to school. Some schools are providing environments that should not be directly interpreted. Rather, they should ameliorate the effects of gender, family background, and 58 In other words, the differences in average EGRA scores between schools. 35 home educational support. It is also possible one of the Another recommendation suggests that the Ministry factors contributing to a healthy environment for student of Education look into the issue of teachers having learning is supporting the teachers. The study showed that a second income-generating job, in addition to many teachers must work other jobs, which predictably has teaching. It would be helpful to determine if and how a negative relationship with student performance. students are negatively affected by teachers who must work another job in addition to teaching. Also useful is It can be easy to infer a cause where there is none exploring the mechanisms that already exist to address using survey data such as the data from the EGRA teachers’ needs, motivations for taking a second job, and questionnaires. As in a number of cases with the mean of mitigating any negative effects. Myanmar EGRA data, one-to-one relationships that appear stable and interpretable across distinct Libraries have a significant, independent positive groups of students may be false. The clearest example influence on students’ reading performance, of this phenomenon is the relationship between teachers’ according to the study. This points to the need for further instructional practices and students’ reading performance. research on how much the presence of a library benefits After analyzing the relationships, the data most likely shows student learning and what a cost-effective strategy for the results of teachers reacting to existing student skill distributing this benefit might look like. levels, rather than students developing skills as a result of the teachers’ practices. The results also indicate several puzzles that require further investigation, mainly the persistent and strong Nevertheless, the data indicates several specific inverse relationships between both summative and directions for future inquiry. The results also point formative assessments for student performance. to many factors where the Ministry of Education could It is unclear whether the tests accurately reflect what introduce interventions to help improve students’ ability to the students have learned (how well they can read) and read, regardless of their original or initial skill levels. whether teachers apply students’ test performance to making their teaching more effective. The data does Notably, it appears that effective use of Myanmar not suggest what is underlying this phenomenon, other exercise books may be the most promising than to confirm that the relationship exists and is likely intervention. However, it is unclear from the data what not explained by non-school factors, such as students’ effect is more prevalent: does the use of the exercise socioeconomic characteristics or initial skills. books help students read better or are students more likely to use the exercise books if they already have strong Finally, the Myanmar EGRA data suggests that the reading skills? In this case, the reality may be a vicious research base may not be sufficient to say which circle, where only the students who read better can take specific teaching techniques are the most (or less) advantage of a learning aid that accelerates their progress. effective with students. Moreover, given the results of In addition, more benefits are possible from developing and this current study, the survey data may be inadequate to advocating best practices for teachers to follow with the draw such conclusions unless the data also sufficiently Myanmar exercise book. captures the initial skill levels of students before they face the specific instructional practices common in the early primary grades. 36 37 THE MINISTRY OF “ EDUCATION COULD PROFITABLY LOOK INTO BUILDING ITS OWN CAPACITY TO CARRY OUT SURVEYS AND ANALYZE FINDINGS.” 6. SUGGESTED RECOMMENDATIONS FOR POLICY-MAKERS INTERESTED IN IMPROVING EARLY GRADE READING SKILLS The analysis and findings presented in Section 5 due to data issues; the World Bank needs to do a better represent a significant step toward systematically job of expediting the reporting and analysis. measuring and better understanding the learning outcomes (for reading) in the early grades in • The Ministry of Education could profitably Myanmar. While specific recommendations based on one look into building its own capacity to carry round of one type of learning assessment should be made out surveys and analyze findings. The capacity with care, a framework of potential actions can be sketched building process is already off to a good start as the out and proposed to the Ministry of Education, policy- Department of Myanmar Education Research (DMER) makers, and their development partners for consideration. took the lead—working closely with the Departments Possible responses to the study can be organized into of Basic Education, the states, and the regions—in three main categories. organizing the field work for the second EGRA survey in January 2015. However, DMER would benefit (and 1. Expanding the research and linking it to more easily accomplish its mission) from developing complementary research and analysis more capacity—specifically upgrading skills and hiring analysts—so that it (or its successor) can process and • The EGRA instrument, research protocol, and analyze data, and draft analytical reports independently. reporting need to be improved. This report has identified specific ways that the EGRA instrument and • Coverage of the research needs to be expanded. research protocol can be improved (see Section 5.2). Given the difficulty with sampling and implementation in Many of these changes have already been incorporated the Yangon area (former DBE 3), it is not recommended in the second round of assessing early grade reading that the Ministry of Education move rapidly to a that took place in four states and regions in January nationally representative survey, which would be costly 2015.59 Further work is needed to resolve data and and difficult to manage. A nationally representative sampling issues, and improve the robustness of the survey could best be a medium-term objective (3-5 reliability analysis by using alternate approaches, such years) of the Ministry. In the meantime, it has already as test-retest, addressing the technical issues with expanded the research to two townships each in four the two subscales that didn’t work well for the Yangon additional states and regions in Myanmar. Based on the exercise. This should help make the reporting process findings from this second EGRA round, the Ministry can for future rounds of assessment more efficient. But the decide whether (and when) to expand it to all townships delays with this first report (just now being finalized 16 in selected states and regions, and eventually to all months after the January 2014 survey) were not only states and regions. 59 A second round of assessment was prepared in 2014 and conducted in the field in 2015. Several of technical issues noted in this Yangon assessment were rectified including 1) coverage of areas where language is more diverse to assess its relationship with literacy, 2) modifications to the instructions protocol of subtask 3 to weaken the floor effect, 3) improvement of the questionnaires to capture more variations in most relevant school, teacher, and student characteristics, and 4) addition of a classroom observation module. 39 • The Ministry of Education could roll out EGRA in will include one or more learning goals, including literacy some of the other ethnic languages by working and numeracy, and related targets and indicators. with local representatives. This process has already started. The instructions for the EGRA protocol (not • The Ministry of Education can further expand the items of the subtasks yet) were adapted for four its knowledge base on student performance ethnic languages in the second round of EGRA in 2015. by launching similar research in the other core Because children learn to read the Myanmar language primary education subject—math. The early grade in government schools, the assessment components math assessment that goes along the EGRA could be of EGRA remain in the Myanmar language. Adapting adapted and used for this purpose. the instrument to other major languages could be a medium-term objective and be carried out in close 2. Using the findings to begin establishing collaboration with ethnic-based, non-government measurable reading standards for the early grades authorities. • Myanmar would be well-served by developing • Broadly sharing EGRA findings could benefit the specific standards (sometimes called Ministry of Education reform initiative in several competencies) for reading in all primary grades. ways. To date, reports and workshops on the findings One of the main objectives of this research is to support have been limited to government officials at various the establishment of specific standards for reading so levels and the participants in the survey. The Ministry they can be embedded in Myanmar’s curriculum for of Education could work with development partners to primary grades. The findings in this report (and those make the public more aware of its initiative to measure to come) can help the Ministry of Education define, and promote reading skills, and communicate the continually measure, and update reading standards in importance of reading. Myanmar could also participate the primary grades. For example, the number of letters in international conferences and workshops that focus or words pronounced correctly in 60 seconds or the on assessing students’ progress in early primary grades. number of words of connected text read correctly in School-level EGRA results could also be shared with 60 seconds could be used by schools to monitor the parents in order to improve school accountability. performance of all early primary grade students in Myanmar. The number of questions answered correctly The EGRA research can offer further insights if it is about a paragraph or text (reading comprehension) linked to other assessments on learning outcomes could also be monitored and reported on. and research on the effectiveness of teaching. An oral survey of early grade reading skills is only one • The EGRA findings can guide the Ministry of picture of learning effectiveness. Written forms of early Education in developing specific, short diagnostic literacy administered in groups (such as the “Provinha,” tests (or classroom-based assessments) that the children’s literacy assessment used in Brazil and teachers can use to measure and report on the in Mozambique) could help bring down the costs. reading performance of all their students. In the Moreover, the EGRA findings can usefully be analyzed design and implementation of standard diagnostic tests, in terms of whether they are consistent with other it is recommended that reporting stay school-based at assessment initiatives for upper grades (currently carried the beginning. Gradually this reporting can be linked out by UNICEF) or with the Ministry of Education’s own into a state-, region-, and national-based education examination system. information system as a medium-term project. School- based reporting is also recommended as a first priority, • The findings could also be linked to measures of so that school headmasters are aware of (and able to teaching effectiveness (for example, classroom report on) which students are reading at grade level, observations or assessments of teachers’ ability to below grade level, or above grade level, as measured by teach reading skills) to investigate the impact of child- the Ministry of Education’s reading standards. centered teaching practices promoted by the Ministry of Education and various development partners. The first • All in-service and new teachers need training in attempt has already started in early 2015. measuring and reporting on reading standards. Helping all of Myanmar’s roughly 300,000 primary • It would be interesting to use the EGRA work as school teachers monitor reading performance and a foundation for creating a more institutionalized report on performance is a medium-term goal. If kept assessment system. It could also support the simple and well-designed, training teachers to measure assessment-related requirements likely to come from reading performance quickly and efficiently is an the upcoming post-2015 Sustainable Development achievable objective for Myanmar. Goals (SDG) and Education for All (EFA) goals. These 40 3. Using the findings to help design and justify (or competencies) and their measurement in pre-service specific interventions aimed at improving reading teacher college curriculums is a separate exercise that outcomes should be approached in parallel. Development partners can provide financing through Ministry of Education • The Ministry of Education and other government- training budget codes and help with the monitoring and actors can use EGRA as the basis of a public evaluation of results. awareness campaign to promote reading. The Ministry of Education can go much further than • Schools need more reading materials. In connection workshops or conferences in using the basic findings of with the public awareness campaign, the Ministry of the EGRA analysis. One idea is to design a multi-media Education can think about launching—with development campaign aimed at promoting reading and making partner assistance—a program that encourages parents the public aware of the importance that all children in to read to their children daily and offers incentives to Myanmar become competent readers at the end of children’s book authors, especially local authors, to write primary education. Development partners can provide more books. A competition could be organized and financing and advice, but the initiative is not complex books could be selected by panels for the Ministry of and can be undertaken by the government. Education to purchase, print, and distribute, or to allow schools to purchase directly from authors or publishers • The Ministry of Education can immediately using school grant funds. Gradually, schools should start the process of defining standards (or become responsible, using school grants and their own competencies) and linking them to the curriculum. resources, for purchasing and maintaining classroom Establishing reading standards does not require major book corners and school libraries. Finally, before financial investment or significant external technical encouraging greater use of any textbooks, it would be assistance. The Ministry currently has the capacity, useful to review them for quality and appropriateness in cooperation with local academics, to define in order to identify possible issues that need to be fixed simple reading standards, design simple diagnostic beforehand. instruments, and to pilot-test them in schools. This action is highly recommended because it is a • Let the Ministry of Education lead the prerequisite for training teachers to use and report on development of systematic monitoring and these standards—a medium-term investment. evaluation. Investing in EGRA is an example of the type of monitoring and evaluation that should be • Once developed, the standards can be integrated systematic in any modern education system. EGRA, into the pre-service curriculum at education complementary assessments, and teacher observation colleges; also the Ministry of Education can can best be carried out under the leadership of consider supporting an off-site program for in- the Ministry of Education, with the support of its service and school-based training. Professional development partners. This is an ongoing investment development for teachers in reading instruction is in reforming the education system in Myanmar most valuable when contingent on the establishment through improving learning outcomes, learning how of standards or competencies for reading by grade. to continuously improve outcomes, and reporting Once this process has started (it does not have on learning outcomes. One example is to use pre- to be complete), the Ministry of Education and its and post- EGRA results as part of a rigorous impact development partners can invest in designing and evaluation of pilot interventions on improving early establishing off-site training programs and school-based reading skills.60 networks aimed at disseminating the standards and diagnostic tools. In-service could be best linked to a Table 4 summarizes the suggested recommendations new cadre of township-based master teachers, who according to the likely timeline of their realization and could be responsible for classroom observation and responsibilities. ongoing teacher support. The integration of standards 60 More than 40 countries have successfully used pre- and post EGRA to rigorously assess the impact of specific interventions on early reading skills, such as Liberia, the Gambia, and Mozambique (www.eddataglobal.org/countries/index.cfm). 41 Table 4: Suggested recommendations and timelines Department or Recommendations Description institution responsible Short term (1-2 years) - Resolve data and sampling issues Improve EGRA tool, - Improve robustness of reliability analysis research protocol, DMER - Redesign the two problem subtasks to fit Myanmar reporting context Build capacity to conduct - Already begun with January 2015 survey DMER, Departments surveys and analyze - Work with development partners to support and of Basic Education, findings design training Development partners Expand EGRA to other - Already begun with January 2015 survey in 4 different DMER, DBE regions states/regions - Share school-level results with parents to promote accountability Share EGRA findings - Use findings to encourage government and teacher Ministry of Education participation in international conferences and workshops on assessing student progress Start process of - Use findings from first two EGRA surveys to start developing specific establishing specific standards for reading Ministry of Education standards (competencies) - Promote a culture of continually reviewing, defining, for reading and updating standards for reading - Review early grades curriculum for quality and Begin to link reading DMER, Development appropriateness in meeting reading standards and standards to curriculum Partners change accordingly to link the standards to curriculum Develop school-based - Design short, periodic assessments for teachers to diagnostic assessments administer in class DMER, DBE for monitoring student - Make reporting by teachers/school officials mandatory performance in reading and easy Offer professional - Develop off-site training programs and school-based Development partners, development to teachers networks for teachers or school clusters Ministry of Education, to learn to measure and - Promote dissemination of tools and information through education colleges, report on students’ school-based networks or school clusters local school networks reading performance - Use public-awareness campaign to emphasize Ministry of Education, Promote a multi-media importance of learning to read in primary grades other government public-awareness - Encourage parents/guardians/adult family members to bodies, Civil Societies, campaign about reading read to children local NGOs, DPs - Offer incentives to authors to write more children’s books Increase access to Development partners, - Offer grants to encourage schools to purchase materials reading material Ministry of Education, and maintain libraries and classroom book corners - Design and pilot courses (pre-service) and workshops Integrate reading (in-service) to train teachers on best approaches to standards into education Education colleges teaching reading, effective use of textbook, and colleges (pre-service) assessment and reporting to track students’ progress 42 - Consider group-administered written assessments of literacy, such as “Provinha” used in Brazil and Mozambique - Design primary grade assessments to link to—or be Link EGRA research consistent with—higher grade assessments UNICEF, development to other types of - Link assessments of student achievement to partners, Ministry of assessments assessments of teachers’ effectiveness Education - Link assessment of student achievement with specific teaching approaches/methods - Use EGRA research to support post-2015 Sustainable Development Goals and Education for All goals Develop research surveys Various development to examine student - Design and adapt EGMA to the Myanmar context partners and Ministry performance in math of Education Medium term (3-5 years) - Upgrade skills of current government personnel to Build capacity to DMER, Departments process and analyze data conduct surveys and of Basic Education, - Hire and develop internal experts to oversee analyses analyze findings Development partners and draft reports Offer professional Development partners, development to teachers - Develop a cadre of “master teachers” to observe Ministry of Education, to learn to measure and students and teachers in classrooms, and to provide education colleges, report on students’ quality mentoring support to teachers local school networks reading performance - Institutionalize courses (pre-service) and workshops Integrate reading (in-service) to train teachers on best approaches to standards into education Education colleges teaching reading, effective use of textbook, and colleges (pre-service) assessment and reporting to track students’ progress - Design and conduct nationally representative survey Expand EGRA survey to once sampling and implementation difficulties of early Ministry of Education all Myanmar surveys are resolved Ministry of Education Adapting the EGRA in - Already started in January 2015 survey, but better with regional/local other languages to wait until difficulties of initial EGRA surveys resolved representatives Develop systematic Development partners system for monitoring - Link school-based assessments of students into a and Ministry of and evaluation of student state-regional-national education information system Education achievement in reading 43 BIBLIOGRAPHY Adams, M.J. 1990. Beginning to Read: Thinking and Lloyd, C.B., B. S. Mensch, and W.S. Clark. 2000. “The Learning about Print. Cambridge, MA, USA: MIT Press. Effects of Primary School Quality on School Dropout among Kenyan Girls and Boys.” Comparative Education Review 44 CESR.org. 2012. “Terms of Reference for the Myanmar (2): 113−47. Comprehensive Education Sector Review, 2012.” http://www.cesrmm.org/index.php/en/. Machuca-Sierra, M., and S. de Silva. 2012. “Lao P.D.R. EGRA Survey Report.” Washington, DC: World Bank, East Deno, S.L. 2003. “Curriculum-Based Measures: Asia and the Pacific Education Unit. Development and Perspectives.” Assessment for Effective Intervention 28 (3−4): 3−12. ———. 2014. “Draft Baseline Report for Laos.” Washington, DC: World Bank, East Asia and the Pacific Deno, S.L., P.K. Mirkinand, and B. Chiang. 1982. Education Unit. “Identifying Valid Measures of Reading.” Exceptional Children 49: 36−45. Ministry of Education, Union of Myanmar. 2004. “Development of Education in Myanmar 2004.” Nay Pyi Eddata//. https://www.eddataglobal.org/reading/. Taw, Myanmar: Ministry of Education. Good, R.H., D.C. Simmons, and S. Smith. 1998. “Effective ———. 1999. “Instruction and Guide for Teachers.” In Academic Intervention in the United States: Evaluating and Myanmar Language Textbook. Nay Pyi Taw, Myanmar: Enhancing the Acquisition of Early Reading Skills.” School Ministry of Education. Psychology Review 27: 45–56. ———. 2008. “Country Report: Myanmar Education Good, G.H., D.C. Simmons, and E.J. Kame’enui. Development Strategy Focusing on Inclusive Education.” 2001. “The Importance and Decision-Making Utility of a Nay Pyi Taw, Myanmar: Ministry of Education. http://www. Continuum of Fluency-Based Indicators of Foundational ibe.unesco.org/National_Reports/ICE_2008/myanmar_ Reading Skills for Third-Grade High-Stakes Outcomes.” NR08.pdf. Scientific Studies of Reading 5: 257−88. ———. 2012. “Education for All: Access to and Quality of Gove, A., and P. Cvelich. 2010. “Early Reading: Igniting Education in Myanmar.” Conference on Development Policy Education for All—A Report by the Early Grade Learning Options with Special Reference to Education and Health in Community of Practice.” Research Triangle Park, NC, USA: Myanmar, 13−16 February 2012, Nay Pyi Taw, Myanmar. Research Triangle Institute. http://yangon.sites.unicnetwork.org/files/2013/05/ Hanushek, E.A., and V. Lavy. 1994. “School Quality, Education-for-All-in-Myanmar-Final-2012-FEB-2.pdf. Achievement Bias, and Dropout Behavior in Egypt.” Living ———. 2013. “Millennium Development Goal Report Standards Measurement Study (LSMS) Working Paper, no. 2013.” Nay Pyi Taw, Myanmar: Ministry of Education. LSM 107. Washington, DC: World Bank. http://documents. worldbank.org/curated/en/1994/12/697806/school-quality- Ministry of National Planning and Economic Development, achievement-bias-dropout-behavior-egypt. and the Ministry of Health, Union of Myanmar. 2011. “Myanmar Multiple Indicator Cluster Survey 2009−2010: Hayden, M., and R. Martin. 2013. “Recovery of the Final Report.” Nay Pyi Taw, Myanmar: Ministry of National Education System in Myanmar.” Journal of International and Planning and Economic Development and the Ministry of Comparative Education 2 (2): 47−57. Health. http://www.childinfo.org/files/MICS3_Myanmar_ Hoover, W.A. 2002. “The Importance of Phonemic FinalReport.pdf. Awareness in Learning to Read.” SEDL Letter 14 (3), Muter, V., C. Hulme, M.J. Snowling, and J. Stevenson. Putting Reading First. http://www.sedl.org/pubs/sedl-letter/ 2004. “Phonemes, Rimes, and Language Skills as v14n03/3.html. Foundations of Early Reading Development: Evidence Japan International Cooperation Agency. 2013. “Data from a Longitudinal Study.” Developmental Psychology 40: Collection Survey on Education Sector in Myanmar.” Tokyo: 665–81. JICA. National Reading Panel. 2000. Teaching Children to Khandker, S.R., V. Lavy, and D. Filmer. 1994. “Schooling Read: An Evidence-Based Assessment of the Scientific and Cognitive Achievements of Children in Morocco: Research Literature on Reading and Its Implications for Can the Government Improve Outcomes?” World Bank Reading Instruction—Reports of the Subgroups. Bethesda, Discussion Paper, no. 264. Washington, DC: World Bank. MD, USA: National Institute of Child Health and Human Development. 44 Peereman, R., B. Lété, and L. Sprenger-Charolles. 2007. Sprenger-Charolles, L., L.S. Siegel, D. Béchennec, and “Manulex-Infra: Distributional Characteristics of Grapheme- W. Serniclaes. 2003. “Development of Phonological and Phoneme Mappings, Infra-Lexical and Lexical Units in Orthographic Processing in Reading Aloud, in Silent Child-Directed Written Material.” Behavior Research Reading, and in Spelling: A Four-Year Longitudinal Study.” Methods 39: 579−89. Journal of Experimental Child Psychology 84 (3): 194−217. RTI International. 2009. Early Grade Reading Assessment Stanovich, K.E. 1986. “Matthew Effects in Reading: Some Toolkit. Prepared for the World Bank Office of Human Consequences of Individual Differences in the Acquisition of Development. (Research Triangle Park, NC, USA: Literacy.” Reading Research Quarterly 21: 360−406. RTI International; and Washington, DC: USAID). Stecker, P.M., L.S. Fuchs, and D. Fuchs. 2005. “Using https://www.eddataglobal.org/documents/index. Curriculum-Based Measurement to Improve Student cfm?fuseaction=pubDetail&id=149. Achievement: Review of Research.” Psychology in the Samejima, F. 1969. “Estimation of latent ability using Schools 42: 795−819. a response pattern of graded scores.” Psychometrika UNDP. 2011. “Integrated Households Living Conditions (Monograph Supplement) 34: 100–114. Survey in Myanmar (2009−2010): Poverty Profile. Yangon, Seidenberg, M.S. 2013. “The Science of Reading and Myanmar: IHLCA Project Technical Unit. Its Educational Implications.” Language Learning and UNESCO Institute for Statistics. 2011. International Development 9: 331–60. Standard for Classification of Education (ISCED). http:// Seymour, P.H.K., M. Aro, and J.M. Erskine. 2003. www.uis.unesco.org/Education/Pages/international- “Foundation Literacy Acquisition in European standard-classification-of-education.aspx. Orthographies.” British Journal of Psychology 94: 143−74. USAID and eddata//. “Early Grade Reading.” Shahar-Yames, D., and D.L. Share. 2008. “Spelling as https://www.eddataglobal.org/reading/. a Self-Teaching Mechanism in Orthographic Learning.” World Bank. 2013. “Report on Early Grade Reading Journal of Research in Reading 31, 22–39. Assessment: Grades 1 and 3 in SEQAP Schools (Vietnam). Share, D.L. 1999. “Phonological Recoding and Unreleased PowerPoint. Washington, DC: World Bank. Orthographic Learning: A Direct Test of the Self-Teaching Wren, S., B. Litke (programmer), D. Jinkins, S. Paynter, J. Hypothesis.” Journal of Experimental Child Psychology 72: Watts, I. Alanis. 2000. “Cognitive Elements of Reading.” In 95−129. Cognitive Foundations of Learning to Read: A Framework. Share, D.L., and A.F. Jorm. 1987. “Segmental Analysis: Austin, TX, USA: SEDL. Co-requisite to Reading, Vital for Self-Teaching, Requiring http://www.sedl.org/reading/framework/elements.html. Phonological Memory.” Cahiers de Psychologie Cognitive/ Ziegler, J.C., A.M. Jacobs, and G.O. Stone. 1996. European Bulletin of Cognitive Psychology 7: 509−513. “Statistical Analysis of the Bidirectional Inconsistency Simmons, D.C., and E.J. Kame’enui, eds. 1998. What of Spelling and Sound in French.” Behavior Research Reading Research Tells Us about Children with Diverse Methods, Instruments, and Computers 28: 504−515. Learning Needs: Bases and Basics. Mahwah, NJ, USA: Lawrence Erlbaum Associates, Inc. Sprenger-Charolles, L., P. Colé, and W. Serniclaes. 2006. “Reading Acquisition and Developmental Dyslexia.” In Essays in Developmental Psychology. Hove, UK, and New York: Psychology Press. 45 ANNEX 1: OVERVIEW OF EGRA TOOLS AND STIMULI A1.1 Presentation of the eight subtasks in the 5. Invented word decoding assessed children’s skill at Myanmar EGRA tool for Grades 1-3 applying letter-sound correspondence rules to decode non-words (or invented words). For Grade 1, the 50 1. Letter name knowledge assessed children’s invented words used the 24 vowel sounds covered automaticity in letter recognition. Each letter was in the Grade 1 textbook. For Grades 2 and 3, the represented proportionally to its occurrence in the invented words had all 50 vowel sounds in the Myanmar grade-appropriate textbook consulted. Letters were language. The children looked at a chart of 50 non- presented in a chart containing 10 rows of 10 letters in a words and were asked to sound out as many words as fixed random order. Students were asked to identify as they could within one minute, yielding a score of correct many letters they could within one minute,61 yielding a invented words per minute (CIWPM). score of correct letter names per minute (CLNPM). 6. Oral reading assessed children’s fluency in reading 2. Initial sound identification assessed children’s a passage of grade-level text aloud and their ability to phonemic awareness. Children were asked to listen understand what they had read. There were two parts to a word and identify the first sound in that word. to this subtask: To ensure that these words were known and familiar to the students, they were chosen from the frequent a. Oral reading fluency: In Grade 1, all the words word list obtained for each grade. After two practice included in the text passage included grade items, children were given 10 test items (no time limit). appropriate letters and letter combinations. The The final score was the number of words where the passage was 44 words long for Grade 1, and 38 children identified the correct initial sound (CIS), with the words long for Grades 2 and 3. A narrative story maximum possible score being 10. was considered appropriate to test connected text reading in those grades. The stories respected 3. Letter sound knowledge assessed children’s the typical features of narrative prose: they had knowledge of letter-sound relationships. The test “a beginning section where the characters are items were selected and displayed following the introduced, a middle section containing some same procedure as described for subtask 1 (a chart dilemma, and an ending section with an action containing 10 rows of 10 random letters). Children were resolving the dilemma.” Students were asked to asked to speak the sound of as many letters as they read aloud as much of the passage they could in could within one minute, yielding a score of correct one minute. The oral reading fluency (ORF) score for letter sounds per minute (CLSPM). this task was derived by calculating the number of 4. Familiar word reading assessed children’s skill at correct words read per minute. reading high-frequency words. The 50 most frequent b. Reading comprehension: After the children finished words in reference textbooks were presented in a the passage or the minute ended, the passage was fixed random order to students. Children were asked removed. Children were asked six questions orally: to read as many words as they could within one four questions were literal and required students to minute, yielding a score of correct familiar words per recall basic facts from the passage read to them, minute (CFWPM). while two were inferential. The reading comprehension score was the number of correct answers, with a maximum possible score of 6 (CRCQ). 60 For all timed subtasks (1, 3, 4, 5, and 6a), two skip rules were established: 1) if the child hesitates or stops on a letter or word for three seconds or more, it was skipped; 2) if the child does not provide a single correct response on the first line (five or 10 items, depending on the subtask), this subtask was discontinued and the child moved on to the next subtask. 46 7. Listening comprehension is the ability to make sense 8. Dictation assessed children’s skill at spelling. Children of oral language. In this subtask, a short passage was could spell familiar words from memory or sound out read to children who were then asked six questions words and apply the sound-spelling correspondences. orally about that passage. The listening comprehension In this subtask, a short sentence of four words was read score was the total correct answers, with a maximum to the children, who then attempted to spell the words possible score of six (CLCQ). in writing. The dictation score was the number of words spelled correctly (WSC). A1.2 Actual subtask stimuli in the Myanmar EGRA tool Subtask 1: Letter naming Subtask 2: Initial sound identification 47 Subtask 3: Letter sound knowledge Subtask 4: Familiar word reading 48 Subtask 5: Nonsense word decoding Subtask 6a and 6b: Oral reading fluency and reading comprehension 49 Subtask 7: Listening comprehension Subtask 8: Dictation 50 ANNEX 2: DETAILED RESULTS FOR THE RELIABILITY OF THE EGRA TOOL A2.1 Internal consistency in reliability of scales lesser extent, listening comprehension (subtask 7). It is also observed in column (v) that removing letter sound fluency The overall Cronbach’s alpha (task scale) is 0.77 for and initial sound segmentation from the construction of Grade 1 and 0.82 for Grades 2 and 3 (Tables A2.1 the overall task scale (the statistics in column [v] indicates and A2.2), which indicates that the overall task is the value of Cronbach’s alpha, if one removes the line sufficiently reliable. At the subtask level, the results subtask) would lead to substantial increases in Cronbach’s indicate that scores on all subtasks have high item-rest alpha. Therefore, these two subtasks, as designed and as and item-test correlation, except for letter-sound fluency administered, are not sufficiently reliable measures of early (subtask 3), initial sound segmentation (subtask 2), and, to a reading skills in the Myanmar language. Table A2.1: Reliability analysis using correlation and Cronbach’s alpha in Grade 1 (i) (ii) (iii) (iv) (v) Subtasks Obs Sign Item-test correlation Item-rest correlation Cronbach’s alpha 1. Letter name fluency 559 + 0.5951 0.4384 0.7568 2. Initial sound segmentation 559 + 0.321 0.1396 0.7996 3. Letter sound knowledge 558 - 0.184 -0.0146 0.8197 4. Familiar word reading fluency 552 + 0.8838 0.8278 0.689 5. Non-word reading fluency 551 + 0.8271 0.7475 0.7032 6a. Oral text reading fluency 540 + 0.8195 0.7401 0.7068 6b. Reading comprehension 322 + 0.7303 0.6216 0.7433 7. Listening comprehension 559 + 0.4596 0.2801 0.7804 8. Dictation (word score) 550 + 0.6796 0.5549 0.7376 Global EGRA task scale 0.7743 Table A2.2: Reliability analysis using correlation and Cronbach’s alpha in Grades 2 and 3 (i) (ii) (iii) (iv) (v) Subtasks Obs Sign Item-test correlation Item-rest correlation Cronbach’s alpha 1. Letter name fluency 1119 + 0.5098 0.3615 0.8162 2. Initial sound segmentation 1119 + 0.2664 0.095 0.8455 3. Letter sound knowledge 1118 + 0.3019 0.1324 0.8415 4. Familiar word reading fluency 1114 + 0.8858 0.8373 0.7564 5. Non-word reading fluency 1113 + 0.8254 0.7552 0.7674 6a. Oral text reading fluency 1110 + 0.8678 0.8126 0.7598 6b. Reading comprehension 1121 + 0.795 0.7141 0.7727 7. Listening comprehension 1118 + 0.5189 0.3734 0.8146 8. Dictation (word score) 1113 + 0.7647 0.6752 0.7779 Global EGRA task scale 0.8162 51 A2.2 Pair-wise correlations across subtasks non-word fluency, familiar-word fluency, and oral fluency in text reading; reading comprehension; and dictation are Similar to the observations for internal consistency highly and positively correlated. and reliability, on the basis of pair-wise correlations, most subtasks appear appropriate, except for Reading and listening comprehension also are the initial sound identification and letter sound significantly correlated with all the other subtasks. knowledge. Because each EGRA subtask is meant to Correlations of scores in reading comprehension with measure the same overall skill—namely early reading— scores on non-words, familiar words, and text reading are appropriate subtasks should be highly correlated to each large in Grades 2 and 3, and moderate to large in Grade other. Pair-wise correlations of scores among most of 1. With score listening comprehension, these pair-wise the subtasks of the EGRA battery are significant and correlations are moderate in Grades 2 and 3, and small positive (see Tables A2.3 and A2.4), except for initial in Grade 1. Correlations with scores on the letter-naming sound identification and letter sound fluency, which show subtask are significant and moderate to small in Grade 1, undersized correlations with other subtasks. In particular, while moderate to large in Grades 2 and 3. Table A2.3: Correlation matrix across all EGRA subtasks in Grade 1 2. 1. 3. 4. 5. 6a. 6b. 7. Initial Letter Letter Familiar Non-word Oral text Reading Listening Subtasks sound name sound word reading reading reading compre- compre- segmen- fluency knowledge fluency fluency fluency hension hension tation 1. Letter name fluency 2. Initial sound 0.0193 segmentation 3. Letter sound 0.0817 -0.0053 knowledge 4. Non-word 0.5325*** 0.1419*** 0.0323 reading fluency 5. Familiar word 0.5755*** 0.11 0.0625 0.88*** reading fluency 6a. Oral text 0.3232*** 0.2315*** -0.0186 0.8238*** 0.7235*** reading fluency 6b. Reading 0.1378 0.1849*** -0.0185 0.6182*** 0.484*** 0.6709*** comprehension 7. Listening 0.2505*** 0.0068 -0.0617 0.2692*** 0.2554*** 0.197*** 0.1587* comprehension 8. Dictation 0.3076*** 0.0599 -0.014 0.6216*** 0.545*** 0.5326*** 0.4026*** 0.2019*** Note: Significance at * p < 0.05, ** p < 0.01, and *** p < 0.001. 52 Table A2.4: Correlation matrix across all EGRA subtasks in Grades 2 and 3 2. 1. 3. 4. 5. 6a. 6b. 7. Initial Letter Letter Familiar Non-word Oral text Reading Listening Subtasks sound name sound word reading reading reading compre- compre- segmen- fluency knowledge fluency fluency fluency hension hension tation 1. Letter name fluency 2. Initial sound 0.019 segmentation 3. Letter sound 0.0371 0.2957*** knowledge 4. Non-word 0.3788*** 0.0602 0.1074*** reading fluency 5. Familiar word 0.355*** 0.0404 0.0747 0.8585*** reading fluency 6a. Oral text 0.3721*** 0.0364 0.092** 0.9202*** 0.8229*** reading fluency 6b. Reading 0.1378 0.0025 0.0513 0.7292*** 0.6499*** 0.7434*** comprehension 7. Listening 0.1866*** 0.0419 0.0161 0.3409*** 0.3048*** 0.3293*** 0.4198*** comprehension 8. Dictation 0.2741*** 0.0334 0.0575 0.7051*** 0.6502*** 0.6844*** 0.6444*** 0.3614*** Note: Significance at * p < 0.05, ** p < 0.01, and *** p < 0.001. A2.3 Predictive power of lower level skills to and 3 (Table A2.5). Independent variables are the subtasks higher level skills measuring lower level skills and a wealth index in order to control (at a minimum) the student’s household wealth level To assess the relationships of lower level skills and grade (for Grades 2-3). For all three higher level skills, to higher level skills, a few hierarchical multiple scores on all lower level skills are positively and significantly regressions (one for each of the three higher level correlated, except for letter naming (negatively correlated skills) were run separately for Grade 1 and Grades 2 with oral reading fluency). Table A2.5: Regression results for predicting higher level skills Skills or cognitive Measurement Grades 2 % of variance Predictors (subtasks) Grade 1 concept (subtask) and 3 explained Letter name per minute -0.12*** 0.06 Oral reading Text reading Non-word per minute 0.26 0.24 fluency Familiar word per minute 0.82*** 0.92*** Letter name per minute 0.0 0.00 Spelling Dictation Non-word per minute 0.02 0.03** Familiar word per minute 0.06*** 0.03*** Oral reading fluency 0.02*** 0.03*** Reading Reading comprehension comprehension Dictation 0.08* 0.09* Note: Significance at * p < 0.05, ** p < 0.01, and *** p < 0.001. 53 ANNEX 3: DESCRIPTIVE STATISTICS FOR EGRA SUBTASKS (GRADES 1, 2, 3) FOR SECTION 5.3 Table A3.1: Descriptive statistics for each EGRA subtask in Grade 1 (including mean, standard deviation, standard error, percentage of zero scores and number of students in the sample and the target population) N N Standard Standard %0 Subtask Mean (sample (pop. deviation error score size) size) 1. Letter names per minute 36.32 18.71 2.79 4% 557 114,378 2. Initial sound segmentation 0.01 0.22 0.01 99% 557 114,374 3. Letter sounds per minute 0.12 1.89 0.06 100% 556 114,342 4. Familiar words per minute 12.82 10.93 2.04 26% 550 112,121 5. Nonsense words per minute 7.89 6.78 1.14 30% 549 112,095 6a. Oral reading fluency 7.58 9.79 1.77 35% 538 110,323 6b. Reading comprehension 0.34 0.57 0.08 77% 321 69,535 7. Listening comprehension 2.91 1.63 0.13 9% 557 114,331 8. Dictation 0.78 1.08 0.19 68% 548 111,925 54 Table A3.2: Descriptive statistics for each EGRA subtask in Grade 2 (including mean, standard deviation, standard error, percentage of zero scores and number of students in the sample and the target population) N N Standard Standard %0 Subtask Mean (sample (pop. deviation error score size) size) 1. Letter names per minute 43.21 14.51 0.87 1% 577 110,154 2. Initial sound segmentation 0.02 0.48 0.01 99% 577 110,154 3. Letter sounds per minute 0.47 5.64 0.19 98% 576 110,107 4. Familiar words per minute 26.36 16.83 1.84 6% 574 109,213 5. Nonsense words per minute 13.19 9.75 1.05 12% 573 108,389 6a. Oral reading fluency 30.02 19.45 2.23 10% 573 109,802 6b. Reading comprehension 2.81 2.15 0.17 27% 577 110,154 7. Listening comprehension 2.63 1.63 0.14 12% 576 110,077 8. Dictation 1.60 1.26 0.14 33% 574 107,665 Table A3.3: Descriptive statistics for each EGRA subtask in Grade 3 (including mean, standard deviation, standard error, percentage of zero scores and number of students in the sample and the target population) N N Standard Standard %0 Subtask Mean (sample (pop. deviation error score size) size) 1. Letter names per minute 47.30 14.34 0.92 0% 542 102,161 2. Initial sound segmentation 0.08 0.38 0.07 98% 542 102,161 3. Letter sounds per minute 0.27 4.89 0.14 97% 542 102,161 4. Familiar words per minute 40.87 21.03 2.63 5% 540 101,308 5. Nonsense words per minute 17.91 11.01 1.28 10% 540 101,308 6a. Oral reading fluency 48.54 23.95 2.75 7% 537 100,663 6b. Reading comprehension 4.00 2.02 0.17 12% 544 102,261 7. Listening comprehension 3.36 1.58 0.11 5% 542 102,161 8. Dictation 2.05 1.37 0.17 22% 539 100,743 55 Table A3.4: Percent of correct responses to five to six questions in reading and listening comprehension for Grades 1, 2, and 3 (literal and inferential questions presented separately, calculated for students who responded to 5-6 comprehension questions) Reading comprehension Listening comprehension Literal question Inferential question Literal question Inferential question Grade 1 34% 15% 82% 32% Grade 2 74% 62% 45% 64% Grade 3 84% 67% 52% 72% Total N (%) of 4 (100%) 2 (100%) 4 (100%) 2 (100%) Questions Table A3.5: Percent of correct responses to five questions in reading and listening comprehension for Grades 1, 2, and 3 (literal and inferential questions presented separately; includes only students asked five questions) Reading comprehension Listening comprehension Literal question Inferential question Literal question Inferential question Grade 1 27% 02% 70% 00% Grade 2 56% 29% 32% 60% Grade 3 67% 41% 53% 67% Total N (%) of 4 (100%) 2 (100%) 4 (100%) 2 (100%) Questions Table A3.6: Percent of correct responses to all six questions in reading and listening comprehension for Grades 1, 2, and 3 (literal and inferential questions presented separately; includes only students asked all six questions) Reading comprehension Listening comprehension Literal question Inferential question Literal question Inferential question Grade 1 34% 15% 82% 32% Grade 2 74% 62% 45% 64% Grade 3 84% 67% 52% 72% Total N (%) of 4 (100%) 2 (100%) 4 (100%) 2 (100%) Questions 56 ANNEX 4: METHODOLOGY AND DETAILED RESULTS TABLE FOR SUBSECTION 5.4 A4.1 Overview of the methodology To ensure the simplest explanation, instead of using individual subtasks scores, the early reading skills In order to identify potential factors that explain the are summarized into one EGRA composite score. differences across schools, grades, and students The composite EGRA score was created using data from in early reading skills (as measured through EGRA the four higher-level subtasks (familiar words, non-words, scores), the correlation between EGRA scores and oral reading fluency, and reading comprehension). First, student and teacher characteristics are analyzed for each of the selected four subtasks, item response first in a bivariate context (as a preliminary and theory (IRT) was used to create continuous interval-scale preparatory stage) and second in a multivariate scores. The graded response IRT model62 was used to context using a set of three-level hierarchical mixed accommodate the differential accuracy of different items models (with students nested within grades and (namely letters, words, or questions) included in each grades nested within schools). The bivariate correlation subtask. The IRT scale scores are more mathematically analysis looks at the one-to-one relationship of various useful than raw scores (used previously to compute mean factors to higher/lower overall reading performance. and zero scores) for analyzing potential explanatory factors. However, one cannot directly extrapolate these results They do not artificially truncate high and low extremes of to possible interventions, which could positively affect the score continuum, and avoid the spurious modification school and teacher effectiveness in imparting reading skills. of relationships with other variables that results from the Indeed, interpretation of bivariate results does not take into truncation.63 Second, these four scores were standardized account that the investigated characteristics (assumed to within the sample to have a mean of zero and a standard be potential factors in making a school or teacher effective) deviation of one, and then averaged to produce the may also reflect (and are therefore correlated with) non- composite score. school related factors, such as students socio–economic status or baseline cognitive skills. In other words, bivariate The multivariate relationships are estimated using a relationships may be the result of many factors, including a mixed effects model; the general form of the model reversal of the expected direction of causality. is a three-level hierarchical structure, with students nested within grades and grades nested within For example, schools with summative assessment schools.64 The purpose of this hierarchical structure practices may have introduced them as a policy to modeling technique is to uncover the contribution of various monitor student performance. But a common factor groups of “intervenable” variables (e.g., education inputs could underlie both variables, such as a broad-based and instructional practices) to ameliorate the differences— school intervention targeting low-performing schools with due to socio–economic characteristics and non-school an assessment component that not all schools used. factors—in reading performance (as measured through In order to deepen the analysis—and ensure that some EGRA scores). Identifying such variables would help identify control for the correlation among all potential explanatory and design interventions to improve early reading skills in factors is accounted for—the three-level hierarchical mixed Myanmar classrooms. model is used. 62 F. Samejima, 1969, “Estimation of Latent Ability Using a Response Pattern of Graded Scores,” Psychometrika (Monograph Supplement) 34: 100-114. 63 The letter sound and initial sound subtasks were not used for the composite score, given the reliability issues identified in Section 5.2. The letter naming subtask was not used either, due to the lower psychometric quality of the item responses for it. The responses did not adequately define a single latent dimension, which is necessary for the application of IRT models. 64 Thus, without any explanatory variables (null model), the score for a specific student is the sum of 1) the grand mean of all students’ scores, 2) the deviation of the student’s school mean score from the grand mean score, 3) the deviation of the within-grade mean score from the school mean score, and 4) the deviation of the student’s score from the within-grade-within-school mean score. 57 This approach helps explain 1) why schools differ A4.2 Bivariate analysis from each other in average reading performance, and 2) why students within the same schools differ Many of the relationships identified in the bivariate from each other in reading performance. Although analysis are intuitive, such as positive correlations this modeling approach could also explain why students for female status, wealth, adult support (for reading in different grades differ in performance, the analysis does and homework), prior enrollment in preschool, not focus on normal progression across grades, which is of use of a Myanmar language exercise book, and no interest in policy and intervention design.65 This analysis availability of library—as well as negative correlations is primarily concerned with understanding differences for teacher travel time and teachers with other between schools because the Ministry of Education can jobs. However, there are also several non-intuitive effectively intervene on this issue. relationships, such as the positive relationship for absence due to sickness, the negative correlations Finally, the modeling approach follows a successive with use of summative assessment and the use of process, which can remove less relevant variables assessment results for modifying instructions. These and prevent unnecessary limits to the degrees of counter-intuitive relationships are a reminder that bivariate freedom. To statistically explain these differences, the relationships may be the result of many factors, including a analysis explores the results from these successive models: reversal of the expected direction of causality. The detailed bivariate results are presented below. • Null model: the original statistical model without any explanatory variables Remoteness and wealth • Base model: the model after including variable groups Unsurprisingly, non-remoteness and wealth are A and B positively correlated with the EGRA composite scores. As illustrated in Figure A4.1, being enrolled • Context model: the model after including groups A, B, in a remote or non-remote school is significantly and and C positively (large coefficient) correlated with the EGRA composite score overall, as well as for each grade. Wealth • Control model: the model after including groups (as measured through the continuous and categorical A, B, C, and D (Although variables in group D reflect variables) is also positively and significantly, but weakly activities of teachers and the school environment, they (small coefficient), correlated with scores. It is, however, are included in the control model because the variables impossible to confirm the robustness of these correlations describe the students’ perspectives. They may also in a bivariate setting, especially since remoteness and reflect student-driven phenomena.) wealth are highly correlated. • Inputs and pedagogical models (E, F, and G): the Gender and class size control model plus variables exclusively from groups E, F, or G (To prevent saturation of the data with coefficient Bearing in mind the caveats regarding the sample, estimations, the variables from groups E, F, and G were being a female appears to be positively and not carried forward to subsequent modeling stages. In significantly correlated with the EGRA composite other words, the models using variable sets E, F, and score (Figure A4.2). However, this difference disappeared G, only include variables from their respective set in in Grade 3, possibly indicating that the boys are catching addition to the variables from variable groups A, B, C, up or that the gender/ability mix has changed as a result and D.) of differentiated drop-outs. Class size is uncorrelated with performance. In this process of successive modeling, all variables in each group were tested for relationships with scores that Language were likely the result of chance.66 Before moving to the next variable group, variables with chance relationships Despite low variation in the language spoken by were removed from the model. The exception was the students, parents, and teachers, the results point to control variables (groups A and B), whose effects varied some possible positive correlation between scores significantly across schools, despite having insignificant and speaking the Myanmar language at home or main effects. Conversely, the procedure has fixed being taught by a native Myanmar speaker. The high student-level predictors, whose effects did not vary proportion of households speaking the Myanmar language significantly between schools. (and consequent low variation for this characteristic in the 65 In this model, differences 1 and 2 are represented by the standard deviation of all schools about the base group mean and the standard deviation of the students about their grade means within their schools, respectively. Although the standard deviation of the grade indicates that the school means vary across schools, it will not be modeled in these analyses because they do not concern understanding why students in different grades differ in performance. 66 For model-building purposes, an arbitrary threshold of p < 0.15 was used to identify chance relationships. 58 Figure A4.1: EGRA scores per remoteness status and grade All grades Grade 1 .6 .5 .4 .4 .3 Density Density .2 .2 .1 0 0 -4 -2 0 2 -2 -1 0 1 2 3 zegracomposite zegracomposite Non-remote Remote Non-remote Remote kernel = epanechnikov, bandwidth = 0.1782 kernel = epanechnikov, bandwidth = 0.2361 Grade 2 Grade 3 .5 .6 .4 .3 .4 Density Density .2 .2 .1 0 0 -2 -1 0 1 2 -3 -2 -1 0 1 2 zegracomposite zegracomposite Non-remote Remote Non-remote Remote kernel = epanechnikov, bandwidth = 0.2215 kernel = epanechnikov, bandwidth = 0.1882 Figure A4.2: EGRA scores per gender and grade All grades Grade 1 .6 .6 .4 .4 Density Density .2 .2 0 0 -3 -2 -1 0 1 2 -2 -1 0 1 2 zegracomposite zegracomposite Female Male Female Male kernel = epanechnikov, bandwidth = 0.1730 kernel = epanechnikov, bandwidth = 0.2273 Grade 2 Grade 3 .6 .8 .6 .4 Density Density .4 .2 .2 0 0 -2 -1 0 1 2 -3 -2 -1 0 1 2 zegracomposite zegracomposite Female Male Female Male kernel = epanechnikov, bandwidth = 0.2201 kernel = epanechnikov, bandwidth = 0.1767 59 Figure A4.3: EGRA scores per mother tongue and grade All grades Grade 1 .6 .6 .4 .4 Density Density .2 .2 0 0 -4 -2 0 2 -2 -1 0 1 2 3 zegracomposite zegracomposite Myanmar spoken at home Other language Myanmar spoken at home Other language kernel = epanechnikov, bandwidth = 0.1650 kernel = epanechnikov, bandwidth = 0.2034 Grade 2 Grade 3 .5 .6 .4 .4 .3 Density Density .2 .2 .1 0 0 -2 -1 0 1 2 -3 -2 -1 0 1 2 zegracomposite zegracomposite Myanmar spoken at home Other language Myanmar spoken at home Other language kernel = epanechnikov, bandwidth = 0.2100 kernel = epanechnikov, bandwidth = 0.1838 sample) implies that it is difficult to robustly estimate the etc.) are significantly correlated with student EGRA correlation between students’ language and EGRA scores. scores. Pre-service training on how to teach reading is This is especially the case since this analysis needs to the notable exception. It is positively correlated with EGRA combine all non-Myanmar languages observed, which are scores, but at a low significance level. In-service training on quite numerous.67 This is also the case for language-related reading instruction is not correlated with scores, possibly variables for teachers. indicating the low quality or relevance of this training. It is worrying because being taught by a teacher who has Remembering that these results are only indicative and another income-generating job is significantly correlated explorative (given the low variation in the language spoken with the EGRA scores. at home), bivariate regressions result points to some positive correlation between speaking the Myanmar Preschool language at home and being a stronger reader (Figure A4.3). Living with an adult (parents, guardian, and Results indicate a positive correlation between grandparents) who can read the Myanmar language fluently EGRA scores and preschool (as expected), as well also seems to positively relate to higher EGRA scores. as with student absenteeism (unexpected). Attending Finally, teachers whose first language is Myanmar have preschool and the duration of preschool attendance are higher performing students, which may indicate that these both also significantly correlated with scores on a bivariate teachers are better at teaching this specific language. basis. Absenteeism, as reported by students, is not significantly correlated with the EGRA score. However, Class types, teacher demographics, qualifications, the absenteeism rate observed on the day of the visits training, and years of experience showed—counter-intuitively—a positive correlation with performance, although at a very low level of confidence. As is generally the case in the education literature, This positive correlation may be an artifact of teachers few of the basic teacher characteristics described asking the weakest students to stay at home on the day of in subsection 5.1 (gender, qualifications, experience, the assessment, which was announced. 67 In order of frequency, the non-Myanmar languages were Kayin (2%), Shan (0.6%), Chin (0.5%), Rakine (0.4%), Kachin (0.2%), Mon (0.2%), and others (0.2%). 60 Teaching and learning material homework is positively correlated with EGRA scores, but the proportion of parents reviewing the homework (as Owning a Myanmar reader, having access to other perceived by the teacher) is not. The variable for students reading material at home, and having an exercise who get outside tutoring (paid or not) in reading or in book were all significantly and positively correlated the Myanmar language is also significantly and strongly with EGRA scores, as was the extent of teachers’ correlated with EGRA scores. The possible usefulness of markings in the exercise books. The use of teaching such tutoring sessions hints that they compensate for what materials other than the Myanmar reader (reported by may not be adequately taught in school. teachers) is negatively correlated with EGRA scores when the material is Pimya and positively when the material Reading activities, pedagogical approaches to teach is Pan Taing. This possibly indicates different values of reading, and assessments these complementary resources. The presence of reading resources in the school (or library, etc.) is also positively None of the activities or pedagogical approaches correlated with scores, but not the frequency of use. is significantly correlated with EGRA scores, and using chapter-end and end-of-term tests (summative Homework, family involvement, and outside tutoring assessments) is, intriguingly, negatively correlated with EGRA scores. Using other assessments is not Contrary to outside tutoring and help at home, significantly correlated with EGRA scores. Every purpose which are positively correlated with scores, the for using the assessments is also negatively correlated with frequency of homework does not seem to matter. scores, but only a few are significant. With regards to students reading skills, it is quite troubling that the frequency of homework, as reported by students Table A4.4 and A4.5 below present a summary of the or teachers, is not significantly correlated with scores. bivariate analysis between EGRA score and student or Contrary to the frequency of homework, getting help with teachers variables. Table A4.4: Bivariate correlation between EGRA score and student variables (Figures in red are significant at a 5% level.) Variable Coeff. Std. err. Z P>|z| [95% Conf. Interval] Non-remote schools (vs. remote schools) 0.29 0.09 3.04 0.00 0.10 0.47 Female 0.19 0.04 4.59 0.00 0.11 0.28 Speaks Myanmar at home 0.26 0.06 4.03 0.00 0.13 0.38 Adult reads Myanmar 0.16 0.06 2.86 0.00 0.05 0.27 Attended preschool 0.10 0.05 2.07 0.04 0.01 0.20 Duration of preschool 0.04 0.02 2.15 0.03 0.00 0.08 Not absent last week 0.06 0.05 1.21 0.23 -0.04 0.16 Absent because sick -0.03 0.06 -0.50 0.62 -0.15 0.09 Owns Myanmar reader 0.36 0.06 5.55 0.00 0.23 0.49 Other reading material at home 0.17 0.05 3.44 0.00 0.07 0.27 Other reading material (in Myanmar) at home 0.22 0.05 4.25 0.00 0.12 0.32 Owns Myanmar exercise book 0.07 0.02 3.71 0.00 0.03 0.10 Number of pages used in Myanmar exercise book 0.00 0.01 0.40 0.69 -0.02 0.03 Frequency of marking by teachers 0.17 0.02 7.11 0.00 0.12 0.22 Own (self-written) stories in exercise book 0.28 0.23 1.21 0.23 -0.17 0.74 Homework last week -0.01 0.05 -0.30 0.77 -0.11 0.08 Help with homework 0.14 0.05 2.51 0.01 0.03 0.24 Help (from adult) with homework 0.05 0.04 1.33 0.18 -0.03 0.13 Gets outside tutoring in reading/Myanmar language 0.24 0.05 5.02 0.00 0.15 0.33 Wealth (PCA) 0.08 0.02 4.54 0.00 0.05 0.11 Wealth (sum of durables) 0.05 0.01 4.76 0.00 0.03 0.07 Wealth group (according to SES1) 0.09 0.02 4.41 0.00 0.05 0.13 61 Table A4.5: Bivariate correlation between EGRA score and teacher variables (Figures in red are significant at a 5% level, orange = significant at a 10% level, and green = significant at a 15% level.) Variable Coeff. Std. err. Z P>|z| [95% conf. Interval] Female teacher 0.00 0.10 -0.02 0.98 -0.20 0.20 Multi-grade class -0.09 0.08 -1.09 0.28 -0.24 0.07 Teacher of Myanmar class only 0.13 0.10 1.28 0.20 -0.07 0.32 First language is Myanmar 0.09 0.05 1.74 0.08 -0.01 0.18 Fluent in Myanmar -0.02 0.10 -0.24 0.81 -0.23 0.18 Class taught in other language than Myanmar 0.01 0.03 0.59 0.55 -0.03 0.06 Academic qualifications -0.07 0.05 -1.22 0.22 -0.17 0.04 0-2 years of teaching 0.12 0.05 2.26 0.02 0.02 0.22 Pre-service training in reading 0.18 0.09 1.87 0.06 -0.01 0.36 In-service training in reading 0.07 0.07 1.00 0.32 -0.07 0.21 Teaching certification 0.00 0.02 -0.13 0.89 -0.04 0.04 Absenteeism rate (on survey day) 0.00 0.00 1.49 0.14 0.00 0.01 Use of Myanmar reader -0.06 0.06 -1.01 0.31 -0.18 0.06 Find Myanmar reader useful 0.38 0.02 15.97 0.00 0.34 0.43 Other material used (own) -0.03 0.11 -0.24 0.81 -0.24 0.19 Other material used (Pimya) -0.23 0.15 -1.48 0.14 -0.53 0.07 Other material used (Du Won) 0.05 0.09 0.57 0.57 -0.13 0.23 Other material used (Pan Taing) -0.19 0.09 -2.19 0.03 -0.37 -0.02 School has library, etc. 0.15 0.07 2.23 0.03 0.02 0.28 Frequency of use of the library 0.00 0.03 0.09 0.93 -0.05 0.06 Written tests 0.03 0.06 0.57 0.57 -0.08 0.15 Oral tests 0.00 0.06 0.01 0.99 -0.12 0.12 Lesson recitation 0.04 0.06 0.60 0.55 -0.08 0.15 Homework -0.03 0.11 -0.31 0.76 -0.24 0.18 Chapter-end test -0.22 0.05 -4.40 0.00 -0.32 -0.12 End-of-term evaluation -0.37 0.14 -2.69 0.01 -0.64 -0.10 Other 0.10 0.10 0.98 0.33 -0.10 0.31 Summarize assessment (chapter-end tests or -0.23 0.05 -4.58 0.00 -0.32 -0.13 end-of-term tests) Categorize students (create instructional groups) -0.08 0.07 -1.22 0.22 -0.21 0.05 Grade students -0.07 0.07 -1.04 0.30 -0.20 0.06 Evaluate students’ understanding of subject -0.15 0.07 -2.14 0.03 -0.28 -0.01 matter Identify children who need help -0.12 0.07 -1.76 0.08 -0.24 0.01 Adapt teaching to better suit students’ needs -0.02 0.07 -0.31 0.75 -0.16 0.12 Inform parents/administration of pupils’ -0.18 0.08 -2.41 0.02 -0.33 -0.03 performance Other 0.15 0.16 0.98 0.33 -0.16 0.46 Assessments used to assess understanding -0.11 0.05 -2.09 0.04 -0.22 -0.01 or adapt to needs 62 Frequency of homework 0.00 0.04 -0.06 0.96 -0.07 0.07 Proportion of parents who review homework -0.04 0.04 -1.04 0.30 -0.13 0.04 Students repeat the name of a letter after the -0.02 0.06 -0.34 0.74 -0.14 0.10 teacher models Students repeat the sound of a letter after the -0.01 0.06 -0.21 0.83 -0.13 0.11 teacher models Shown a letter, students give the name of a -0.08 0.07 -1.17 0.24 -0.21 0.05 letter without model Shown a letter, students give the sound of a 0.04 0.06 0.58 0.56 -0.09 0.16 letter without model Students repeat isolated words after the -0.07 0.17 -0.39 0.70 -0.41 0.27 teacher models Students read isolated words without model Teacher reads first, student(s) read/repeat after 0.09 0.11 0.85 0.39 -0.12 0.31 Student(s) read a new, unseen text or passage 0.04 0.05 0.79 0.43 -0.06 0.13 without teacher modeling Engage students in whole class reading -0.09 0.14 -0.62 0.54 -0.36 0.19 Engage students in silent reading -0.01 0.05 -0.14 0.89 -0.11 0.09 Engage students in individual reading -0.29 0.20 -1.41 0.16 -0.69 0.11 Teacher provides text comprehension 0.01 0.05 0.19 0.85 -0.09 0.10 questions and answers first Students answer questions on text after 0.02 0.05 0.33 0.74 -0.08 0.12 Teacher tells a story orally, students repeat the -0.01 0.05 -0.13 0.90 -0.10 0.09 story verbatim Teacher tells a story orally, students retell the 0.06 0.07 0.85 0.39 -0.08 0.21 story in their own words Teacher explains the meaning of new words -0.19 0.20 -0.95 0.34 -0.59 0.20 Students repeat new words -0.03 0.08 -0.32 0.75 -0.19 0.13 Students construct new sentences by using -0.03 0.05 -0.49 0.62 -0.12 0.07 newly learned words Students write compound words using known -0.05 0.05 -1.03 0.30 -0.15 0.05 stem words Don’t know/Refuse 0.00 (omitted) Traditional method of “thin bone gyi” (reading -0.09 0.08 -1.19 0.24 -0.24 0.06 by spelling word by word) Sound-based reading -0.01 0.06 -0.15 0.88 -0.13 0.11 Look and say method -0.03 0.06 -0.48 0.63 -0.15 0.09 Lecture and discussion method 0.00 0.07 0.02 0.99 -0.13 0.13 Questioning method -0.06 0.06 -0.88 0.38 -0.18 0.07 Pictorial story telling method 0.01 0.06 0.24 0.81 -0.10 0.13 Role play method -0.07 0.06 -1.07 0.29 -0.19 0.06 Playing with words 0.05 0.06 0.76 0.45 -0.07 0.17 Others 0.18 0.05 3.26 0.00 0.07 0.28 Times it takes teacher to come to school -0.04 0.03 -1.29 0.20 -0.11 0.02 Teacher has another income-generating job -0.22 0.06 -3.84 0.00 -0.33 -0.11 63 A4.3 Selecting variables for multiariate • Group B— students’ educational environment at analysis home or family and home variables that affect school outcomes, where the Ministry of Education could For the simplest explanation and relevance, only have some influence (e.g., through communication a subsample of the available variables (student, campaigns) teacher, and school characteristics) is included in the multivariate model below. This subsample of • Group C—variables that can account for students’ variables was based on similar bivariate correlations with initial skill levels when they enter primary education and EGRA scores as the one presented above. Without using that serve as a proxy of school readiness formal hypothesis testing, bivariate correlations for each of the possible variables (original responses as well as • Group D—variables that represent student-teacher derived or constructed ones) were reviewed to identify interactions in a classroom setting those with the highest potential as explanatory variables. Due to the generally weak observed relationships between • Group E—school variables that represent education these variables and the EGRA composite scores, the inputs (or education support in schools) and other selection was also influenced by cross-validating the elements affected by education policy correlation between grades. The variables were retained for later analysis if they showed consistent or predictable • Group F—variables that represent teaching methods for relationships with the EGRA composite score across early reading skills, such as letter naming and repetition, different grades.68 Table A4.6 lists and defines the appropriate to non-literate students. explanatory variables examined in these models. • Group G—variables that represent teaching methods To facilitate the selection, these explanatory variables appropriate for more advanced reading skills (at primary were grouped into specific sets, namely, areas level), such as answering comprehension questions and important to the study. creating new words • Group A—variables of explanatory importance that The first column in table A4.6 describes the variable name, cannot be intervened upon, or control variables at the second column describes the data source, and the third the student and teacher level, which describe factors column describes the original question(s) or, for derived contextualizing the students’ educational environment, variables, the calculation of the variable from the original but are not directly amenable to educational practice questions. 68 This criterion is less likely to capitalize on chance relationships, compared to arbitrarily setting alpha levels for significance. 64 Table A4.6: List of variables used in the econometric analysis and corresponding set Set Source Description A Female Student [Are you female?] A Material possessions Student [The number of the durables item in a student’s home] A Absent (sick) Student Were you absent last week [because you were sick]? B Adult reader Student Does a [parent or grandparent] read? B Outside tutoring Student Do you get reading or Myanmar language tutoring outside of school? B Adult help with homework Student [Is there a parent or grandparent at home who helps you in your studies?] C Travel time (to school) Teacher How long does it take you to come to school? C Other job Teacher Do you have to work another job to get enough income for your family? D Preschool (duration) Student How many years did you attend pre-school? D Preschool (at least 2 years) Student [Did you attend two or more years of preschool?] E Exercise book (own) Student Do you have a Myanmar language exercise book? E Exercise book (use) Student [How many pages used in the Myanmar exercise book?] E Exercise book (marking) Student [How many pages have teachers’ marks, corrections, or comments?] E Exercise book (stories) Student Have you written any of your own stories in your exercise book? In your school or classroom, do you have a library, a reading corner, a E Library Teacher book box, or book shelf? [Without being prompted, teacher reports using chapter-end tests or E Assessment (summative) Teacher end-of-term tests.] [Teacher uses assessments to evaluate student understanding or adapt E Assessment (use) Teacher to student needs.] F Isolated words, no model Teacher Students read isolated words without model. F Letter name, model Teacher Students repeat the name of a letter after the teacher models. F Lecture/discussion Teacher Lecture and discussion method F Letter name, no model Teacher Shown a letter, students give the name of a letter without model. F Isolated words, model Teacher Students repeat isolated words after the teacher models. Read/repeat words, after F Teacher Teacher reads first, student(s) read/repeat after. teacher F Repeat story, after teacher Teacher Teacher tells a story orally, students repeat the story verbatim. F Letter sound, model Teacher Students repeat the sound of a letter after the teacher models. F Pictorial story telling Teacher Pictorial story telling method F Letter sound, no model Teacher Shown a letter, students give the sound of a letter without model. F Read text, no model Teacher Student(s) read a new, unseen text or passage without teacher modeling. Retell story (own words), F Teacher Teacher tells a story orally, students retell the story in their own words. after teacher F Silent reading Teacher Engage students in silent reading. F Sound-based reading Teacher Sound-based reading F Whole class reading Teacher Engage students in whole class reading. F Thin bone gyi Teacher Traditional method of “thin bone gyi” (reading by spelling word by word) F Look and say Teacher Look and say method Text comprehension G Teacher Students answer questions on text after. questions, no help Text comprehension G Teacher Teacher provides text comprehension questions and answers first. questions, help G Write compound words Teacher Students write compound words using known stem words. G Repeat new words Teacher Students repeat new words. G Playing with words Teacher Playing with words G Role play Teacher Role play method Construct new sentences, G Teacher Students construct new sentences by using newly learned words. with new words G Questioning Teacher Questioning method 65 Is ol M Le at at -.400 -.300 -.200 -.100 .000 .100 .200 .300 .400 .500 tte e er ia Fe m -.300 -.200 -.100 .000 .100 .200 .300 L n r no d w o lp al os e Le ect am mo rds se tte ure e, de , ss m l Ab io Is r na /dis od ns ol m c se Re at e us el nt ad ed , no sio (s Re /r w n Ad ic k) pe ep ord mo at e a s d e ul st t w , m l Ad O tr ea o ut or o d ul si de Le y, a rds el th el de r tte fte , a f p tu Pi r so r te ter w to rin ith Re c u a Tr a g te Le to n c ve ho ll tte ria d, her m st or r s st mo l lt im ew y o o de e or (o R un ry O (to k w ea d, te l n d l th er sc w te no ling ho or x m jo Grade 1 o b ol Grade 1 ds t, Pr ) ), no de Pr es (te af m l es ch ac te o So r d ch oo he r) un Sile tea el oo l( du d n ch l( 66 Te W ba t re er at ra xt ho se ad le tio as n) co le d in Ex t2 m cl rea g er a Grade 2 ye Grade 2 Te pr ci Figure A4.8: Correlations between variable sets F and G xt eh ss di se ar co en Th re ng Ex bo s) m sio in ad er ok pr n L b in Ex ci (o eh q oo one g er se w en ue k g ci bo n) Figure A4.7: Correlations between variable sets A, B, C, D, and E si sti an yi se W n o o bo ok ns d s Ex (u rit qu , n ay ok se e er ) C Grade 3 (m Grade 3 on co esti o h ci m on el se ar st Re po s, p bo ki ru ng ct pe un he ok ) ne Pl at d w lp (s w ay ne o As to se in w rds se rie nt g ss s) en w wo m ce ith rd en Li s, w s t( br ar w Ro ord su y ith le s m ne pl As m se at w ay ss iv Q wo m e) ue r en st ds t( io us ni e) ng The simple correlations between the retained For variable set F, the correlations are positive for variables and the outcomes are presented in Figure Grade 1 students and negative for Grade 3 students A4.7 and A4.8. In each figure, there are three lines, (with effects for Grade 2 students generally in the middle). representing Grades 1, 2, and 3, which connect points This pattern is consistent with teachers using instructional showing the correlation between each variable and the methods appropriate to younger students when they are outcome for each grade. The lines are illustrative only, to teaching older students with poor reading skills. On the emphasize the relative similarity or difference between the other extreme, we see a possible similar effect (although results for the different grades. Figure A4.7 displays the weaker in magnitude), where more techniques for more correlations for variables sets A, B, C, D, and E. Figure A4.8 advanced readers have a positive correlation for Grade 3 displays the correlations for variables in sets F and G. In students and weaker or negative correlations for students both figures, the variables have been reordered from left to in Grades 1 and 2. In either case, these patterns should right to correspond to the variable sets and to provide clear serve as a warning against simplified interpretations of the illustrations. findings. As is the case with most survey data analysis, any strong inference of causality should be suspended The notable difference between figure A4.7 and figure until more conclusive follow-up case studies or focused A4.8 is that the correlations for variable sets A-E are experimental research can be conducted. almost identical for all grades, even though these variable groups include a wide range of household, A4.4 Analysis of variance decomposition personal, and contextual conditions. Although the ranges of magnitude of the correlations are similar in the Analysis of variance decomposition between the two figures, this consistency of the correlations in Figure different successive models, described earlier (null, A4.7 suggests the presence of commonly observable base, context, control, and inputs), indicates that a phenomena. Indeed, as also discussed in subsection high proportion of the variation between schools is A5.1 (see subheading “Bivariate analysis”), many of the explained by non-educational variables. The potential relationships are intuitive, such as positive correlations effects of attending preschool may already be for female status, wealth, adult support (for reading covered by student and family background variables, and homework), prior attendance in preschool, use of a the use of Myanmar exercise book plays a key Myanmar language exercise book, and availability of a role, and pedagogical and reading activities have library, as well as the negative correlations for teacher travel low explanatory power. Table A4.1 summarizes the time and teachers having other jobs. However, there are variance decomposition in each of the successive models. also several non-intuitive relationships, such as the positive The order of the columns from left to right reflects the relationship for absence due to sickness and the negative sequence of model development: the results for the base correlations with use of summative assessment and the model are furthest left, and those for the interpretation use of assessment results for modifying instruction. These models are furthest right. The top three data rows in table counter-intuitive relationships are a reminder that bivariate A4.1 contain the variance attributable to the school, grade, relationships may be the result of many factors, including a and student (the main random effects in the model). For reversal of the expected direction of causality. each model, there are two columns, “Var.” and “%” as shown. The “Var.” column describes the unexplained The possibility of reverse causality—or education variance of the EGRA scores on its original scale. The practice responding to students’ needs, rather than “%” column contains the relative size of the variance driving student performance—is also a likely factor component. The values are calculated as percentages of underlying the pattern of results of figure A4.8. On the total variance. the left hand side of the figure, the variables correspond to variable group F, the instructional practices more suitable to non-literate students or students just learning to read. The variables on the right side in Figure A4.8 correspond to set G and are typically associated with more advanced students. Table A4.9: Variance decomposition across models Null Base Context Control Set E Set F Set G Var. % Var. % Var. % Var. % Var. % Var. % Var. % School 0.132 19 0.060 45 0.062 47 0.038 29 0.022 17 0.047 36 0.046 35 Grade 0.016 02 0.008 47 0.008 51 0.015 97 0.006 35 0.009 56 0.010 64 Student 0.548 79 0.503 92 0.503 92 0.481 88 0.481 88 0.481 88 0.480 88 67 Because there are no explanatory variables in the null explanation, they do not necessarily explain the model, the sum of the variance components (in the underlying phenomena. The explanatory variables “Var.” column) equals the total sample variance in the in the model may simply be highly correlated with a EGRA score and the sum of the three percentages (in phenomenon that is in turn highly correlated with school the “%” column) equals 100. For all other models, the effectiveness without actually having any relationship to values are calculated as a percentage of the corresponding school effectiveness themselves. For this reason, it is variance component of the null model and the percentage useful to examine the variable coefficients themselves, reflects how well the model explains differences between as the patterns may suggest what some of the true schools, grades, and students. underlying causes may be. However, in many cases, the variables represent proxies for conditions that the study The null model indicates that 19 percent of the cannot directly measure, despite a wish to. Accordingly, variation in EGRA scores is a result of school the magnitude of the effect sizes (coefficient) in the results differences and only two percent is attributable to should not be directly interpreted. Rather, they should grade. In other words, knowing which school a student be taken as lower-bound estimates that indicate the attends is five times better at predicting a student’s reading direction of association between the real world conditions performance than knowing what grade a student is in.69 they represent and the outcome of student reading Accordingly, reviewing the results, the base model alone, performance. which does not include any educational variables, explains over half of the variation between schools. The remaining For the most part, the results of this multivariate school level variance is only 45 percent of the original. analysis reflect the observations in the bivariate analysis, although with some differences—namely In contrast, the base model only explains eight the vanished correlation between scores and adult percent of the student-level variance. The context help with homework, teacher travel time, and model offers no improvement on the base model enrollment in preschool. The estimated coefficients because the potential effects of attending preschool from the base, context, and control models are shown are already covered by student and family background in Table A4.2. (The null model is not included because variables. Also, the control model again provides no coefficients are estimated for it.) The effects of all substantially more explanatory power of school differences: the student-level variables in the base model also vary the additional 18 percent of school-level variance is primarily significantly across schools. This indicates that, despite explained by the use of the Myanmar exercise book. Of consistency of the effects across grades, there may still the last three models, only set E offers any improvement be ways for schools to ameliorate the effects of student over the control model, indicating that 83 percent of the background. Female status and wealth are still relatively school-level variance is accounted for. Sets F and G actually strong predictors. Interestingly, the effects of adults provide worse explanatory power relative to the control reading and providing extra lessons (outside tutoring) are model, likely due to lack of degrees of freedom to fit the relatively strong and positive, effectively explaining away larger numbers of coefficients with the data. the effect of adult help on homework. Teacher travel time, which also had a strong (but negative) bivariate relationship, A4.5 Coefficient Analysis also has a null effect when controlling for other variables. This most likely is because the effects of travel time are As with the bivariate analysis results, some caution explained by socioeconomic factors related to school is required in the interpretation of the variance surroundings, which are already captured by student decomposition since, despite providing statistical family background. 69 Although the student variance component is relatively large, this is not problematic. Since the purpose of the analysis is to explain differences between schools, we are more concerned with how the models reduce the school variance component. 68 Table A4.10: Estimated coefficients across base, context, and control models Base Context Control Female 0.19 *** 0.19 *** 0.17 *** Speaks Myanmar at home 0.23 *** 0.23 *** 0.21 *** Non-remote 0.12 0.11 0.12 Wealth (durable) 0.03 *** 0.03 *** 0.03 *** Absent (sick) -0.02 -0.02 -0.02 Other job -0.19 *** -0.20 *** -0.21 *** Adult reader 0.13 *** 0.14 *** 0.13 *** Outside tutoring 0.18 *** 0.17 *** 0.15 *** Adult help with homework 0.03 - - Travel time (to school) -0.04 - - Preschool duration - 0.02 0.02 Preschool for at least 2 years - -0.06 -0.06 Exercise book (own) - - 0.18 ** Exercise book (use) - - 0.03 * Exercise book (marking) - - 0.14 *** Exercise book (stories) - - 0.40 *** * significant at 1%; ** significant at 10%; *** significant at 5%. Note: Not all variables are included in all models. Where an effect is missing to the right of its first appearance in the table, the variable was removed from subsequent models because they exhibited chance relationship (effect was not significantly different from 0). Of particular note in the base model is the large Table A4.2 summarizes the corresponding results for effect of teachers having another job, which is the three variables groups describing instructional strongly negatively related to student performance. practice. Only group E variables had additional significant The negligible effects of attending preschool indicate that relationships to the outcome, consistent with the bivariate preschool itself is likely strongly correlated with other strong relationships observed. The fact that these relationships predictors of performance, such as family background. In persist after inclusion of the control variables suggests that the control model, there are relatively strong effects related these effects are unrelated to school intake and likely have to use of the Myanmar exercise book. Although owning an a direct effect on performance, despite the non-intuitive exercise book has a significant relationship by itself with nature of the assessment-related effects. In contrast, the performance, each additional variable describing the use effects of variables in groups F and G are consistently of the book (quantity of use, teacher feedback, students negligible after the inclusion of the control variables, despite writing original stories) is associated with additional the interpretable pattern seen in the exploratory analysis. performance increases. 69 Table A4.11: Estimated coefficients across model with variable sets E, F, and G Set F Set G Set E Pedagogy for poor Pedagogy for good Educational Inputs readers readers Female 0.17 *** 0.16 *** 0.17 *** Speaks Myanmar at home 0.22 *** 0.21 *** 0.20 *** Non-remote 0.11 0.10 0.12 Wealth (durable) 0.03 *** 0.03 *** 0.03 *** Absent (sick) -0.01 -0.03 -0.02 Other job -0.17 *** -0.24 *** -0.25 *** Adult reader 0.13 *** 0.13 *** 0.13 *** Outside tutoring 0.16 *** 0.16 *** 0.15 *** sq18_homeworkadulthelp - - - tq_30travel_time - - - Preschool duration - - - Preschool for at least 2 years - - - Exercise book (own) 0.18 ** 0.16 * 0.18 ** Exercise book (use) 0.03 * 0.03 * 0.03 * Exercise book (marking) 0.13 *** 0.14 *** 0.14 *** Exercise book (stories) 0.45 *** 0.40 *** 0.39 *** Library 0.15 *** - - Assessment (summative) -0.20 *** - - Assessment (use) -0.11 *** - - Isolated words, no model - -0.07 - Letter name, model - 0.06 - Lecture/discussion - -0.03 - Letter name, no model - -0.13 ** - Isolated words, model - -0.04 - Read/repeat words, after teacher - 0.05 - Repeat story, after teacher - -0.01 - Letter sound, model - -0.04 - Pictorial story telling - 0.04 - Letter sound, no model - 0.09 - Read text, no model - 0.05 - 70 Retell story (own words), - 0.07 - after teacher Silent reading - -0.03 - Sound-based reading - 0.01 - Whole class reading - -0.09 - Thin bone gyi - -0.03 - Look and say - -0.01 - Text comprehension questions, - - -0.01 no help Text comprehension questions, - - -0.02 help Write compound words - - -0.07 * Repeat new words - - -0.06 Playing with words - - 0.03 Role play - - 0.03 Construct new sentences, - - 0.03 with new words Questioning - - -0.08 * significant at 1%; ** significant at 10%; *** significant at 5%. Note: As previously explained, note that although the control variables remain the same for each model, the variables from sets E, F, and G effects are entered and removed in block form. 71 THE WORLD BANK 30th Floor, Siam Tower, 989 Rama 1 Road, Pathumwan, Bangkok 10330 Tel: (66) 0-2686-8300 Fax: (66) 0-2686-8301 E-mail: thailand@worldbank.org www.worldbank.org/th