72347 v2 eading ra d e R ar ly G n ga E m en t To Assess line Survey ) Bas e GR A (TE R e s u lt s R e p o r t ©2012 International Bank for Reconstruction and Development / International Development Association or The World Bank 1818 H Street NW Washington, DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this work is subject to copyright. Because The World Bank encourages dissemination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. For permission to reproduce any part of this work for commercial purposes, please send a request with complete information to the Copyright Clearance Center Inc., 222 Rosewood Drive, Danvers, MA 01923, USA; telephone: 978-750-8400; fax: 978-750-4470; internet: www.copyright.com. All other queries on rights and licenses, including subsidiary rights, should be addressed to the Office of the Publisher, The World Bank, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2422; email: pubrights@worldbank.org Since going to press, the Ministry of Education, Women’s Affairs and Culture in Tonga has changed its name to the Ministry of Education and Training. For consistency and referencing, the Ministry’s former name is used in this work. Table of Contents Table of Contents.......................................................................................................................................... 1 List of Tables................................................................................................................................................. 3 List of Figures................................................................................................................................................ 4 ABSTRACT................................................................................................................................................... 5 .............................................................................................................................. 6 ACKNOWLEDGEMENTS. ................................................................................................................................ 7 EXECUTIVE SUMMARY. .............................................................................................................. 8 Survey results and main findings. ........................................... 9 Factors contributing to greater reading fluency and comprehension in Tonga. At the student level.............................................................................................................................. 9 At the teacher level............................................................................................................................ 10 From assessment to intervention: next steps.......................................................................................... 12 Structure of the Report........................................................................................................................... 14 Chapter 1- Introduction........................................................................................................................... 15 Chapter 2: Survey Implementation......................................................................................................... 17 Sample Design....................................................................................................................................... 17 Development of the TEGRA Instrument.................................................................................................. 17 ....................................................................................................................... 19 Fieldwork and Data Entry. Reliability of the Instrument..................................................................................................................... 19 Chapter 3: TEGRA Results...................................................................................................................... 20 .................................................................................................................. 20 Structure of the Assessment. ................................................................................................ 21 Administration of the TEGRA Instrument. TEGRA Results per Sub-test.................................................................................................................. 24 Sub-test 1 – Letter Name Knowledge................................................................................................ 24 Sub-test 2 – Initial Sound Recognition............................................................................................... 25 Sub-test 3 – Letter Sound Identification............................................................................................. 26 1 .................................................................................................. 28 Sub-test 4 – Familiar Word Reading. Sub-test 5 – Invented Word Reading................................................................................................. 29 Sub-test 6a – Oral Passage Reading................................................................................................. 30 Sub-test 6b – Reading Comprehension............................................................................................. 31 Sub-test 7 – Listening Comprehension.............................................................................................. 34 Sub-test 8 – Dictation........................................................................................................................ 36 Summary of Assessment Results........................................................................................................... 38 Chapter 4: Performance in Oral Reading Fluency and Reading Comprehension. ........................... 39 Fluency and comprehension................................................................................................................... 39 Chapter 5: Analysis of Student Factors Associated with Better Reading Scores............................. 43 Chapter 6: Analysis of Teacher Factors Associated with Better Reading Scores............................. 51 Effect of Teacher Characteristics on Student Performance................................................................. 52 Effect of Teacher Expectations on Student Performance.................................................................... 52 Effect of Teacher Instructional and Assessment Methods................................................................... 55 Chapter 7 - Next Steps............................................................................................................................ 59 .......................................................................................................................... 62 Bibliographical References. ANNEX 1 / TABLES. ................................................................................................................................. 64 ANNEX 2 / READING INSTRUCTION IN TONGA. .................................................................................. 68 ANNEX 3 / TONGAN LANGUAGE............................................................................................................ 71 ANNEX 4 / DEVELOPMENT OF TEGRA INSTRUMENTS. ..................................................................... 72 2 List of Tables Table 1 - TEGRA Sample by school type, class level and gender............................................................... 18 Table 2 - TEGRA Sample by region, class and gender............................................................................... 18 Reliability of TEGRA assessment................................................................................................. 19 Table 3 - TEGRA Instrument Structure and Early Skills Tested. Table 4 - ................................................................... 22 Table 5 - Zero-score cases per class and sub-test.................................................................................... 23 Sub-test 1 Letter Name Knowledge: Results by Class and Gender............................................. 25 Table 6 - Table 7 - Sub-test 2 Initial Sound Recognition: Results by class and gender.............................................. 26 Table 8 - Percentage of correct initial sounds identified per item in the sub-test, by class.......................... 27 .......................................... 27 Table 9 - Sub-test 3 Letter Sound Identification: Results by class and gender.. ............................................... 28 Table 10 - Sub-test 4 Familiar Word Reading: Results by Class and Gender. Sub-test 5 Invented Word Reading: Results by Class and Gender.............................................. 29 Table 11 - Sub-test 6a Oral Passage Reading: Results by Class and Gender.............................................. 30 Table 12 - Sub-test 6b Reading Comprehension: Results by Class and Gender.......................................... 31 Table 13 - Percentage of Correct Answers by Questions in Sub-test 6b...................................................... 33 Table 14 - Percentage of Correct Answers in Sub-test 6b. Table 15 - ........................................................................... 34 Table 16- Sub-test 7 Listening Comprehension: Results by Class and Gender........................................... 35 Percentage of Correct Answers by Questions in Sub-test 7........................................................ 36 Table 17 - Table 18 - Sub-test 8 Dictation: Results by Class and Gender..................................................................... 37 Distribution of Students by Fluency............................................................................................. 40 Table 19 - ......................................... 41 Table 20- Percent Distribution of Students by Fluency (no zero-score students). ................. 41 Table 21 - Average Fluency Levels (Fluent Vs. Less than Fluent) Excluding Zero-score Students. Average Reading Comprehension (Fluent Vs. Less than Fluent) Excluding Zero-score Students. Table 22 - . 42 Characteristics of students in the sample along several student and family factors...................... 43 Table 23 - Summary of ANOVA Results by Student Factors. Table 24 - ........................................................................ 44 - Results from differences in average scores determined by factor (tables 5.1 to 5.10)................... 45 Table 25 ...................................................................................................... 51 Table 26 - Profile of Teachers in TEGRA. 3 - Regression analyses of teacher characteristics on student literacy performance.......................... 52 Table 27 Table 28 - Regression analyses of average effects of teachers’ expectations on student literacy performance.................................................................................................. 54 Table 29 - Regression analyses of average effects of frequency of instructional methods on student literacy performance.................................................................................................. 56 Table 30 - Regression analyses of average effects of frequency of assessment methods on student literacy performance.................................................................................................. 57 ............................................................................................................. 64 Table 31 - TEGRA Reliability Matrix. Table 32 - ANOVA Results: Differences in Means across Sub-tests (ST)...................................................... 65 Table 33 - Frequency of methods of instruction used by Tongan teachers in the early grades...................... 67 List of Figures ................................................................................................. 20 Figure 1 - Stages of Reading Development. ......................................................................... 21 Figure 2 - Early Grader Reading Assessment Components. ................................................................. 32 Figure 3- Distribution of percentage of correct answers by class. ............................................................. 32 Figure 4 - Distribution of percentage of correct answers by gender. Figure 5 – Percentage of Correct Answers in Each Task of Sub-test 8 by Class and Gender....................... 38 Figure 6 – Percentages of Average Reading Comprehension by Number of Correct Words Read Per Minute . in the Oral Reading Passage....................................................................................................... 39 ....................................................... 40 Figure 7 – Average Reading Comprehension Levels in Fluent Students. 4 ABSTRACT As part of the Government of Tonga’s effort to improve reading levels, an early grade reading baseline assessment was conducted in November, 2009 with support from regional education partners. Survey results are cause for concern. While most students develop some fundamental skills in grades Classes 1, 2 and 3, only 3 in 10 students at the end of Class 3 are able to develop fluency in reading, ability strongly related to reading comprehension. Factors that are shown to be predictors of better reading performance in the early grades include: having literate parents and/or siblings, having books at home, and receiving help from a family member to do homework. Among teachers, having and using the recommended Tongan reading texts demonstrated a statistically significant association to better reading performance. In regards to teacher instructional methods, from the seven reading instructional activities, only frequent story retelling associated to better reading outcomes. Finally, in terms of teacher expectations, having high expectations –i.e., above median value- about student reading outcomes did not render significant results; however, if a teacher had low expectations –i.e., below median- about when a student should demonstrate a given skill, low expectations appeared to negatively affect student performance at statistically significant levels. Based on the analysis presented in this report, it is recommended that Tongan educators address reading deficits through interventions that provide additional support to teachers to improve their practice, increase the exposure of children to books and other reading materials beyond the classroom, and promote greater parental involvement in the reading development of their children. 5 ACKNOWLEDGEMENTS The Tonga Early Grade Reading Assessment (TEGRA) baseline survey is the result of the Government of Tonga’s commitment to improve reading levels in the country through a mid-term process incorporating assessments, improvements in reading instruction and greater parental and community involvement. This report jump-starts such process -- the first among Pacific island countries – by providing Tongan education officials with a system-level diagnosis of how well – and at what pace -- Tongan children are developing foundational skills needed to become literate. The report was prepared by Myrna Machuca-Sierra (Education Specialist) and James A. Stevens (Senior Operations Officer) of the World Bank’s East Asia and the Pacific Education Unit (EASHE). Jose Ramon Laguna and Margaret Triyana provided support during data cleaning and validation, and the analysis of results. Ms. Heti Veikune and Ms. ‘Ungatea Kata contributed with technical annexes on reading development in Tonga, the orthography of the Tongan language, and the development of the TEGRA instruments. The report benefitted from useful comments from Eduardo Velez-Bustillo (Education Sector Manager, EASHE), Stephen D. Close (Human Development Specialist, EASHE), and Eleanor Wang (Junior Professional Associate, EASHE). Amber Gove (Senior Education Research Analyst, RTI International), and Souhila Messaoud-Galussi (Lecturer, University College London / City University) provided expert commentary and guidance during survey design and administration. Stakeholders in Tonga, including officials from the Ministry of Education, Women’s Affairs and Culture, are expected to provide further comments and advice. Errors or omissions are the authors’ sole responsibility. The authors would like to express their gratitude to the Tongan Ministry of Education, Women’s Affairs and Culture, in particular to the Hon. Min. Dr. Tevita Hala Palefau and Dr. Viliami Fukakuoa (Director General of Education), for their leadership in this project. The TEGRA Survey received great support from senior MEWAC officials including Ms. Peaua Heimuli (Deputy Director of Schools), Dr. Raelyn L’Esau (Deputy Director of Policy and Planning) and Dr. Ana Taufe’ulungaki (Senior Advisor, Tonga Education Support Program). In particular, the authors would like to acknowledge the work of the TEGRA core team members, enumerators and supervisors whose hard work and dedication made the TEGRA survey possible: Ms. ‘Ungatea Kata, Ms. Siosiana Tapueluelu, Ms. Heti Veikune, Ms. Momea Tuifua, Mr. Colin Lutui, Ms. Malia Falesiva, Ms. Matelita Taufa, Ms. Lute ‘Aipolo, Ms. Faleola Selupe, Ms. Penelatita Tuipulotu, Mr. ‘Etimani Soakai, Ms. Seini Launga, Ms. Leti Tangi, Ms. Ikatonga Vaka’uta, Mr. Ifalemi Malafu, Ms. Manutala’aho Tupou, Ms. Tol’ia Fifita, Mr. Saia Vehikite, Ms. ‘Olivia Katoa, Ms. Fipe Siale Latu, Ms. ‘Emalata Manukeu, Ms. Mele Tavite Fisiliu, Mr. Setikia Vea, Ms. Penina Kailomani, Ms. Kalesita Hakaumotu, Ms. Kakatisi Taulava, Mr. Salesi Faasolo, Ms. Suliana Vea, Ms. Lupe Mahe Latu, and Mr. Semisi Tongia. Last but not least, the authors would like to thank the 1,203 young Tongan students who enthusiastically participated in the survey. To all, mÄ?lo aupito. 6 EXECUTIVE SUMMARY This report summarizes the results of a Tongan language early grade reading assessment conducted in a sample of 1,203 students from 60 randomly-selected primary schools in the Kingdom of Tonga. The Tonga Early Grade Reading Assessment (TEGRA) was carried out in November, 2009 by consultants and staff of the Ministry of Education, Women’s Affairs and Culture (MEWAC) with financial support from the Australian Agency for International Development (AusAID), the New Zealand Agency for International Development (NZAID), and the Education for All – Fast Track Initiative (EFA-FTI). Technical assistance and management support was provided by the World Bank. The assessment, the first among Pacific island countries, is part of a global initiative aimed at helping countries measure how well children are learning to read in the early grades of primary education. It aims to help educators develop local knowledge about the specific skills students are struggling with and the factors that appear contribute to reading development in their country. Equipped with such evidence, education stakeholders can come together to devise response strategies to improve reading instruction, monitor student progression in the classroom, and promote greater parental and community involvement to ensure all children develop the skills needed to become effective readers. The TEGRA assessment consisted of eight modules or sub-tests covering basic reading skills such as phonemic awareness, recognition of letter names and sounds, automatic word reading, decoding, oral reading fluency, and comprehension – both in terms of reading of and listening to short narrative passages. A short dictation exercise was included to test early writing skills such as spelling, orientation to write, spacing, capitalization, and punctuation. The TEGRA student test was complemented by a student contextual interview which collected information about socioeconomic characteristics such as availability of reading books at home and literacy prevalence among family members. TEGRA also included a teacher questionnaire which gathered data on teacher qualifications, expectations and methods of instruction and assessment. All survey instruments were developed in Tongan, the national language, and the language of instruction in the first three years of primary education. The reliability of the instrument to capture reading abilities in Tonga was estimated using Cronbach’s alpha – a common measure of reliability of survey instruments- at a coefficient of 0.9 – the minimum coefficient acceptable in research is 0.7. Because TEGRA instruments were developed in Tongan to suit language and reading performance in the country, results are not meant for cross-country comparison, but to assist MEWAC to establish reference reading benchmarks in the Tongan language against which improvements in reading outcomes can be measured. The analysis of TEGRA student data included descriptive statistics (means and standard deviations) to measure average levels in basic reading skills; an Analysis of Variance was carried out to determine the statistical significance of differences in average scores between groups with and without factors identified as predictors of reading. Finally, regression analyses were carried out to estimate the association of a given teacher characteristic and student reading outcomes. The most relevant results are presented below, followed by policy implications on reading instruction and teacher professional development in the country. 7 Survey results and main findings The TEGRA survey results showed evidence of good and poor performance. While some basic foundation of pre-reading skills is established for the majority of children, students lack the ability to distinguish the sounds of letters in the context of a word and do not understand the relationship between letters and their sounds. Poor knowledge of how letters and sounds go together to make words, appears to be one of the main reasons why students struggle to read words both in isolation and in the context of a text, and may be contributing to hinder the development of oral reading fluency. Failure by many children to read narrative passage fluently to focus on meaning puts in doubt their ability to cope with an increasingly difficult curriculum in later grades. Some specific survey findings are summarized below. Average scores across sub-tests show evidence of both good and poor performance among Tongan students. On average, students demonstrated a good knowledge of the names of letters, were able to extract meaning from a short story they listened to, and were able to spell some words correctly and use some of the basic writing conventions appropriately in a 10-word sentence dictated to them. By contrast, students lack the ability to recognize the initial sound in a word, to understand letter-sound correspondences in the Tongan language, and to distinguish Tongan letter-sound patterns in invented words. Poor knowledge of letter-sound correspondences may be hindering the development of automatic word reading and oral reading fluency. Low scores are partly due to the sizeable number of students for whom the test had to be discontinued due to the lack of the minimum knowledge tested. In TEGRA, student performance in timed-sub-tests could be considered “early stop casesâ€? if a particular sub-test had to be discontinued due to the student’s inability to read the minimum number of letters or words needed. Early-stop cases –i.e., zero-score students- serve as a measure of the number of students with the lowest score possible. In TEGRA, the proportion of zero-score students is larger in subtasks dealing with letter-sound correspondences and word reading. 21% of the students in the sample were unable to recognize the correct sound of at least one in the first 10 letters presented in a page with 100 randomly distributed letters. 35% of the students were unable to identify the initial sound in a word from a list of 10 one and two-syllable words, signaling very low levels of phonemic awareness -an essential skill in reading acquisition. Similarly, 13% and 15% of the students in the sample were unable to read familiar and invented words, respectively. Early-stop cases were allowed in all sub-tests except in two -listening comprehension and dictation. Achieving oral fluency in reading is crucial to improve reading comprehension among students. An analysis of oral reading fluency and reading comprehension levels among Tongan students showed that Tongan students achieve greater levels of reading comprehension (between 60% and 100%) when they read at an average rate of at least 50 correct words per minute in the oral reading passage. At this standard, only 192 out of 1,203 students (17% of the sample) could be considered fluent in reading. Differences in reading comprehension between fluent and less-than-fluent students are important: while fluent students read an average of 67 correct words per minute and were able to understand about 80% of the text they read; less- than-fluent students read at an average pace of 23 correct words per minute for an average comprehension of only 16%. 8 Low oral fluency levels in Tongan early graders suggest only few students are able to read with enough fluency to extract meaning. Students do increase their reading fluency from Class 1 to Class 2 and from Class 2 to Class 3, but average levels are low and progress is modest. Using a standard of reading at least 50 words per minute correctly in the oral reading passage, only 2% of Class 1 students, 15% of Class 2 students, and 34% of Class 3 students were able to demonstrate enough reading fluency to extract meaning from the text they read. This is the same as only 3 in 10 Class 3 students who after three years of primary education are able to read with enough fluency to focus on comprehension. More girls appear to learn to read fluently than boys do. In terms of fluency, 54 out of 596 boys (9% of the male sample) were able to read 50 or more correct words per minute in the oral reading passage whereas 152 of 607 girls (25% of the female sample) achieved the fluency standard. Also, survey results provided evidence of a gender gap in all the skills tested. Factors contributing to greater reading fluency and comprehension in Tonga At the student level We explored the association between student characteristics and reading performance using student responses to a series of questions about their family and personal characteristics such as attendance to kindergarten before primary school enrollment or prevalence of literate family members, among others. Students with literate parents, with a literate sibling, who had books at home, and those who do homework and receive help from a family member showed better reading outcomes in some of TEGRA sub-tests. Some of these differences are even statistically significant across gender. Also, students who attended kindergarten, were overage - as a proxy for repetition and were absent for more than one week in the school year performed worse in some sub-tests, and some of these differences are also significant across gender. A summary of this analysis is presented below. On family literacy: • Across classes, students with both literate parents performed better when asked to identify the initial sound in a word (sub-test 2) than those without. These differences were also significant across gender, especially for boys where the differences were statistically significant at the 1% level. • Having at least one literate sibling is positively associated with better performance in sub-test 2 across classes and gender; for boys, having a literate sibling is also associated to better reading performance in all sub-tests. • Family literacy seems to play a greater role on the reading outcomes of boys than of girls. Having literate parents was associated to better reading performance among boys in half of the sub-tests included in TEGRA whereas the presence of a literate sibling was associated to better performance in all sub-tests. For girls, literate parents only associated to better student performance in sub-test 2 whereas literacy of siblings associated to sub-test 2 and 5, only. 9 On the availability of books at home: • Students with books at home -other than school books such as school readers and/or text books- showed better reading performance than those without. This difference was statistically significant only for Class 3 students in word-level sub-tests. In the oral reading passage, the average difference in reading performance between students with books at home and those without is 7 words read per minute. • Having books at home was associated with better performance for girls and boys, in almost all sub-tests except identification of the initial sound in a word, letter sound recognition, and invented word reading –only for boys On doing homework and receiving help from a family member to do it: • On average, students that reported doing homework had better performance than those reporting they did not. The differences were statistically significant for two out of three class levels. For Class 1 students, doing homework was associated to better average scores in three sub-tests: letter- name knowledge, invented word reading and the oral reading passage. For Class 2 students, doing homework is associated to the same sub-tests that showed significance in Class 1 plus the sub-test assessing familiar word reading. Differences in student performance brought about by doing homework were not statistically significant in any sub-test. • Doing homework appeared to have a positive and statistically significant effect on boys’ performance in all but one sub-test –initial sound recognition. For girls, doing homework was only significant in the learning of the names of letters. • Receiving help with homework from a family member appeared to improve performance in boys’ knowledge of letter names, familiar word reading, and reading ability in the oral reading passage, but showed no significance to affect girls’ performance in any TEGRA sub-test. At the teacher level We also explored the association between teacher characteristics and student performance using data collected through the teacher questionnaire on experience, certification, methods of instruction and assessment, and learning expectations. Interestingly, teacher experience and certification exerted almost no effect on oral reading fluency. The same was observed if the teacher reported attending in-service teacher training on general content or reading instruction. This could be possibly related to the small number of in- service courses offered in the last 3 years (2 courses on average), the scope of such courses, or the small proportion of teachers from the sample that attended (37%). A summary of this analysis is presented below. Among teacher characteristics, only two of them (having the recommended reading texts and using them in the classroom) were found to be associated with better student performance: • Students, whose teachers have the recommended Tongan reading text, on average, read 4 more words than those whose did not. 10 • Students, whose teachers use the recommended text, on average, read 7 words more than students whose teachers do not use the text. In terms of teacher expectations to the appropriate class level in which students should develop certain basic reading skills: • On average, teachers expect students to write their name, recognize and say letter names, recite the alphabet, and understand stories they hear by the end of Class 1; to understand stories they read and sound out invented words by the end of Class 2; and to read aloud with few mistakes by the end of Class 3. • Reading outcomes in Tonga are consistent with these expectations. Students whose teachers think students should understand stories they hear by Class 3 instead of class 1 read an average of 4 fewer words. Students whose teachers think students should recite the alphabet in Class 3 instead of Class 1 read an average 5 words fewer. Students whose teachers think reciting the alphabet is not important at all read about 17 words fewer in the oral reading passage than students whose teachers considered knowledge of the alphabet to be a skill to be developed in the first three grades of primary education. Finally, in terms of the methods used during reading instruction and assessment: • Most teachers ask students to repeat sentences, retell a story they read, sound out invented words, learn meanings of new words, read aloud, or perform silent reading between 1-2 days a week to daily. Few teachers require students to copy text from the board. • From these methods, only retelling of a story was associated with higher levels of oral reading fluency. Students who retell a story either 2-4 days per week or daily read on average 5 and 4 more words per minute respectively relative to students who never retell a story they read. • Students that copy text from the chalkboard, learn the meaning of new words or perform silent reading, on average, had lower reading performance than those whose teachers never assign the tasks. These results may be related either to the small proportion of students doing these activities or the way in which the activity is typically carried out in the classroom. • In terms of assessment, most teachers provide written and oral evaluations, review portfolios and projects, copy books and homework. Average student scores associated to these methods were low probably because of the way and the type of feedback provided to students. These issues were outside of the scope of the survey. 11 From assessment to intervention: next steps Survey results call for an immediate response to improve reading instruction and ensure all Tongan children are equipped with the abilities needed to become skilled readers. Specific recommendations to be considered are as follows: • Improve instruction in Tongan phonics and increase phonemic awareness levels among students. TEGRA data suggested students were either not receiving any instruction to develop phonemic awareness or this training was not sufficient or appropriate. Survey results suggest this may also be the case of instruction of letter-sound correspondences. Without the ability to distinguish the sounds in words and a solid understanding of how letters and sounds go together to create words, students are bound to struggle in their reading development paths and have a limited understanding of what they read. However, improvements in reading instruction should not focus on phonics exclusively but should be incorporated into a comprehensive system of reading instruction aimed at developing reading skills as early and comprehensively as possible. • Better reading instruction calls for additional support to teachers. Improvements to reading instruction start with additional support to have shape more informed expectations about reading development in the early grades. TEGRA results identified how low expectations on reading outcomes had a negative impact on student performance, especially when teachers believed specific fundamental skills should be developed in later years. Better informed expectations on reading outcomes should be coupled with information the specific methods, classroom activities and assessment methods that can allow teachers improve their practice through clear instruction and monitoring indicators. • Teacher support should be constant. TEGRA survey results showed how only 27% of the teachers had received any in-service training on reading instruction in the last three years. If less than 3 out of 10 teachers in the country benefit from learning about specific ways in which they can improve their practice, Tongan teachers will continue to practice their profession in isolation. Moreover, teachers need to become aware of the fundamentals of reading development in both pre- and in-service training. The current syllabus for primary teachers at the Tongan Institute of Education (TIOE) has no mention of grade-level methods or expected reading outcomes in the early grades. Specific reading instruction training should be a centerpiece of the Tonga Education Sector Program (TESP) Phase II. • Increase the availability of books and other reading materials to children to read at home. Survey results showed how students who have reading books at home have better reading outcomes and are more likely to become fluent. Having access to books at home and the possibility to read and develop a sense of enjoyment will be critical to ensure students engage in reading on a regular basis. While English language books tend to be available, most reading materials in Tongan are limited and complex for beginning readers –e.g., newspapers and the Bible. Production of reading books and stories in Tongan written by Tongans would be important not only to allow students to read child- appropriate stories but to preserve the rich oral tradition of Tongan story tellers. • Introduce policy actions that increase student exposure to literacy outside the school. Making more books available to students will not per se ensure better reading outcomes. Along with access 12 to more and relevant reading materials in Tongan, children will need support to develop a reading habit beyond the requirements of the school curriculum. One way of improving this would be to develop community literacy programs where schools become focal point of literacy in the community. Building on the success created by the school-grants program, schools can encourage parents and literate siblings –a predictor of better reading scores in Tonga- to become more involved in the reading development of young children by participating in “peer readingsâ€? and activities reading clubs both in school and at home. • Establish reading standards to monitor improvements across classes. As MEWAC moves on to establish an oral reading fluency benchmark for the early grades to monitor reading improvements, it is important to consider that initial standards should be considered temporary reference standards since not enough is currently known about when and at what rate Tongan children should progress when learning to read in their own language. In this sense, reading standards should not be seen as high-stakes but an essential piece to monitor reading progression in the classroom. One way of doing this would be to use the percentage of zero-score students in selected sub-tests as a marker and track reductions in the shares at least biannually. Monitoring achievements over time will eventually provide more information on the rate and the way in which average fluency develops in the Tongan language. A modified version of the test could be used to screen students during the school year in need of additional support. • Support teachers to establish reading goals for individual students which they can monitor throughout the school year. In order for teachers and schools to be able to be held accountable for reading outcomes, teachers, school officials and parents need to understand what these standards mean and how each can support reading development in their own school in order to reach national standards. School development plans should contain reading improvement goals as part of their minimum service standards, as well as a description of activities aimed at encouraging reading. Parents and the community as a whole should be brought into this effort. • More research is needed on how instruction methods and activities are carried out in the classroom. Many of the teacher factors traditionally associated to better reading outcomes such as teacher certification or learning of new words showed contradictory or counterintuitive results in Tonga. Although there are several potential explanations for this phenomenon, this calls for more research on how instruction takes place in the classroom. For example, average reading scores among students whose teachers promote the learning of new words are lower than those whose teachers rely less in this activity. A question that could be look further into would have to analyze how learning of new words takes place –e.g., through explicit decoding and association with similar vocabulary or by memorization- in order to recommend specific improvements to current instructional methods. • Last but not least, it is clear that more research is needed to better understand the factors that contribute to differences in reading performance between boys and girls. An analysis of the factors that contribute to these differences is beyond the scope of this survey. However, it is cause of great concern that boys are already in academic disadvantage after completing just one year of education. A better understanding of this phenomenon is critical to inform sector policies and increase the success of future reading development programs. 13 Structure of the Report Chapter 1 briefly presents the purpose of the survey and how TEGRA results are expected to improve learning outcomes in Tonga. Chapter 2 summarizes the overall implementation of the survey, in particular, on the process followed to develop the TEGRA instrument. Chapter 3 presents the main results from each of the sub-tests administered. Chapter 4 presents the analysis of oral fluency and reading comprehension levels and the discussion on the establishment of a reference standard of oral reading fluency in the country. Chapter 5 and Chapter 6 present the results of the analysis of student and teacher factors associated with reading acquisition among Tongan students. Finally, Chapter 7 draws conclusions about the key results to present possible recommendations for improving the quality of reading instruction in Tongan schools. 14 1 Chapter 1- Introduction In 2007, the Tongan Ministry of Education, Women’s Affairs and Culture (MEWAC) administered the Standardized Test of Achievement of Tonga (STAT), a national assessment of literacy and numeracy skills for students in Class 4 and Class 6 of primary education1. The test, administered in both English and Tongan, aimed to evaluate if students were developing the literacy and numeracy skills expected at their grade levels. STAT reading results revealed the need for major improvements on reading comprehension and writing composition in both languages and grade levels (MEWAC, 2008). Although informative of the overall status of reading in the country, STAT results provided a limited diagnosis. Since the test only measured reading comprehension and composition –two skills expected of Class 4 and Class 6 students-, it is difficult to know if STAT results are due to the lack of assessed knowledge or to the lack of basic reading and comprehension skills typically developed before Class 4. To complement STAT data and inform literacy improvements, the Tongan government sought to learn if students in Class 1 to Class 3 develop the basic reading skills needed to read and understand what they read. If the poor performance in STAT could be linked to low mastery of specific basic reading skills, it was necessary to identify the where additional instruction was needed. Moreover, in order to keep track of quality improvements in reading, MEWAC required a suitable set of indicators to identify students falling behind and monitor student progression in the classroom and at the school level. In response to these issues, the Tongan government, local education stakeholders, and the World Bank joined efforts to conduct a national assessment of basic reading skills, using an adapted version of the Early Grade Reading Assessment (EGRA) tool appropriate to the Tongan context. The assessment2, the first among Pacific island countries, is part of a global initiative aimed at helping countries measure how well children are learning to read in the early grades of primary education. From October 12 and December 15, 2009, a team of Tongan reading and language specialists, consultants, and staff from the Ministry of Education worked together with the World Bank to develop, trial, and administer the Tonga Early Grade Reading Assessment (TEGRA) Survey before the end of the 2008-2009 school year. The purpose of the TEGRA survey was three-fold: 1. To develop baseline survey of basic reading skills and temporary reference standards to monitor reading performance in schools and system wide; 2. To build local capacity to replicate early grade reading assessments in the future; and 3. To work with local education stakeholders to interpret TEGRA findings and analyze their policy and sector investment implications. 1 In the Tongan education system, grade levels are called classes so that Grade 1 corresponds to Class 1. This report will follow this convention and Class should be understood as a synonym for grade level. 2 Financial support for this survey was given by the Australian Agency for International Development (AusAID), the New Zealand Agency for International Development (NZAID), and the Education for All – Fast Track Initiative (EFA-FTI). Technical assistance and management support was provided by the World Bank. 15 In particular, the Tonga assessment aimed to answer the following questions: • What are the basic reading skills acquired by Tongan students in Class 1, Class 2 and Class 3? • What are the reading fluency levels where Tongan students reach high enough levels of comprehension to understand what they read? • What are the factors that influence the acquisition of reading skills among Tongan students? To answer these questions, the EGRA tool was developed in Tongan language, the national language and the language of instruction in the first years of primary education.3 Because the EGRA tool is an orally- administered test –i.e., carried out as an interview-, it is suitable to be administered to young children whose reading and writing skills have not fully developed. The Tonga-adapted EGRA tool (i.e., TEGRA) comprised three instruments: (1) a diagnostic instrument assessing basic reading, listening and writing skills among Class 1, Class 2, and Class 3 students; (2) a student contextual interview gathering information on the student’s background, administered to all participating students; and (3) a teacher questionnaire regarding teacher characteristics, expectations, and assessment and instruction methods, answered by all Class 1, Class 2 and Class 3 teachers in sample schools. This report summarizes the survey’s main findings and provides policy recommendations to inform sector discussions and literacy improvements in Tonga. Equipped with information about the specific skills students are struggling with and the factors that appear to contribute to reading development in their country, education stakeholders in Tonga can come together to develop response strategies to improve reading instruction and monitor student progression, to ensure all children develop the skills needed to become effective readers. 3 The new curriculum for primary education –in effect since 2007- requires the use of Tongan as the language of instruction for the first three years of education (Class 1 to Class 3) and the progressive introduction of English as the language of instruction starting in grade Class 4 to full instruction in English in grade Class 6. 16 2 Chapter 2: Survey Implementation4 In order to build local capacity to replicate early grade assessments in the future, MEWAC requested technical assistance from the World Bank during survey preparation and administration. From October 12 to November 13, 2009, the World Bank provided in-country support to Ministry staff and national consultants appointed to conduct the survey, to select the sample design; develop the TEGRA instruments in Tongan language; facilitate the training of enumerators and supervisors visiting schools; coordinate survey logistics during the pilot and fieldwork; and carry out test marking and data entry. Sample Design On October 12 and 13, 2009, a series of preparatory meetings took place to discuss the scope and purpose of the survey. MEWAC was mainly interested in generating baseline data on average reading skills in Classes 1 to 3 to (a) identify areas where additional instruction is needed, to (b) inform the establishment of reference reading standards that can help educators monitor student’s reading progression throughout the school year. With advice from the World Bank, MEWAC chose a national representative sample with contrast groups according to class level (Classes 1, 2, and 3). The final sample design did not incorporate contrast groups by school type and regions, thus, survey results can only suggest estimates by class level and gender. The target population was defined as students enrolled in Classes 1 to 3 in primary schools implementing the national curriculum in Tongan language.5 Using the 2008 School Census completed in March of 2009 as a sample frame, a sample of 60 schools was selected using a stratified random design with proportional allocation based on school type –single-class (SCT), composite-class (CCT), and multi-class (MCT)6 - and region7, to ensure all school types and regions would have a probability of selection equal to their actual distribution in the country. The final sample consisted of 1,203 students –607 girls and 596 boys- which reflects the gender parity characteristics of the target population (Tables 1 and 2). Development of the TEGRA Instrument Due to differences in language, culture, and expectations about learning outcomes, the EGRA tool is adapted and piloted anew to fit the context of each country where applied. From October 14 to 26, Tongan language and reading specialist focused on learning the research foundations and structure of the EGRA tool to develop the TEGRA instrument in the Tongan language. The TEGRA team produced a series of reference 4 Throughout the report, the term “survey instrumentsâ€? is used to describe the set of TEGRA documents administered to students –i.e., diagnostic instrument (or assessment) and student background questionnaire- and teachers –i.e., teacher questionnaire- during data collection. The term “assessmentâ€? or “instrumentâ€? refers to the TEGRA diagnostic instrument consisting of 8 sub-tests or sections. The term “EGRA toolâ€? is used as reference to the latest English, French and Spanish versions of the tool which have been adapted in all EGRA-participating countries to fit the local context. 5 A small share of private primary schools (3%) has the autonomy to implement changes to the national curriculum. To avoid the risk of bias in the results, these schools were considered non-eligible. 6 School types include: (1) SCT or single-class teacher –i.e., one teacher per class/grade level-; (2) CCT or composite- class teacher –i.e., one teacher per two class/grade levels; and (3) MCT or multi-class teacher –i.e., one teacher per three or more class/grade levels. 7 Four of the five regions in the country were selected: Tongatapu (TBU), Vavau (VAV), Hapai (HAP), and Eua (EUA). The Niuas (NIU) was excluded since TEGRA took place in the aftermath of the 2009 tsunami hitting the region. 17 Table 1 - TEGRA Sample by school type, class level and gender School Class 1 Class 2 Class 3 Class Total School Type Girl Boy 1 Total Girl Boy 2 Total Class Girl Class Boy 3 Total Total Type SCT Girl 173 Boy 168 Total 341 Girl 161 Boy 163 Total 324 Girl 161 Boy 155 Total 316 981 SCT CCT 173 20 168 21 341 41 161 29 163 29 324 58 161 35 155 33 316 68 981 167 CCT MCT 20 8 21 10 41 18 29 12 29 8 58 20 35 8 33 9 68 17 167 55 MCT Total 8 201 10 199 18 400 12 202 8 200 20 402 8 204 9 197 17 401 55 1203 Total 201 199 400 202 200 402 204 197 401 1203 Table 2 - TEGRA Sample by region, class and gender Table 2 - TEGRA Sample by region, class and gender Table 2 - TEGRA Sample by region, class and gender Class 1 Class 2 Class 3 Region Class 1 Class 2 Class 3 Total Region Girl Boy Total Girl Boy Total Girl Boy Total Total TBU Girl 143 Boy 140 Total 283 Girl 140 Boy 140 Total 280 Girl 138 Boy Total 136 274 837 TBU 143 140 283 140 140 280 138 136 274 837 EUA 14 12 26 13 11 24 12 11 23 73 EUA HAP 14 14 12 13 26 27 13 16 11 13 24 29 12 18 11 16 23 34 73 90 VAV HAP 30 14 34 13 64 27 33 16 36 13 69 29 36 18 34 16 70 34 203 90 VAV 30 34 64 33 36 69 36 34 70 203 Total 201 199 400 202 200 402 204 197 401 1203 Total 201 199 400 202 200 402 204 197 401 1203 Development of the TEGRA Instrument Development documents–e.g., Due a list of to differences of the TEGRA familiar wordsInstrument in language, in Tongan culture, andlanguage taught in expectations Classlearning 1 to Class about 3- that were outcomes, the not readilyDue available EGRA totool differences for is the adapted language, in and Tongan language piloted culture, (see anewAnnex toandfit expectations 4 for the a context ofabout full description each onlearning the process). country outcomes, where draft the Aapplied.version EGRA of theFrom TEGRA tool October is adapted assessment14 towith 26,and piloted Tongan several anew language narrative to fit the and passages context reading was of each specialist discussed country focused in two focusonwhere groups applied. learning with the MEWAC From research officials October and non-government14 to foundations and 26, Tongan structure literacy language experts ofinthe theEGRA and country. reading tool The to specialist develop TEGRA team focused the on TEGRA instrument incorporated learning the in the participants’ research the Tongan foundations language. and The structure TEGRA of the team EGRA tool produced a to develop series of the TEGRA reference instrument documents–e.g., a in list feedback and the TEGRA instrument was piloted on October 26, 2009. the Tongan language. The TEGRA team produced a series of reference of familiar words in Tongan language taught in Class 1 to Class 3- that were not readily documents–e.g., a list of familiar Fromavailable October for 28 words the to in Tongan Tongan November language language 06, 2009, (see taught Annex 28 Ministry 4 in forClass staff a full from 1 the to Class 3- description 4 provinces on that inthe thewere process). countrynotattended readily A draft the available version training for of the workshop the Tongan to TEGRA learn about language assessment the scope (see withAnnex andseveral 4 purpose for a full narrative description passages of the survey, on the process). was discussed the research A in draft foundations twobehind version focus of the groups TEGRA with MEWAC assessment officialswith andseveral narrative passages non-government was discussed literacy experts in two in the country. the EGRA tool and its administration guidelines. On November 05, a pilot of the TEGRA assessment took focus The groupsteam TEGRA MEWAC officials withincorporated and non-government the participants’ feedback and literacy experts instrument in the country. the TEGRA was place in 3 schools in Nuku’alofa, collecting data for 108 students in Classes 1, 2 and 3. A team of 4 national The TEGRA piloted team incorporated on October 26, 2009. the participants’ feedback and the TEGRA instrument was consultants piloted on a and Tongan26, October 2009. specialist was trained on how to test mark survey instruments and were language responsible for marking From October 28 to pilot and fieldwork allNovember 06, 2009, scripts for analysis. 28 Ministry staff from the 4 provinces in the country From October 28 to November attended the training workshop to learn about thestaff 06, 2009, 28 Ministry scopefrom andthe 4 provinces purpose the country in survey, of the the TEGRA pilot data attended the prompted a few training workshop changes to learnto specific about items that appear to be the survey,for very difficult most research foundations behind the EGRA tool the and scope and purpose of guidelines. its administration the On students. 05, reliability Inter-rater research November foundations a pilot of –i.e., behind thethe ability the TEGRA of EGRA enumerators tool and assessment toits took in 3 the administer administration place assessment schools correctly guidelines. in Nuku’alofa,On and November consistently- was collecting 05, calculated data a pilot for of the 108 with results students TEGRA in of 85% assessment Classes and 1, 2 above and 3. took A all for place sub-tests team of in schools 3except 4 national in–Sub-test one Nuku’alofa, consultants and 2:ainitial soundcollecting Tongan data language identification- for 108 students specialist where most was in Classes trained on enumerators 1, had 2 how and 3. A team of 4 to test understanding difficulty national consultants mark survey instruments and and were student responses. 8 a Using Tongan language responsible for specialist marking all was pilot trained and on fieldwork how to test scripts for mark survey analysis. instruments enumerator and supervisor feedback, the instrument’s layout was adjusted to improve readability and ease and were responsible for marking all pilot and fieldwork scripts for analysis. administration. Additionally, pilot versions of the student and teacher questionnaires were improved to TEGRA pilot data prompted a few changes to specific items that appear to be very difficult ensure questions were properly framed. TEGRA for mostpilot Inter-raterareliability data prompted students. few changes –i.e., specific to the items ability that appear to of enumerators tobe very difficult administer the for most students. assessment correctly Inter-rater reliability and consistently- was–i.e., the ability calculated with enumerators ofresults of 85% to administer above for the and all assessment correctly and consistently- was calculated with results of sub-tests except one –Sub-test 2: initial sound identification- where most enumerators had 85% and above for all sub-tests except one –Sub-test 2: initial sound identification- where most enumerators had 18 Fieldwork and Data Entry Data collection took place between November 09 and December 12, 2009. Over a period of 5 weeks, teams of enumerators and supervisors visited sample schools in the country. Complete survey documents were brought back to Nuku’alofa by the TEGRA team members for marking. Data entry took place between January 25 and February 05, 2010. For the 171 teachers surveyed, the overall response rates to the survey items are above 86%. For the 1,203 students in the sample, the overall survey response rates are above 93%. Reliability of the Instrument Cronbach’s alpha was calculated for the TEGRA instrument to measure the reliability of the test to capture reading abilities. Results showed a strong internal consistency with a coefficient of 0.91. As a rule of thumb, an alpha coefficient of 0.80 considered good and 0.7 an absolute minimum. The alphas for each sub-test were close to 1, suggesting high reliability across TEGRA sub-tests (see Table 25 in Annex 1 for the TEGRA reliability matrix). Table 3 - Reliability of TEGRA assessment Table 3 - Reliability of TEGRA assessment Average Item-test Item-rest inter-item Item Sign correlation correlation correlation Alpha Letter Name Knowledge (correct letters per minute) + 0.8137 0.7556 0.5439 0.9051 Initial Sound Identification + 0.5934 0.4883 0.5989 0.9227 Letter Sound Knowledge (correct sounds per minute) + 0.6462 0.5503 0.5857 0.9188 Familiar Word Reading (correct words per minute) + 0.8993 0.8656 0.5225 0.8975 Invented Word Reading (correct invented words per + 0.9088 0.8781 0.5202 0.8966 minute) Oral Reading (correct words per minute) + 0.8892 0.8525 0.525 0.8984 Reading Comprehension (percentage correct) + 0.8328 0.7798 0.5391 0.9035 Listening Comprehension (percentage correct) + 0.5733 0.4648 0.6039 0.9242 Writing (percentage correct; weighted score) + 0.8312 0.7779 0.5395 0.9036 Test scale 0.5532 0.9176 8 This section showed an average inter-rater reliability (IRR) of 73%. To correct for this, changes were made to the instrument to avoid two similar sounds (e.g. “sâ€? and “tâ€?) to be placed in sequence. After additional practice was given to all enumerators, the IRR for this section increased to 79%. 19 3 Chapter 3: TEGRA Results Structure of the Assessment As has been confirmed by scholarsChapter 3: TEGRA working to understand Results reading acquisition in multiple languages, (Jimenez and O’Shanahan Juan, 2008; Linan-Thompson and Vaughn, 2007; Abadzi, 2006; Sprenger-Charolles, 2004; Structure Chiappe of the et al., 2002), Assessment in almost any alphabetic language in which print can be decoded into sounds, being As has been able to read well requires a graspscholars confirmed by working of five basic to understand skills (National Reading reading Panel, acquisition 2000): in multiple languages, (Jimenez and O’Shanahan Juan, 2008; Linan-Thompson and Vaughn, 2007; Abadzi, 2006; Sprenger-Charolles, • phonemic awareness–focusing on, Chiappe 2004; 2002), in apart, et al., breaking manipulating, anyputting almostand alphabetic together language soundsin which print can be decoded into sounds, being able to read well requires a grasp orally; five of • basic skills (National phonics–linking written Reading Panel, letters to their 2000): sounds and forming spelling patterns; • phonemic awareness–focusing on, manipulating, breaking apart, and putting • fluency–achieving speed, accuracy, and expression in reading; together sounds orally; • vocabulary–knowing words (both oral and written) and their meaning; and • phonics–linking written letters to their sounds and forming spelling patterns; • • comprehension–understanding fluency–achieving speed, accuracy, the concepts read orin and expression heard. reading; • vocabulary–knowing words (both oral and written) and their meaning; and • comprehension–understanding the concepts read or heard. Though not all children develop their reading abilities in the same way or pace, the literature shows that Though not all children develop their reading abilities in the same way or pace, the all readers progress through a series of phases or stages –sometimes simultaneously- in their reading literature shows that all readers progress through a series of phases or stages –sometimes development process (RTI, 2010). simultaneously- in their reading development process (RTI, 2010). 1 -1Stages Figure Figure ofReading - Stages of ReadingDevelopment Development Source: RTI, 2010. Source: RTI, 2010. Once children Once to apply learn learn children the foundational to apply reading reading the foundational skills with a certain skills withlevel of reflex a certain or automaticity, level of reflex or they can automaticity, move beyond thethey canof task move beyond decoding the a text task of (Stage 1) decoding a text (Stage to begin deriving 1) to (Stage begin deriving its meaning 2). As children its meaning (Stage 2). As children learn sounds that link to form words, they can begin 20 connecting those sounds to printed words and the idea behind those words. Then they can link words to form sentences, paragraphs, and stories. In other words, children transition from learning to read to reading to learn (Stage 3 and beyond). Comprehension is the ultimate learn sounds that link to form words, they can begin connecting those sounds to printed words and the idea behind those words. Then they can link words to form sentences, paragraphs, and stories. In other words, children transition from learning to read to reading to learn (Stage 3 and beyond). Comprehension is the ultimate prize—the difference between “reading itâ€? and “getting it.â€? (RTI), 2010). The structure of the EGRA tool in different international applications used this knowledge as a reference point during the adaptation of the tool to local contexts. EGRA tool in different international applications used this knowledge as a reference point during the adaptation of the tool to local contexts. Figure 2 - Early Figure GraderReading Grader 2 - Early Assessment Reading Assessment Components Components Source: RTI, Source: RTI,2010. 2010. In particular, the TEGRA assessment focused on measuring these skills plus basic listening and writing skills In particular, the TEGRA assessment focused on measuring these skills plus basic listening through eight modules or sub-tests: (1) letter name knowledge; (2) identification of initial sounds in words; and writing skills through eight modules or sub-tests: (1) letter name knowledge; (2) (3) letter sound knowledge; (4) familiar word reading; (5) invented word reading; (6) oral reading fluency identification of initial sounds in words; (3) letter sound knowledge; (4) familiar word comprehension; withreading; (7) listening (5) invented wordcomprehension; and reading reading; (6) oral (8) dictation with 5 (see Annex fluency for copies of the comprehension; TEGRA (7) listening Table instruments). 4 below shows comprehension; how and (8)these skills relate dictation (see to each of Annex for TEGRA 5 the copies components, of the TEGRA measures and instruments). indicators. Table 4 below shows how these skills relate to each of the TEGRA components, measures and indicators. Administration of the TEGRA Instrument Table 4 - TEGRA Instrument Structure and Early Skills Tested The TEGRA assessment was administered via face-to-face interviews of about between an enumerator9 and Skill demonstrated by students’ Measure and Indicator Sub-test Early reading skill the student. Each interview lasted 15 to 25 minute ability to: from the onset of the test to completion of the student Provides the name of upper- and Letter name fluency in terms of background questionnaire. 1. Letter name In six of the eight sub-tests in TEGRA instrument, students had 60 seconds to Letter recognition lower-case letters distributed in correct letter names identified per knowledge random order in a given skill. To be complete the sub-test in order to assess automaticity (CLPM) readers, basic reading successful minute Segment words into 2 to 5 Phoneme segmentation as the competencies have to be 2. Identification of automatic. Fluency Phonemic phonemes measures assess not only whether or not a child knows number of sounds correctly something, but whether they initial sounds have integrated awareness the knowledge Identify words with and can process different the information automatically identified beginning or ending phoneme Provides the sound of upper- and Letter name fluency in terms of 3. Letter sound Phonics lowercase letters distributed in correct letter sounds identified per 9 Theknowledge term enumerator is used in reference to purposely trained interviewers administer random order minuteearly grade reading assessments. (CLPM) 4. Familiar word Read simple and common one- and Familiar word fluency in terms of 21 Word reading correct familiar words read per reading two-syllable words minute (CFWPM) Make grapheme-phoneme Invented word fluency in terms of Table - TEGRAInstrument 4 -4 TEGRA Table Structure Instrument Structure and and Early Early Skills Skills Tested Tested Skill demonstrated by students’ Measure and Indicator Sub-test Early reading skill ability to: Provides the name of upper- and Letter name fluency in terms of 1. Letter name Letter recognition lower-case letters distributed in correct letter names identified per knowledge random order minute (CLPM) Segment words into 2 to 5 phonemes Phoneme segmentation as the 2. Identification of initial Phonemic number of sounds correctly identified Identify words with different sounds awareness beginning or ending phoneme Provides the sound of upper- and Letter name fluency in terms of 3. Letter sound Phonics lowercase letters distributed in correct letter sounds identified per knowledge random order minute (CLPM) Familiar word fluency in terms of Read simple and common one- and 4. Familiar word reading Word reading correct familiar words read per two-syllable words minute (CFWPM) Make grapheme-phoneme Invented word fluency in terms of 5. Invented word Alphabetic correspondences (GPCs) through the correct invented words read per reading principle reading of simple invented –i.e., minute (CUWPM) invented- words to test decoding skills Oral reading fluency in terms of Oral reading Read a text with little effort and at a correct words read per minute in a fluency sufficient rate narrative passage (CWCPM) 6. Oral reading fluency Respond correctly to different types Response to questions after reading with comprehension Reading of questions, including literal and a story as a percentage of correct comprehension inferential questions about the text answers they have read Respond correctly to different types Response to questions after hearing 7. Listening Listening of questions including literal and a story as a percentage of correct comprehension comprehension inferential questions about the text answers the enumerator reads to them Write, spell, and use grammar properly through a dictation exercise, Alphabetic Write, spell, and use grammar determined by the percentage of 8. Dictation principle properly through a dictation exercise overall early writing skills (spelling and basic conventions), weighted score. Note: Adapted by the authors, based on RTI, 2009 and Linan-Thompson, 2010. (Linan-Thompson, 2007). Time-limitation allows proper comparison of fluency across slow readers and fast Administration of the readers who may register TEGRA the same Instrument scores at different periods of time (RTI, 2009). The TEGRA In TEGRA, assessment student scores in was sub-testsvia administered time-limited face-to-face are calculated interviews as the number ofof about correct between items an –i.e., letter enumerator 9 and the student. Each interview lasted 15 to 25 minute from the onset of the test to names, letter sounds, or words- read per minute. If a student completes all of the words before the time completion of the student background questionnaire. In six of the eight sub-tests in TEGRA expires, the time of completion is recorded and the number of items correctly read per minute is estimated instrument, students had 60 seconds to complete the sub-test in order to assess automaticity in a on that time period.10 Selected sub-tests applied an “early-stop ruleâ€? to discontinue the administration of a given skill. To be successful readers, basic reading competencies have to be automatic. Fluency sub-test if students were unable to correctly respond to any of the items in the first of ten lines (Sub-tests, 1,3,4,5, and 6), or if their responses for the first five items were incorrect (Sub-test 2).11 In this situation, the 9 The term enumerator is used in reference to purposely trained interviewers administer early grade reading assessments. Correct Items Per Minute = (Total items read – Total items incorrect) / [(60 – Time remaining on stopwatch) / 60] (RTI, 2009) 10 The rule was established to avoid frustrating students who did not have the skill or did not understand the task of the 11 sub-test (RTI, 2010). 23 22 enumerator was asked to mark the box that read “Check this box if the exercise was discontinued because the child has no correct answers in the first lineâ€? and to proceed to the next sub-test in the test (RTI, 2010).12 reasons. First, the early stop rule helps avoid frustration among students whose dispirit may affect theirtoperformance The justification discontinue ain sub-test subsequenthas two reasons.Second, sub-test. First, the early the stopstop early rule helps rule avoid is alsofrustration an among students whose dispirit may affect approximation of zero-scores throughout the test: as in most psychometric tests, it is early their performance in subsequent sub-test. Second, the stop assumed that rule is also students who of an approximation fail the first initial zero-scores items throughout thewill fail test: asthe remainder in most of the tests, test, it is psychometric especially assumed if the first that students who items also arefirst fail the the items initial easiest, as will it the fail in the case wasremainder of of theTEGRA. Sub-tests test, especially 7 first if the and 8 did not apply the “early stop ruleâ€? so results in these sub-tests relate items are also the easiest, as it was in the case of TEGRA. Sub-tests 7 and 8 did not apply the “early stop to the total sample of students and report percentage of correct answers over the total number of items ruleâ€? so results in these sub-tests relate to the total sample of students and report percentage of correct in the sub-test. answers over the total number of items in the sub-test. Across Across sub-tests sub-tests and class and class levels, levels, the largest the largest proportions proportions of early-stop of early-stop cases appeared cases appeared in sub- in sub-test 2 (initial test 2 (initial sound identification), sub-test 3 (letter sound knowledge), and sub-test 6 (oral sound identification), sub-test 3 (letter sound knowledge), and sub-test 6 (oral passage reading). Sub-test passage reading). Sub-test 2 and 3 entail skills related to letter-sound correspondences so 2 and 3 entail skills related to letter-sound correspondences so students struggling to match letters and students struggling to match letters and sounds in isolation are likely have difficulties when sounds in isolation reading isolated arewords likely have difficulties (sub-tests 4 andwhen reading 5) and words in the words isolated (sub-tests oral reading 4 and passage 5) and words 6). in (sub-test Thereading the oral sub-test passage (sub-test that showed 6).lowest the The sub-test that showed percentage the lowest of early-stop percentage cases of early-stop was sub-test 1 (lettercases was name sub-testknowledge) 1 (letter namewhere the test had knowledge) wheretothe be test discontinued had to be only for 3% of discontinued the only students for 3% of thein students the sample in the sample -8% -8% ininClass Class 1% 1,1, 1% ininClass Class 22and and0%0% Class inin Class3-. 3-. As 13 13 expected, As expected, the assessment the assessment had toto be had be discontinued for a larger percentage of Class 1 students (see Figure 1 below) discontinued for a larger percentage of Class 1 students (see Figure 1 below) though significant reductions though significant reductions in the number of discontinued cases are observed in Class 2 and Class in the number of discontinued cases are observed in Class 2 and Class 3. 3. 5 – Zero-score Table Table cases 5 – Zero-score per cases per class class and and sub-test sub-test Oral Letter Familiar Unfamiliar Class Letters Initial Sound Passage Sounds Word Word Reading Class 1 32 206 140 120 141 148 Percentage/Total 8% 52% 35% 30% 35% 37% Sample in Class 1 Class 2 5 133 69 26 30 35 Percentage/Total 1% 33% 17% 6% 7% 9% Sample in Class 2 Class 3 1 79 40 6 9 12 Percentage/Total 0% 20% 10% 1% 2% 3% Sample in Class 3 Total (per sub-test) 38 418 249 151 180 276 Percentage over 3% 35% 21% 13% 15% 23% total sample TEGRA Results per Sub-test TEGRA results show reading gains across the three class levels tested. Overall, students struggled with sub-tests requiring a solid knowledge of letter-sound correspondences -even as late as in Class 3 students. Across grades, Tongan students seem to have mastered letter 12 Annex recognition Seename ascopy 5 for a bilingual ofas early thein Tongan 1, with a minimal number of students classified as an Classinstrument. 13 Total percentages per class per skill –i.e., number of cases / total sample size per class- do not add up to the total percentage of early-stop cases per skill –i.e., number of cases per skill / total sample size. 13 23 Total percentages per class per skill –i.e., number of cases / total sample size per class- do not add up to the total percentage of early-stop cases per skill –i.e., number of cases per skill / total sample size. TEGRA Results per Sub-test TEGRA results show reading gains across the three class levels tested. Overall, students struggled with sub-tests requiring a solid knowledge of letter-sound correspondences -even as late as in Class 3 students. Across grades, Tongan students seem to have mastered letter name recognition as early as in Class 1, with a minimal number of students classified as an early-stop cases. TEGRA results showed evidence of gender differences, with girls outperforming boys in all sub-tests. For each of the sub-tests below, average results are presented for the sample as a whole –i.e., total average- as well as without the proportion of zero-score students –i.e., early stop cases. There are two reasons behind the decision to present these two results. One the one hand, some researchers argue that in cases where there is a large presence of zero-score students, overall means tend to underestimate the true average score of the population. On the other, the use of average means without zero-score students tends to overestimate the true average score. Since the purpose of this survey is to inform policy decisions over the establishment of temporary reference standards of oral reading fluency in the country, we believe it was important to present both results in order to inform policy discussions and future decisions over where and how to establish adequate reference reading standards for Tonga. Sub-test 1 – Letter Name Knowledge14 The test of letter name knowledge is the most basic of assessments of student reading preparedness (and risk). Letter name knowledge is a consistent predictor of reading development for native speakers of English, French, and other alphabetic languages (Chiappe, Siegel, & Wade-Woolley, 2002). It has also proved to be a useful indicator for nonnative speakers (Chiappe, 2006). In the sub-test of letter name knowledge, students are asked to provide the names (not the sounds) of all of the letters they could read, within a one-minute period. The full set of letters of the Tongan alphabet was listed in random order, 10 letters to a row, for a total of 100 letters. Letters were selected based on the frequency with which they occur in Tongan. Randomization was used to prevent students from reciting a memorized alphabet—that is, to test for actual automaticity of letter recognition and translation of print to sound. TEGRA students showed an overall positive performance on recognizing letters, defined by the indicator “number of correct letter names identified per minuteâ€? (CLPM). Table 6 shows average scores in sub-test 1 by class and gender, and overall scores excluding zero-scoring students for those unable to recognize any letters. TEGRA scores show students in Tonga have a good understanding of names of letters. On average, students correctly identified 55 letter names per minute. Zero-score students appear to reduce the average mean by two letter names correctly identified (from 55 to 57). Class and gender bring about important differences to the knowledge of letter names. The largest increase in letter name knowledge takes place between Class 1 and Class 2 with an average difference of 22 letter names -twice the gain observed between Class 2 and Class 3. Overall, girls are more accurate readers of letter names (about 59 letters correctly identified) than boys (51 letter names). This difference is even greater (7 letter names) if zero-score students are excluded. 14 Throughout this section, sub-test description is based on the Early Grade Reading Assessment Toolkit (RTI, 2009). 24 names per minute. Zero-score students appear to reduce the average mean by two letter names correctly identified (from 55 to 57). Class and gender bring about important differences to the knowledge of letter names. The largest increase in letter name knowledge takes place between Class 1 and Class 2 with an average difference of 22 letter names -twice the gain observed between Class 2 and Class 3. Overall, girls are more accurate readers of letter names (about 59 letters correctly identified) than boys (51 letter names). This difference is even greater (7 letter names) if zero-score students are excluded. Table 6 - Sub-test Table 1 Letter 6- Sub-test Name 1 Letter Knowledge: Name Knowledge: Results Results by Class by Class and Gender and Gender Sub-test 1 - Number of correct letter names N Mean SD Min Max identified per minute (CLPM) CLPM - overall 1,203 55.1 25.6 0 130 CLPM - minus zero score students 1,165 56.7 24.1 1 130 Class CLPM - overall 400 37.0 22.7 0 100 Class 1 CLPM - minus zero score students 368 40.1 20.8 1 100 CLPM - overall 402 59.0 21.8 0 130 Class 2 CLPM - minus zero score students 397 59.6 21.1 4 130 CLPM - overall 401 69.3 21.0 0 124 Class 3 CLPM - minus zero score students 400 69.3 20.9 4 124 Gender CLPM - overall 607 58.7 26.3 0 130 Girls CLPM - minus zero score students 590 60.3 24.7 3 130 CLPM - overall 596 51.3 24.4 0 118 Boys CLPM - minus zero score students 575 53.0 22.9 1 118 Sub-test 2 – Initial Sound Recognition In order to2read, Sub-test – Initial each Sound Recognition of us must turn the letters we see into sounds, sounds into words, and words into In order Successfully meaning. to read, each of us must managing turn the this process letters requires we the see to ability into sounds, work sounds in reverse; into that is, words, students and should words into also grasp meaning. that Successfully words are composed managing of individualthis process sounds requires the and understand ability the to of process work in reverse; separating (and manipulating) words into sounds (Snow et al., 1998). The ability to identify sounds in words, to separate words into sounds, and to manipulate those sounds is termed phonemic awareness, found to play an important 26 role in reading acquisition and the number one predictor of success in reading, better than socioeconomic status, preschool attendance, or reading time in the home (Share, Jorm, Maclearn, & Matthews, 1984). Testing for and remediating this skill is thus important for later reading development. Thus far, the EGRA tool has piloted an assessment of phonemic awareness in two different ways: using phoneme segmentation and identification of onset and rime sounds (first and last sounds). Phoneme segmentation –i.e., the division of words into phonemes- is one of the most complex skills of phonological awareness and should be emphasized in the early grades (Linan-Thompson & Vaughn, 2007). It is also one of the most predictive of later learning skills. Thus far, phoneme segmentation has proved difficult to administer. The TEGRA instrument selected a simpler task –i.e., initial sound identification- to assess student’s ability in phoneme segmentation. A set of 10 familiar words in the Tongan language was selected. Students were asked to identify the initial sound in each of the words. The enumerator read each word aloud twice before asking the student to identify the sound before recording the student’s answer –i.e., correct, incorrect, no answer- in each item in the sub-test. Student performance in this sub-test showed one of the lowest average scores in the TEGRA test, suggesting most students in Tonga struggle to understand and separate the sounds in the words they hear, an essential skill in reading development. Scores are defined as the average number of initial sounds identified from a list of 10 one and two-syllable words included in the exercise. On average, students are able to correctly 25 identify 4.2 initial sounds of words; excluding zero-score students (418 in the sample) the average increases to 6.2 initial sounds. Moreover, although average scores improve as a result of student progression across classes, these are Moreover, although average scores improve as a result of student progression across classes, these -only very small 1 more are very initial small -onlysound correctly 1 more initialidentified, on average- sound correctly to the on identified, extent that after average- three to the years of extent primary education, that Class after three 3 students years able to identify are education, of primary Classonly 3 half of theare students words they able hear (average only score to identify of 5.2 half of the correct words initial they hear sounds). (average Although girlsscore of 5.2 appeared tocorrect be onlyinitial sounds). slightly Although better than girls appeared boys in identifying to initial sounds be only slightly better than boys in identifying initial sounds in words –about half a sound- in words –about half a sound- fewer girls than boys scored zero in the sub-test (197 and 221, respectively). fewer girls than boys scored zero in the sub-test (197 and 221, respectively). Table 7 – Sub-test 2 Initial Sound Recognition: Results by class and gender. Table 7 – Sub-test 2 Initial Sound Recognition: Results by class and gender. Sub-test 2 -Number of correct initial sounds Lower Upper N Mean SD identified bound Bound Overall 1,203 4.2 3.3 4.0 4.4 Minus zero score students 785 6.2 2.1 6.0 6.3 Class Overall 400 3.1 3.4 2.8 3.4 Class 1 Minus zero score students 194 6.0 2.1 5.7 6.3 Overall 402 4.3 3.3 4.0 4.6 Class 2 Minus zero score students 269 6.2 2.0 5.9 6.4 Overall 401 5.2 3.0 4.9 5.4 Class 3 Minus zero score students 322 6.2 2.1 6.0 6.5 Gender Overall 607 4.4 3.4 4.1 4.6 Girls Minus zero score students 410 6.2 2.1 6.0 6.4 Overall 596 4.0 3.3 3.7 4.2 Boys Minus zero score students 375 6.1 2.0 5.9 6.30 Average performance in sub-test 2 related to the complexity of the word presented. A review of percentage Average performance in sub-test 2 related to the complexity of the word presented. A correct answers review to each of correct of percentage words presented the 10 answers to each ofshows the 10 that students words have shows presented a goodthatknowledge studentsof the sounds have a goodbut of vowels are unable knowledge ofto correctly the sounds of vowels and understand but separate are unable sounds theto of consonants correctly understand –especially and separate of “ngâ€?- the sounds in the context of consonants of a word. were of –especially Sample students able“ngâ€?- in the identify to correctly context the of initial a word. Sample sounds of words students that begin withwere to correctly able –ifo, a vowel ako- in the uta, ou,identify initial about 54% sounds to 59% words that of percent begin of the with cases. a vowel – for Performance ifo, uta, ou, ako - in about 54% to 59% percent of the cases. Performance consonants decreases to 24% - 28%, and is particularly low for the nasal consonant “ngâ€? (13%). for consonants decreases to 24% - 28%, and is particularly low for the nasal consonant “ngâ€? (13%). Sub-test 3 – Letter Sound Identification how letters Knowledge of Table correspond 8 - Percentage to initial of correct sounds is another sounds critical identified skill per item in children must the sub-test, master to become by class successful readers. Letter-sound correspondences are typically taught through phonics-based approaches. In this sub-test, students were asked to provide the sounds (not the names) of as many letters they could Class 1 Class 2 Class 3 Total identify within a one-minute period. The full set of letters and graphemes in the Tongan alphabet –e.g., Items“aâ€? and “Ä?â€?- graphemes Girl wasBoy Total listed in Girl random Boy order, Total 10 letter Girl to a Boy sounds Total row, for a total Girl of 100 Boy Total letter sounds. Initial sound: niu 0.27 0.23 0.25 0.35 0.28 0.32 0.37 0.32 0.35 0.33 0.28 0.30 15 Initial sound: A grapheme istali 0.17basic the most unit in 0.20 0.23 0.29 the alphabetic 0.22 system 0.26can be and 0.35 0.33 made 0.34or more up of one 0.27 letters. 0.27 0.26Graphemes combine to create phonemes –i.e., smallest distinctive unit of sound which allows differentiation between words such as the phoneme Initial sound: kovi “mopâ€?. 0.19 0.21 vs 0.18 “tâ€? in “topâ€? 0.35 0.26 0.31 0.42 0.43 0.42 0.33 0.29 0.31 26 Initial sound: ifo 0.45 0.42 0.43 0.65 0.56 0.61 0.77 0.70 0.73 0.62 0.56 0.59 Initial sound: uta 0.46 0.42 0.44 0.66 0.58 0.62 0.74 0.69 0.72 0.62 0.56 0.59 - Percentage Table 8Table of correct 8 - Percentage initial of correct sounds initial per identifiedper sounds identified item item in in thethe sub-test, sub-test, by class by class Class 1 Class 2 Class 3 Total Items Girl Boy Total Girl Boy Total Girl Boy Total Girl Boy Total Initial sound: niu 0.27 0.23 0.25 0.35 0.28 0.32 0.37 0.32 0.35 0.33 0.28 0.30 Initial sound: tali 0.17 0.23 0.20 0.29 0.22 0.26 0.35 0.33 0.34 0.27 0.26 0.27 Initial sound: kovi 0.21 0.18 0.19 0.35 0.26 0.31 0.42 0.43 0.42 0.33 0.29 0.31 Initial sound: ifo 0.45 0.42 0.43 0.65 0.56 0.61 0.77 0.70 0.73 0.62 0.56 0.59 Initial sound: uta 0.46 0.42 0.44 0.66 0.58 0.62 0.74 0.69 0.72 0.62 0.56 0.59 Initial sound: ou 0.40 0.41 0.41 0.63 0.53 0.58 0.73 0.68 0.70 0.59 0.54 0.56 Initial sound: hao 0.24 0.23 0.24 0.37 0.28 0.32 0.40 0.41 0.40 0.33 0.30 0.32 Initial sound: ako 0.46 0.42 0.44 0.63 0.58 0.61 0.66 0.66 0.66 0.59 0.56 0.57 Initial sound: ngu 0.10 0.10 0.10 0.14 0.11 0.13 0.15 0.15 0.15 0.13 0.12 0.13 Initial sound: 'eni 0.44 0.35 0.40 0.57 0.51 0.54 0.65 0.68 0.67 0.56 0.51 0.54 Note: Weighted averages. TEGRA average scores in sub-test 3 indicate students have not mastered knowledge of the relationship Sub-testsounds between identified 3 16 and Sound – Letter sounds letters perTongan in the Identification of letter minute. alphabet. Scoresincreases This average in sub-testto323 at defined are the end as ofthe number Class 2, andof Knowledge correct 25 letter at the end of ofhow sounds identified Class 3. Oncorrespond letters per minute average, to sounds (CSPM) girls and is correctly another use critical the 60-second identify 3 sound skill time children base letters of 4, (or one master must timed-sub-tests allif excludes to become zero-score to compare successful students) fluency andreaders. more Letter-sound than accuracy boys in even letter correspondences soundthough are the proportion identification. On typically of zero-score average, taught students through students correctly phonics- between identified 21 based the two letter approaches. per similar. is fairly sounds In this sub-test, students were asked to provide the sounds minute. Excluding the 249 students with zero-scores (21% of the sample), the average (not the names) of as many letters they could identify within a one-minute period. The full set of letters and graphemes 9the 15 inTable Table Tongan – Sub-test alphabet 9 – Sub-test 3 Letter 3 Letter Soundgraphemes –e.g., Sound “aâ€? Identification: Identification: and by Results Results â€?- was “Ä?by class and class listed gender.in random order, and gender. 10 letter sounds to a row, for a total of 100 letter sounds. Sub-test 3 - Number of correct letter sounds identified per minute (CSPM) N Mean Min Max TEGRA averageCSPM scores in sub-test 3 indicate students - Overall 1,203 have21.5not mastered 0 knowledge 89 of the relationship between CSPM - sounds Minus zeroand letters score in the Tongan students 954 alphabet. Scores in 1 26.3 sub-test 3 are 89 defined as the number of correct letter sounds identified per minute (CSPM) and use the 60-second time Class base ofClass CSPM - Overall all timed-sub-tests to compare fluency and 400 accuracy 16.3 in letter sound 86 0 identification. On 1 CSPM - Minus zero score students 260 23.8 1 86 average, students correctly identified 21 letter sounds per minute. Excluding the 249 students CSPM - Overall 402 23.2 0 83 with zero-scores Class 2 (21% of the sample), the average score CSPM - Minus zero score students 333 increases 27.2 by about 1 5 words83 from 21 to 26 correct letterCSPM sounds per minute. At the end 401 - Overall of Class 1, students 0 25.1 are able to 89 correctly Class 3 CSPM - Minus zero score students 361 27.3 1 89 Gender 15 A grapheme is the most -basic CSPM 607can be made unit in the alphabetic system and Overall 22.8up of one or0 more letters. 88Graphemes combine Girls to create phonemes –i.e., smallest distinctive CSPM - Minus zero score students unit of 479sound which allows 28.1 differentiation 1 between 88 words such as the phoneme “tâ€? in “topâ€? CSPM - Overall vs “mopâ€?. 596 20.2 0 89 Boys CSPM - Minus zero score students 475 24.4 1 89 29 27 CSPM - Minus zero score students 479 28.1 1 88 score increases by about 5 words from 21 to 26 correct letter sounds per minute. At the end of Class 1, CSPM - Overall 596 20.2 0 89 Boys students are correctly able toCSPM identified - Minus 16students zero score sounds of letter 475 24.4This average per minute. 1 increases 89 to 23 at the end of Class 2, and 25 at the end of Class 3. On average, girls correctly identify 3 sound letters (or 4, if one excludes zero-score students) more than boys even though the proportion of zero-score students between the two is fairly similar. Sub-test 4 – Familiar Word Reading Sub-test – Familiar 4 who Children read Reading Word are able to words that are familiar to them often do that by automatic recognition. Automated word recognition in reading allows a beginning reader to ‘read’ a Children are able whoword familiar to read not by words its letters, that but asare familiar a whole. to this For them often do that assessment, by automatic high-frequency recognition. familiar words word Automated selected from were recognition early allows in reading grade areading materials, beginning reader tostory books, ‘read’ andword a familiar school notvisits by its to letters, primary but as schools a whole. to For this ask teachers assessment, in Classes 1 familiar high-frequency to 3 about wordsthe vocabulary were usedearly selected from during their grade reading lessons. Words were arranged horizontally, in good separation from each other, written materials, story books, and school visits to primary schools to ask teachers in Classes 1 to 3 about in a the familiar (lower case) font, comprising 10 rows, five familiar words per line. vocabulary used during their lessons. Words were arranged horizontally, in good separation from each other, written in a familiar (lower case) font, comprising 10 rows, five familiar words per line. TEGRA average scores in sub-test 4 showed a low automaticity in word reading, ability TEGRA average scores in sub-test 4 showed a low automaticity in word reading, ability closely associated closely associated to word reading in the oral reading passage and ultimately, to word oral reading reading in the Scores comprehension. passage in sub-test and 4 are ultimately, defined comprehension. as the Scores number of correct in sub-test familiar words4 are defined read per as the minuteof number (CFWPM). On average, correct familiar words readstudents were able per minute to correctly (CFWPM). read about On average, 19 familiar students were able to word sounds in a minute. Excluding the 151 students with zero-scores (13% of the correctly read about 19 familiar word sounds in a minute. Excluding the 151 students with zero-scores (13%sample) the of the average sample) score the increases average scoreby 2 words. increases At by 2 the end At words. of Class the end1, students are of Class 1, able to read students about are able to read 9 familiar words per minute. This average increases to 19 and 28 at the end of Class about 9 familiar words per minute. This average increases to 19 and 28 at the end of Class 2 and Class 2 and Class 3, respectively. On average, girls correctly read 6 familiar words more than boys (22 3, respectively. On average, girls correctly read 6 familiar words more than boys (22 and 16, respectively) and 16, respectively) but showed more variability in their results (with a standard deviation but showed and a in more variability of 16 CFWPM) their range wider (with resultsof a standard scores (0 – 130 deviation CFWPM) of 16 CFWPM) compared toand a wider those range of of boys scores CFWPM) compared (0 – 130deviation (standard of 12 CFWPM to those andofa boys range(standard deviation of of 0-72 CFWPM). 12 difference The CFWPM and is a range (6 similar of 0-72 familiar CFWPM). The words) if zero-score difference students is similar (6 familiar are excluded. words) if zero-score students are excluded. Table 10 – Sub-test 4 Familiar Word Reading: Results by Class and Gender Table 10 – Sub-test 4 Familiar Word Reading: Results by Class and Gender Sub-test 4 - Number of correct familiar words read N Mean Min Max per minute (CFWPM) CFWPM - Overall 1,203 18.5 0 103 CFWPM - Minus zero score students 1,051 21.1 1 103 Class CFWPM - Overall 400 9.0 0 50 Class 1 CFWPM - Minus zero score students 280 12.7 1 50 CFWPM - Overall 402 19.1 0 59 Class 2 CFWPM - Minus zero score students 376 20.4 1 59 CFWPM - Overall 401 27.6 0 103 Class 3 CFWPM - Minus zero score students 395 27.9 1 103 Gender CFWPM - Overall 607 21.5 0 103 Girls CFWPM - Minus zero score students 542 24.0 1 103 CFWPM - Overall 596 15.5 0 72 Boys CFWPM - Minus zero score students 510 18.0 1 72 28 29 Sub-test 5 – Invented Word Reading Invented or invented word reading is a measure of decoding ability and is designed to avoid the problem of sight recognition of words. Many children in the early grades learn to memorize or recognize by sight a broad range of words. Children’s decoding skills are often Sub-test 5 – Invented Word Reading assessed using reading lists of unrelated words. This allows for a purer measure of word recognition Invented and word or invented decoding is athan skills reading doesof measure reading comprehension decoding paragraphs, ability and is designed as children to avoid the problem are unable to of sight recognition of words. Many children in the early grades learn to memorize or recognize two- guess the next word from the context. This sub-test included 50 one- and by sight a syllable invented words, five per row, with the vowel-consonant patterns of letters typical to broad range of words. Children’s decoding skills are often assessed using reading lists of unrelated words. the Tongan language. This allows for a purer measure of word recognition and decoding skills than does reading comprehension paragraphs, as children are unable to guess the next word from the context. This sub-test included 50 one- and two-syllable invented words, five per row, with the vowel-consonant patterns of letters typical to the TEGRA results in this sub-test showed a slightly lower performance to the average scores in Tongan language. familiar word reading. Scores in sub-test 5 are calculated as the number of correct invented words read per minute (CIWPM). On average, students attempted to read 16 familiar words TEGRA results in this sub-test showed a slightly lower performance to the average scores in familiar word and were able to correctly read 19 familiar words sounds in a minute. Excluding the 180 reading. Scores in sub-test 5 are calculated as the number of correct invented words read per minute students with zero-scores (15% of the sample) the average score increases by 3 words. (CIWPM). On average, students attempted to read 16 familiar words and were able to correctly read 19 familiar words sounds in a minute. Excluding the 180 students with zero-scores (15% of the sample) the average score At the of Class by increases end 3 words. are able to read about 8 invented words correctly per minute. 1, students This average increases to 17 and 24 at the end of Class 2 and Class 3, respectively. On At the end of Class 1, students are able to read about 8 invented words correctly per minute. This average average, girls correctly read 5 invented words more than boys (19 and 14, respectively) and increases to 17 and 24 at the end of Class 2 and Class 3, respectively. On average, girls correctly read 5 had a wider (SD of 16 words in a range of 0 – 130 familiar words read correctly) than boys in invented words more the sample 12boys than (SD of (19 and familiar respectively) 14, in words a range of and had 0-72 a wider familiar (SD of read words 16 words in a range correctly). Thisof 0 – difference 130 familiar is similar words (6 invented than words) read correctly) if zero-score boys in the sample (SDstudents are excluded. of 12 familiar words in a range of 0-72 familiar words read correctly). This difference is similar (6 invented words) if zero-score students are excluded. Table 11 – Sub-test 5 Invented Word Reading: Results by Class and Gender Table 11 – Sub-test 5 Invented Word Reading: Results by Class and Gender Sub-test 5 - Number of correct unfamiliar words N Mean Min Max read per minute (CUWPM) CUWPM - Overall 1,203 16.3 0 67 CUWPM - Minus zero score students 1,023 19.0 1 67 Class CUWPM - Overall 400 8.3 0 49 Class 1 CUWPM - Minus zero score students 259 12.4 1 49 CUWPM - Overall 402 17.3 0 53 Class 2 CUWPM - Minus zero score students 372 18.6 1 53 CUWPM - Overall 401 23.5 0 67 Class 3 CUWPM - Minus zero score students 392 23.9 2 67 Gender CUWPM - Overall 607 18.7 0 67 Girls CUWPM - Minus zero score students 531 21.2 1 67 CUWPM - Overall 596 13.9 0 53 Boys CUWPM - Minus zero score students 492 16.6 1 53 Sub-test 6a – Oral Passage Reading 30 29 Sub-test 6a – Oral Passage Reading Oral reading Oral reading fluencyfluency of overall of is a measure is a measure overall reading reading competence: competence: the ability tothe ability translate to translate letters into sounds, letters into sounds, unify sounds into words, process connections, relate text to meaning, unify sounds into words, process connections, relate text to meaning, and make inferences to fill in and make inferences to fill in missing information (Hasbrouck & Tindal, 2006). As skilled missing information (Hasbrouck & Tindal, 2006). As skilled readers translate text into spoken language, readers translate text into spoken language, they combine these tasks in a seemingly combine these they effortless tasks manner. in a seemingly Because oral readingeffortless fluencymanner. captures Because oral reading this complex process,fluency it can captures be used this complex process, it overall to characterize can be used to characterize reading on skill. overall reading skill. Poor performance Poor performance a reading comprehension on a reading tool tool that would suggest comprehension wouldthe studentthat suggest hadthe trouble with student decoding, had or with trouble with reading decoding, fluently or enough with reading fluently to comprehend, or with vocabulary. enough to comprehend, or with vocabulary. Sub-test Sub-test 6a produced 6a produced a 60-word a 60-word narrative narrative passagepassage from children’s from children’s reading reading materials. materials. The narrative The passage narrative passage began where the characters are introduced, a middle section containing began where the characters are introduced, a middle section containing some dilemma, and an ending some dilemma, and an ending section with an action resolving the dilemma. The passage section with an action resolving the dilemma. The passage provided the basis for the comprehension provided the basis for the comprehension questions presented in sub-test 6b. questions presented in sub-test 6b. Scores in sub-test 6a are calculated as the number of correct words read in the 60-word oral Scores in sub-test reading 6a On passage. calculated are average, number as the were students ofto able correct read 27words wordsread in the correct 60-word words oral reading per minute passage. On average, (CWCPM). students Excluding 195 able thewere to read students 27 words with correct zero-scores words (16% of per theminute (CWCPM). sample), Excluding the average score the 195 increases students withby 5 words. At zero-scores theof (16% end theof Class 1, sample), students the averageare able score to read 12 increases by correct 5 words. words At the end of the oral reading passage in a minute. This average increases to 28 and 42 at the end of Class 1, students are able to read 12 correct words of the oral reading passage in a minute. This average of Classto increases and 2 28 andClass 42 at3,the respectively. end of Class On Class girls average, 2 and correctly On 3, respectively. read 10 more average, words girls of the correctly read 10 passage than boys (32 and 22, respectively) leading to a wider range of responses (0-148 more words of the passage than boys (32 and 22, respectively) leading to a wider range of responses (0- CWCPM) compared to that of boys (0-91 CWCPM) – a difference of 57 correct words read in 148 CWCPM) compared to that of boys (0-91 CWCPM) – a difference of 57 correct words read in the oral the oral reading passage. The difference is slightly larger (11 words) if zero-score students are passage. reading excluded. The difference is slightly larger (11 words) if zero-score students are excluded. – Sub-test Table 12Table 6a Oral 12 – Sub-test Passage 6a Oral Reading: Passage Reading: Results Results by and by Class Class and Gender Gender Sub-test 6a - Number of words correct words read per N Mean Min Max minute (CWCPM) in the oral reading passage CWCPM - Overall 1,203 27.1 0 148 CWCPM - Minus zero score students 1,008 32.0 1 148 Class CWCPM - Overall 400 11.8 0 120 Class 1 CWCPM - Minus zero score students 252 18.3 1 120 CWCPM - Overall 402 28.1 0 103 Class 2 CWCPM - Minus zero score students 367 30.5 2 103 CWCPM - Overall 401 41.8 0 148 Class 3 CWCPM - Minus zero score students 389 42.6 1 148 Gender CWCPM - Overall 607 31.9 0 148 Girls CWCPM - Minus zero score students 519 37.0 1 148 CWCPM - Overall 596 22.2 0 91 Boys CWCPM - Minus zero score students 489 26.6 1 91 Sub-test 6b – Reading Comprehension Average reading comprehension levels in sub-test 6b are largely explained by the poor 30 performance in the oral reading passage. Without sufficient skills to read into the text, Sub-test 6b – Reading Comprehension Average reading comprehension levels in sub-test 6b are largely explained by the poor performance in the oral reading studentspassage. Without focus on sufficient reading words skills to read into one-by-one, the text, students sometimes focus onBy letter-by-letter. reading words the time one-by- they one, reach the end sometimes of the text, students letter-by-letter. have By the time already they reachforgotten what the end of theystudents the text, read first. haveScores in sub- already forgotten test 6b are calculated as the percentage of correct answers in the sub-test -5 questions what they read first. Scores in sub-test 6b are calculated as the percentage of correct answers in the in sub- total. test -5 The number questions questions ofThe in total. number a of student received questions depended a student on the received number depended onof words the read number ofin words sub-test 6a, so that students had to respond only to questions related to the segment of the read in sub-test 6a, so that students had to respond only to questions related to the segment of the text text they were able to read. they were able to read. The early-stop The early-stop marker marker at 19 at was placed was placed 19 words words –i.e., –i.e., first inrow first row the in the text- text- so thatso that unable those unable those to correctly to correctly read any of the first 19 words received a zero-score in this task. Students that read any of the first 19 words received a zero-score in this task. Students that read the first 19 words with read the first 19 words with at least one word read correctly were given only first question of at least one word read correctly were given only first question of the sub-test. Students that read up to 28 the sub-test. Students that read up to 28 words -at least one of the first 19 correct- were given words -at the least first one questions. two of the first 19 correct- Only were who students the first given read alltwo narrative Only thequestions. students passage who read –a total all the of 60 narrative passage words- –a total were given all 60 ofof words- the were in questions given the all of the questions in the sub-test. sub-test. On average, students correctly answered 24% of the text they read. Excluding the 660 students with zero- On average, students correctly answered 24% of the text they read. Excluding the 660 scores the sample), (55% ofwith students the average zero-scores (55% of scores increase to the sample), the52%, so students average scores who were to increase able to complete 52%, so students the exercise whoofwere (45% complete the able tounderstand, the sample) exercise about on average, of the (45% half sample) of the understand, text they read. At the on end of classaverage, about half 1, the average of the student text they correctly read. At answered the of 10% end ofquestions. the class 1, theInaverage class 2,student correctly the average increases answered 10% of the questions. In class 2, the average increases considerably to considerably to 26% and by class 3, the average student correctly answered 44% of the questions. 26% and by On class 3, the average student correctly answered 44% of the questions. On average, girls average, girls performed better, correctly answering 34% of the questions while boys answered 22%. performed better, correctly answering 34% of the questions while boys answered 22%. Table 13 – Sub-test 6b Reading Comprehension: Results by Class and Gender Table 13 – Sub-test 6b Reading Comprehension: Results by Class and Gender Sub-test 6b - Percentage of overall reading comprehension in Lower Upper N Mean the oral reading passage bound Bound Percentage of correct answers - Overall 1,203 24% 22% 26% Percentage of correct answers - Minus zero score 543 52% 50% 55% students Class Percentage of correct answers - Overall 400 6% 5% 8% Class 1 Percentage of correct answers - Minus zero score 60 37% 31% 43% students Percentage of correct answers - Overall 402 24% 21% 27% Class 2 Percentage of correct answers - Minus zero score 184 50% 46% 54% students Percentage of correct answers - Overall 401 43% 40% 46% Class 3 Percentage of correct answers - Minus zero score 299 57% 54% 60% students Gender Percentage of correct answers - Overall 607 30% 27% 32% Girls Percentage of correct answers - Minus zero score 314 55% 52% 59% students Percentage of correct answers - Overall 596 19% 16% 21% Boys Percentage of correct answers - Minus zero score 229 48% 44% 51% students Looking at the distribution of the average percentages of correct answers presented in Table 8, we observe that an overwhelming majority of Class 1 (76%) have an average 31 comprehension of 0%. This proportion reduces significantly in Class 2 (50%) and Class 3 (23%). Less than 3% (7 out of 252 Class 1 students) have developed enough fluency in Looking at the distribution of the average percentages of correct answers presented in Table 8, we observe that an overwhelming majority of Class 1 (76%) have an average comprehension of 0%. This proportion reduces significantly in Class 2 (50%) and Class 3 (23%). Less than 3% (7 out of 252 Class 1 students) have developed enough fluency in reading to read all of the passage and provide correct answers to 4-5 of the question presented. Class brings about an increase in average reading comprehension as shown in a better distribution of Class 3 students at various levels of reading comprehension. In terms of gender, differences in average comprehension levels between boys and girls are evident at the 0%, 80% and 100% reading comprehension levels. 80% 3- Distribution of percentage of correct answers by class Figure 70% 60% 80% % students 50% 70% 40% 60% 30% % students 50% 20% 40% 10% 30% 0% 20% 0% 20% 40% 60% 80% 100% 10% Reading Comprehension (% correct responses) 0% 0% 20% 40% Class1 60% Class3 80% Class2 100% Reading Comprehension (% correct responses) Class1 Class2 Class3 Figure 4 - Distribution of percentage of correct answers by gender 60% 50% 60% 40% % students 50% 30% 40% 20% % students 30% 10% 20% 0% 0% 20% 40% 60% 80% 100% 10% Reading Comprehension (% correct responses) 0% 0% 20% 40% Girls 60% Boys 80% 100% Reading Comprehension (% correct responses) Girls Boys 32     Average  performance  in  this  subâ€?test  related  to  the  complexity  of  the  question  presented  and  Average performance the  order   in  which in this  sub-test it  was presented   –i.e.,to related the   questions   later of the question complexity  required   students presented   to  read and   more the  words order   in which it  was fluently. A  review   of  –i.e., presented later questions percentage   correct required answers   in  the to students 5  read more  words questions in  sub fluently. â€?test  6a A   review shows  ofquestion percentage  1  to   be  very correct   difficult answers   for in the 5  students questions in  answer   to sub-test   for all  students 6a   shows question the   in 1 to  sample,   including be very difficult for   those  in students   Class to answer A  possible   3. for all students  explanation in the sample, this  is  that   for including   question those in Class   13.   required A possible   student   to  show for   explanation mathematical this is that question   inference   along 1 required   with to student comprehension. show mathematical   The inference explicit  question along with   included   in  the  The comprehension. subâ€? test   had   the explicit question included in the sub-test had the smallest variance from students. Across grades, student     smallest   variance   from   students.   Across   grades,   student   comprehension   increases irrespective of comprehension  of question  the typeirrespective increases of asked. the type    of question asked.   Table 14 - Percentage Table of  Correct  14 â€? Percentage Answers of Correct by  Answers by Questions  Questions in Sub-test  in Sub â€?test 6b  6b   Class 1  Class 2 Class 3 Total Items   Girl  Boy  Total  Girl Boy Total Girl Boy Total  Girl  Boy Total RC Q1: How  many live in  Malakai's home?  0.08  0.09  0.08  0.15  0.13  0.14  0.23  0.16  0.20  0.16  0.13  0.15  Five  RC Q2: Where  did Malakai and  Mina go? To the  0.26  0.19  0.22  0.53  0.34  0.44  0.76  0.65  0.70  0.55  0.42  0.49  shop  RC Q3: What  happened to  Mina? She fell  0.11  0.08  0.10  0.41  0.22  0.32  0.62  0.42  0.53  0.41  0.26  0.34  and hurt herself  RC Q4: Why was  she hurt?  Malakai  0.05  0.05  0.05  0.31  0.16  0.24  0.53  0.33  0.43  0.33  0.20  0.27  challenged her  to a race  RC Q5: Who  fixed Mina's  wound? Their  0.04  0.01  0.03  0.21  0.07  0.15  0.46  0.18  0.33  0.26  0.10  0.18  grandparent  Note: Weighted means.     Looking at differences in the difficulty of the questions suggest question 1 was the most difficult item in the sub-test –i.e., lowest percentage of correct answers-, which may be one of the reasons behind the large number of incorrect answers to this question. Average comprehension in questions 2 to 5 show most students get at least 75% of the questions correctly. 34  A  closer look at differences in comprehension between students who were asked to answer questions based on the oral reading passage show important differences in the share of students in each category and the average comprehension based on questions asked. Table 15 shows the distribution of correct answers depending on the number of questions asked. 33 questions 2 to 5 show most students get at least 75% of the questions correctly.   A  closer  look  at  differences  in  comprehension  between  students  who  were  asked  to  answer  questions based on the oral reading passage show important differences in the share of students  in  each  category  and  the  average  comprehension  based  on  questions  asked.  Table  15  shows  the  distribution of correct answers depending on the number of questions asked.     Table 15 â€? Percentage of Correct Answers in Subâ€?test 6b    Table 15 - Percentage of Correct Answers in Sub-test 6b Overall  # questions asked  # questions correct  0  1  2  3  4  5  N  0  195  419  44  4  0  1  663  column %  100%  94%  21%  8%  0%  0%  1     27  138  6  6  0  177        6%  66%  12%  6%  0%  2        28  31  22  10  91           13%  62%  23%  5%  3           9  54  47  110              18%  56%  23%  4              15  104  119                 15%  51%  5                 43  43                    21%  Total (N)  195  446  210  50  97  205  1203    Numbers Numbers   in in   gray gray   show show   column column   percentages: percentages:   of of the   the 446   446  students students   that  correctly that correctly   read  between read between   1  –  1 – 19 words 19the in   words in  the  first first  segment segment of  the   oral  reading   of  thepassage, oral reading   passage, only 6%   only (27) were   6% able   (27) to   were answer   able the   to  answer question   the  correctly. question 195   correctly. students were not  195   students given   were  not any questions   given – zero   any  questions questions asked- as   –  zero they   questions correspond to  early asked â€?  as stop   they  cases. correspond On to  early average,   the more  stop   cases. fluent students  the   On  average, in reading   fluent morethey are  –i.e.,   in  reading receive   students 3 or more   are  –i.e., questions-,   they  the better receive  3  or  more comprehension   questions as more â€?,  the students are  better   comprehension able to as  moreto provide correct  answers   students not   all, most, if  are able   toquestions the   provide  correctthe asked:  answers  to most, proportion  if not in of students all, 0  and  the the questions  asked: 1 correct  proportion  thethan are less 15% for of  students columns 3, 4 in  the and 5. 0Among  and 1  correct  are  less  than  15%  for  columns  3,  4  and  5.  Among  the  most  fluent  (columns  5),  72%  are  the most fluent (columns 5), 72% are able to understand 80% or more of the text they read (51% with 4/5 able  to  understand  80%  or  more  of  the  text  they  read  (51%  with  4/5  correct  answers  and  21%  correct answers and 21% with correct answers to all questions). with correct answers to all questions).   Sub-test 7 – Listening Comprehension Sub-test 7 – Listening Comprehension Assessment of listening comprehension does not involve any reading from the student but involves the processing of oral language information only. Testing of listening comprehension separately from reading comprehension is important due to the different ways in which learners approach, process, and respond 35    text. More importantly, listening comprehension is an important contributor to reading comprehension, to which tends to increase with reading acquisition. The narrative passage in TEGRA’s sub-test 7 was about 30 words long and narrated an activity or event familiar to Tongan children. Students then responded to oral comprehension questions –5 in total. Scores in sub-test 7 are calculated as the percentage of correct answers in the sub-test. On average, students in the sample responded 55% of the questions correctly (Table 15). There is evidence of progression in listening comprehension abilities, shown in the significant increase of percentage of correct answers given by Class 1 (43% correct answers), Class 2 (56% correct answers) and Class 3 (68% correct answers). 34 increase of percentage of correct answers given by Class 1 (43% correct answers), Class 2 (56% correct answers) and Class 3 (68% correct answers). Table 16- Sub-test Table 7 Listening 16- Sub-test Comprehension: 7 Listening Results Comprehension: Results by Class by Class and Gender and Gender Sub-test 7 - Percentage of overall listening Lower Upper N Mean comprehension bound Bound Percentage of correct answers - Overall 1,203 55% 54% 57% Percentage of correct answers - Minus 1,111 60% 58% 61% zero score students Class Percentage of correct answers - Overall 400 43% 40% 45% Class 1 Percentage of correct answers - Minus 334 51% 48% 53% zero score students Percentage of correct answers - Overall 402 56% 54% 59% Class 2 Percentage of correct answers - Minus 386 58% 56% 61% zero score students Percentage of correct answers - Overall 401 68% 65% 70% Class 3 Percentage of correct answers - Minus 391 69% 66% 71% zero score students Gender Percentage of correct answers - Overall 607 54% 52% 56% Girls Percentage of correct answers - Minus 560 58% 56% 60% zero score students Percentage of correct answers - Overall 596 57% 55% 59% Boys Percentage of correct answers - Minus 551 61% 59% 63% zero score students As in sub-test 6b, average performance is related to the question presented. Overall, students were able to As in sub-test 6b, average performance is related to the question presented. Overall, students draw meaning of the story read to them. Of the 5 questions presented, only one (Question 3) showed average were able to draw meaning of the story read to them. Of the 5 questions presented, only one performance below 50% even among students in Class 3.16 A possible explanation for this performance (Question 3) showed average performance below 50% even among students in Class 3.16 A may be related to the way the enumerators read the story to children –either too slow or unintelligible. possible explanation for this performance may be related to the way the enumerators read the story to children –either too slow or unintelligible. 16 Based on pilot data, a common word for stones/sticks across the islands was included to improve understanding. 16 Based on pilot data, a common word for stones/sticks across the islands was included to improve understanding. 38 35 (Question 3) showed average performance below 50% even among students in Class 3.16 A possible explanation for this performance may be related to the way the enumerators read the story to children –either too slow or unintelligible. Table 17 - Percentage Table Correct ofof 17 - Percentage Answers Correct Answers by by Questions Questions in Sub-test in Sub-test 7 7 Class 1 Class 2 Class 3 Total Items Girl Boy Total Girl Boy Total Girl Boy Total Girl Boy Total LC Q1 Who's Peta's 0.39 0.45 0.42 0.60 0.53 0.57 0.65 0.76 0.70 0.55 0.58 0.56 friend? Siale LC Q2 Who wanted to eat 0.42 0.43 0.42 0.58 0.58 0.58 0.69 0.71 0.70 0.56 0.57 0.57 Mangoes? Peta LC Q3 What were they going to 0.24 0.30 0.27 0.32 0.36 0.34 0.40 0.50 0.45 0.32 0.39 0.35 throw to the trees? Sticks/stones LC Q4 What where they throwing stones/sticks 0.58 0.61 0.59 0.75 0.72 0.73 0.82 0.86 0.84 0.71 0.73 0.72 to the trees? to get the mangoes LC Q5 Why did Siale refused? The 0.44 0.40 0.42 0.58 0.62 0.60 0.65 0.74 0.69 0.56 0.59 0.57 neighbors would be mad at them Note: Weighted means. Sub-test 8 – Dictation Sub-test 8 – Dictation Dictation assessment Dictation assessment is frequently is frequently by by used used teachers teachers to to test test both both oral oral comprehension comprehension and and writing writing skills. skills. Students’ ability to hear sounds and correctly write the letters and words corresponding Students’ ability to hear sounds and correctly write the letters and words corresponding to the sounds to the sounds they hear demonstrates their success with the alphabetic principle. The dictation they hear demonstrates their success with the alphabetic principle. The dictation sentence in the TEGRA sentence in the TEGRA assessment was 10 words long and contained two difficult or irregular items: thewas assessment use of words 10the long glottal and stop contained and difficult two 17 the macron or irregular . Students received items: the use a weighted of the score glottal stop capturing and macron .for theaccuracy the 17 vowelreceived Students a weighted and consonant score capturing sounds, spelling,the accuracy spacing andfor direction vowel andof consonant text, sounds, spelling, spacing and direction of text, capitalization, and punctuation. As in sub-test 1 and 7, student performance in dictation was positive in terms of scores and overall progression across classes. 16 Based Scores on pilot data, in sub-test a common 8 are word using calculated for stones/sticks weights18 across the islands to create was included a variable with a improve understanding. tomaximum score of 100%. 17 The glottal stop (’) has a distinct sound in isolation and can modify the sound of letter when preceding it. The For the spelling macron component: ( ¯ ) serves spelling as a modifier ofincreasing of letters word kaukau, use the length of of glottal the soundsstop in the context of a word, and the of vowels. use of macron, each question receives 10 points (for a total of 60% of the total weights). The remaining 37use 4 items focused on writing conventions: capitalization and glottal stop, direction, spacing and the of the full stop, receive 5 points each (for a total of 40% of the total weights).19 Each of these weights were multiplied by the score, 0 for incorrect, 1 for partially correct or 2 for correct response, and added to obtain a final score for each student. The spelling component received more weight due to our interest in understanding reading ability. 36 component: spelling of word kaukau, use of glottal stop in the context of a word, and the use of macron, each question receives 10 points (for a total of 60% of the total weights). The remaining 4 items focused on writing conventions: capitalization and glottal stop, direction, spacing and the use of the full stop, receive 5 points each (for a total of 40% of the total weights).19 Each of these weights were multiplied by the score, 0 for incorrect, 1 for partially correct or 2 for correct response, and added to obtain a final score for each student. The spelling component received more weight due to our interest in understanding reading ability. On Onaverage, students average, in sample in the students responded the sample 50% of responded the questions 50% correctly. of the questions ExcludingExcluding the 123 students correctly. the 123 with with(10% zero-scores students of the sample), zero-scores (10% ofthe average the score sample), increases the average 55% (Table to score 13). There increases is evidence to 55% of (Table 13). There is evidence progression of progression in basic spelling and writing basic inskills, spelling shown and in the writing increase significant skills, shown in the significant of percentage of correct increase items percentage ofin written of correct Class 1 (27%), Class 2 items written (53%) and in Class 3Class (27%), (69%).1Girls Class 2 (53%) boysand outperformed and Class 3 (69%). the difference is Girls significant.boys and the difference is statistically significant. outperformed statistically Table 18 –Sub-test Table 8 Dictation: 18 –Sub-test Results 8 Dictation: Class Results by Class and and Gender Gender Sub-test 8 - Percentage of overall early writing skills (spelling Lower Upper N Mean and basic conventions) bound Bound Percentage of correct answers - Overall 1,203 50% 48% 51% Percentage of correct answers - Minus zero score 1,080 55% 53% 56% students Class Percentage of correct answers - Overall 400 28% 26% 31% Class 1 Percentage of correct answers - Minus zero score 312 36% 33% 38% students Percentage of correct answers - Overall 402 53% 50% 55% Class 2 Percentage of correct answers - Minus zero score 373 56% 54% 58% students Percentage of correct answers - Overall 401 69% 67% 71% Class 3 Percentage of correct answers - Minus zero score 395 69% 67% 71% students Gender Percentage of correct answers - Overall 607 52% 50% 55% Girls Percentage of correct answers - Minus zero score 556 57% 55% 59% students Percentage of correct answers - Overall 596 47% 45% 49% Boys Percentage of correct answers - Minus zero score 524 53% 51% 55% students Performance in spelling items shows a similar pattern to previous sub-tests, with significant progression Performance in spelling items shows a similar pattern to previous sub-tests, with significant across grades, especially from Class 1 to Class 3. However, results of basic writing skills indicate that while progression across grades, especially from Class 1 to Class 3. However, results of basic writing most students acquired an understanding of the orientation to write as early as Class 1 (77% correct answer), other skills such as capitalization and spacing show great variance even as late as Class 3. Moreover, the use of the full stop is not developed in almost all students (61% non-response in Class 3) although this is a 19 In spelling the use of the glottal stop had the purpose of altering the sound of a letter and a marker to separate two frequent element during dictation in the classroom20. Although gender differences are present, the average words. As a convention, it was considered as part of the capital letter at the beginning of the sentence. patterns in performance are similar. 17 40 The glottal stop (’) has a distinct sound in isolation and can modify the sound of letter when preceding it. The macron ( ˉ ) serves as a modifier of letters increasing the length of the sounds of vowels. 18 Due to similarities in the components used in the TEGRA instrument, the weights for this score follow the score used in the Guyana EGRA test. 19 In spelling the use of the glottal stop had the purpose of altering the sound of a letter and a marker to separate two words. As a convention, it was considered as part of the capital letter at the beginning of the sentence. 20 During the pilot of the instrument, the team monitored that enumerators were not dictating the final point to students, as it is often the way teachers do during dictation lessons. 37 developed spacing during element in almost show great all students variance dictation (61% even in the non-response as late classroom as 20 Class 3.in . Although Moreover, gender Class 3)differences although this the use ofare is the a frequent full stop present, is not the element during developed in dictation in the classroom 20 . Although gender differences are present, the average Figure 5 –almost patterns all students in performance Percentage are(61% of Correct non-response similar. Answers in Each Taskinof Class 3) although Sub-test and is 8 by Classthis a frequent Gender average patterns in performance are similar. element during dictation in the classroom20. Although gender differences are present, the average patterns in performance are similar. Figure 5 – Percentage of Correct Answers in Each Task of Sub-test 8 by Class and Gender Figure 5 – Percentage of Correct Answers in Each Task of Sub-test 8 by Class and Gender Figure 5 – Percentage of Correct Answers in Each Task of Sub-test 8 by Class and Gender Figure 5 – Percentage of Correct Answers in Each Task of Sub-test 8 by Class and Gender Summary of Assessment Results 20 During the pilot TEGRA of the results instrument, provide ofteam the evidence both monitored that enumerators good and poor performance were not dictating in reading the final abilities among point to beginning 20 During the as it is in students,readers pilot of often the instrument, way teachers Tonga. Across the grades,do team students monitored during dictation that perform well enumerators lessons. were not dictating the final point to on three fundamental skills: letter name knowledge, 0 students, as it is often the way teachers do during dictation lessons. During the pilot of listening the instrument, comprehension, the team and basic writingmonitored thatmost skills. However, enumerators were not students struggle dictating to identify the final of point to the sounds tudents, as it is often the way teachers do during dictation lessons. letters when presented as both isolated letters and as part of a larger word. 38 Students also struggle to read 20 During the pilot of the instrument, the team monitored that enumerators were not dictating the final point 38 to familiar and invented words, which extends to errors in reading fluency, reading comprehension and spelling. students, as it is often the way teachers do during dictation lessons. 38 Comparing across groups, there are significant gains in reading from Class 1 to Class 3, and girls appear to perform better than boys; however, there are no differences in reading skills across school types and the 38 regions of Tonga. In all eight skills, there is evidence of reading progression across classes as students in Class 2 perform better than students in Class 1, and students in Class 3 perform better than students in Class 2. The fraction of fluent girls is almost three times that of boys. 38 4 and Reading Comprehension Fluency and comprehension As stated before, oral reading fluency is a measure of overall reading competence: the abilit to translate letters into sounds, unify sounds into words, process connections, relate text t meaning, and make Chapter 4: Performance inferences in Oral to fill in missing Reading information Fluency (Hasbruck and Tindal, idem Fuschs et al, 2001). This is notand Reading to say, however,Comprehension that oral reading fluency is the only predicto of reading comprehension among readers. Because oral reading fluency captures th complex process and it is strongly associated to both listening and reading comprehension Fluency and comprehension it is typically used as a marker of overall reading ability. As stated before, oral reading fluency is a measure of overall reading competence: the ability to translate For more researched languages, letters into sounds, unify sounds specific into words, benchmarks process connections,of oral relate reading fluency text to meaning, have alread and make inferences to fill in missing information (Hasbruck and Tindal, idem; Fuschs et al, 2001). This is not to say, been developed. These benchmarks help educators monitor student progression in readin however, that oral reading fluency is the only predictor of reading comprehension among readers. Because school during the oral fluencyto readingyear identify captures students this complex process atandrisk of failing it is strongly associated to tobecome literate. both listening and reading Until 200 oral reading had not been comprehension, measured it is typically used as a in Tonga marker which of overall reading made ability. it impossible to have a Tongan language standard for oral For more researched reading languages, fluency. specific benchmarksFor of this reason, oral reading fluency we have sought to developed. identify what ar already been the oral reading fluency These benchmarks levels monitor help educators where Tongan student progression children in reading duringdemonstrate the school year to higher identify readin studentslevels. comprehension at risk of Using data from failing to become sub-test literate. Until 2009, oral 6a andhad reading 6b, been observed not we that childre measured in Tonga which made it impossible to have a Tongan-language standard for oral reading fluency. For this reason, reading less than 40 correct words per minute in the oral reading passage understand, o we sought to identify what are the oral reading fluency levels where Tongan children demonstrate higher average, 40% or less reading of what comprehension they levels. read. Using Conversely, data from sub-test 6a and students that that 6b, we observed read least children 50 correct reading less word understand per minutethan 40 correct words about 60% per minute or oral in the more readingofpassage the text they read understand, (Figure on average, 40% or5). Even less of what though small group students ofread. they Conversely, read between students 35 and that read least 50 correct 50 correct words words per minute per minute understand about 60% and showed 60% or more of the text they read (Figure 5). Even though a small group of students read between 35 and 50 correct or more understanding of the text, all the students who were able to understand all of th words per minute and showed 60% or more understanding of the text, all the students who were able to text read more thanall50 understand correct of the text readwords more than per minute. 50 correct words per minute. Figure 6 – Figure 6 – Percentages Percentages of of Average Average Reading Reading Comprehension byComprehension Number of Correct Words Read by Number of Correct Words Per MinuteRead in thePer Minute Oral Readingin the Oral Reading Passage Passage 39 Using a reference standard of 50 correct words per minute to classify students as “fluen Using a reference standard of 50 correct words per minute to classify students as “fluent readersâ€?, we able to understand observe that with more of fewof than 60% the exception the text students who they read 50read (Figure correct 7).minute words per Moreover, students but showed little able or no understanding of the text, students at this fluency level are able to understand more than 60% of the to provide correct answers to all the questions asked read between 50 and 139 correct words text they read (Figure 7). Moreover, students able to provide correct answers to all the questions asked read per minute (average fluency at 67 CWPM). between 50 and 139 correct words per minute (average fluency at 67 CWPM). able to understand more than 60% of the text they read (Figure 7). Moreover, students able to provide –answers correct Figure 77 Figure Average – the questions all toReading Average Reading asked read Comprehension Comprehension between Levels Levels in in 50 andStudents Fluent Fluent 139 correct words Students per minute (average fluency at 67 CWPM). Figure 7 – Average Reading Comprehension Levels in Fluent Students In terms In terms of theirofdistribution their distribution in the in the sample, sample, fluent fluent students students -i.e., reading -i.e., reading at at a reference a reference standard of at least standard of at least 50 CWCPM - account for 17% of the sample. Differences 50 CWCPM - account for 17% of the sample. Differences by class level show that fluency emerges among by class level show that fluency emerges among students in Class 3 but only for a third of them. In terms of studentstheir 3 but only for in distribution in Class the a third of sample, fluent students them. Overwhelmingly, 25% of the Overwhelmingly, 25% of the girls in the sample can be considered fluent in reading, -i.e., reading girls in the sample at cana be reference standard of at least considered whereas fluent only50 in9% CWCPM reading, the boyswhereas- account only the achieved thefor 9% fluencyboys17% of standard. achieved reference the sample. the fluency Differences reference standard. by class level show that fluency emerges among students in Class 3 but only for a third of them. Table 19-19-Distribution of Students by by Fluency Overwhelmingly, 25% of the girls Table in the sample Distribution of Students can be considered fluent in reading, Fluency whereas only 9% the boys achieved the Less than fluency reference standard. fluent Fluent students >50 correct words in a narrative Reading 0 to 49 words per minute passage Table Mean (%) 19- SD Distribution (%) Nof Students by Fluency Mean (%) SD (%) N Overall 83% 38% 1,203 17% 38% 1,203 Class1 Less 98% fluent than 14% 400 2% Fluent students 14% 400 Class2 85% 36% 402 36% 15% >50 correct words402 in a narrative Class3 0 to 49 words Reading66% 47% per minute 401 34% 47% passage 401 Female 75% 43% 607 25% 43% 607 Mean (%) SD (%) N Mean (%) SD (%) N Male 91% 28% 596 9% 28% 596 Overall 83% 38% 1,203 17% 38% 1,203 Class1 40 Excluding the98% 14% early-stop cases 400 brings slight from the sample 2% 14% changes to the 400 distribution of students in the36% Class2fluent (20% 85% sample) and less-than-fluent 402 of the sample). students (80% 36% 15% 402 Differences by class level remain almost the same between fluent and less-than-fluent Class3 66% 47% 401 34% 47% 401 Excluding the early-stop cases from the sample brings slight changes to the distribution of fluent (20% students in the sample) and less-than-fluent students (80% of the sample). Differences by class level remain almost the same between fluent and less-than-fluent students. The ratio of fluent girls to fluent boys remains also the same (about three-times that of boys). Table 20- Percent Distribution of Students by Fluency (no zero-score students) Table 20- Percent Distribution of Students by Fluency (no zero-score students) Table 20- Percent Distribution of Students by Fluency (no zero-score students) Less than Fluent Students Fluent Students Less than Fluent Students Fluent Students >50 correct words in a narrative Reading 1 to 49 words per minute >50 correct words in a narrative passage Reading 1 to 49 words per minute passage Mean (%) SD (%) N Mean (%) SD (%) N Mean (%) SD (%) N Mean (%) SD (%) N Overall 80% 40% 1,008 20% 40% 1,008 Overall 80% 40% 1,008 20% 40% 1,008 Class1 97% 18% 252 3% 18% 252 Class1 97% 18% 252 3% 18% 252 Class2 83% 37% 367 17% 37% 367 Class2 83% 37% 367 17% 37% 367 Class3 66% 48% 389 34% 48% 389 Class3 66% 48% 389 34% 48% 389 Female 71% 45% 518 29% 45% 519 Female 71% 45% 518 29% 45% 519 Male 90% 31% 489 10% 31% 489 Male 90% 31% 489 10% 31% 489 Note: This table excludes students in the early-stop group for CWCPM Note: This table excludes students in the early-stop group for CWCPM Differences in the number of words are also significant across grades and gender. While less-than-fluent Differences students in read an in the number average of words of 23 CWCPM, are also fluent significant students read 67 across CWCPM grades and gender. –a difference While in fluency Differences the number of words are also significant across grades and gender. Whileof 44 less-than-fluent students read an average of 23 CWCPM, fluent students read 67 CWCPM –a less-than-fluent correct read words per students minute inread the an oralaverage readingof 23 CWCPM, passage. fluent In terms students of average read 67 CWCPM comprehension –a while levels, difference in fluency of 44 correct read words per minute in the oral reading passage. In difference in less-than-fluent fluency students doof 44 correct increase theirread words average per minuteas comprehension inththe oral reading eir fluency passage. average level increases, In terms of average comprehension levels, while less-than-fluent students do increase their terms reading of average comprehension comprehension for fluent students levels, remainswhile less-than-fluent around 70% atincreases, to 80%. students do increase their average comprehension as their fluency level average reading comprehension for average comprehension as their fluency level increases, average reading comprehension for fluent students remains at around 70% to 80%. fluent students remains at around 70% to 80%. Table 21 - Average Table Fluency 21 - Average Levels Fluency (Fluent Levels Vs. Less Vs. (Fluent than Less than Fluent) Fluent) Excluding Excluding Zero-score Students Students Zero-score Table 21 - Average Fluency Levels (Fluent Vs. Less than Fluent) Excluding Zero-score Students Less than Fluent Students Fluent Students Less than Fluent Students Fluent Students Reading 1 to 49 words per minute >50 correct words in a narrative passage Reading 1 to 49 words per minute >50 correct words in a narrative passage Mean SD N Mean SD N Mean SD N Mean SD N Overall 23 13 815 67 15 192 Overall 23 13 815 67 15 192 Class1 17 12 244 60 8 7 Class1 17 12 244 60 8 7 Class2 24 12 308 65 13 59 Class2 24 12 308 65 13 59 Class3 29 12 263 68 16 126 Class3 29 12 263 68 16 126 Female 24 13 377 69 15 141 Female 24 13 377 69 15 141 Male 22 13 438 62 13 51 Male 22 13 438 62 13 51 Table 22 – - Average Reading Comprehension (Fluent Vs. Less than Fluent) Excluding Zero-score Students Table 22 – - Average Reading Comprehension (Fluent Vs. Less than Fluent) Excluding Zero-score Students Less than Fluent Students Fluent Students 41 Less than Fluent Students Fluent Students Reading 1 to 49 words per minute >50 correct words in a narrative passage Reading 1 to 49 words per minute >50 correct words in a narrative passage Table 22 Table - –Average 22 - AverageReading Comprehension Reading Comprehension (Fluent (Fluent Vs. than Vs. Less Less than Fluent) Fluent) Excluding Excluding Zero-score Zero-score Students Students Less than Fluent Students Fluent Students Reading 1 to 49 words per minute >50 correct words in a narrative passage Mean (%) SD (%) N Mean (%) SD (%) N Overall 16% 22% 815 79% 16% 192 Class1 7% 15% 244 84% 22% 7 Class2 16% 23% 308 76% 15% 59 Class3 25% 25% 263 80% 15% 126 Female 16% 21% 377 79% 15% 141 Male 16% 23% 438 78% 16% 51 Interestingly, once students read at a fluent level, word reading gains among fluent students are not Interestingly, once students read at a fluent level, word reading gains among fluent students are substantial (5 words from Class 1 to Class 2, and 3 more from Class 2 to Class 3) which may suggest that not substantial (5 words from Class 1 to Class 2, and 3 more from Class 2 to Class 3) which may once students reach fluency in oral reading, these remain constant in the following class levels. All in all, suggest that once students reach fluency in oral reading, these remain constant in the following average scores in CWCPM and reading comprehension levels suggest that although the system seems to class levels. All in all, average scores in CWCPM and reading comprehension levels suggest that contribute to develop fluency in reading, once Tongan children are able to read fluently, instruction in the although the system seems to contribute to develop fluency in reading, once Tongan children classroom is falling short to support their transition onto more complex texts and/or promoting adequate are able to read fluently, instruction in the classroom is falling short to support their transition reading comprehension onto more complex textsstrategies. and/or promoting adequate reading comprehension strategies. In summary, there is much variation in reading fluency and comprehension among Tongan students. Greater oral reading fluency is associated with higher levels of reading comprehension, with fluent students— In summary, identified as thethere 17% of much variation is students who werein reading able to read fluency and at least 50 comprehension correct among Tongan words per minute—being able students. to comprehend Greater oral of about 80% reading what they fluency is associated read, compared with higherstudent to only less-than-fluent levels who ofarereading able to comprehension, with fluent students—identified as the 17% of students who were able understand only 16% of what they read. The results suggest that reading fluency differs across both classes to read at and least 50 correct gender. words In particular, theper minute—being results able provide evidence to schooling that comprehend about increases 80% levels, fluency of whatas they read, students in compared to only less-than-fluent student who are able to understand only 16% of what Class 3 demonstrated the highest fluency levels while students in Class 1 demonstrated the lowest fluency they read. The results suggest that reading fluency differs across both classes and gender. In levels. Girls also exhibited greater reading fluency than boys. However, comparing at the same fluency level, particular, the results provide evidence that schooling increases fluency levels, as students in girls and boys exhibited the same level of reading comprehension, suggesting that while reading fluency Class 3 demonstrated the highest fluency levels while students in Class 1 demonstrated the varies across gender, reading comprehension depends on reading fluency rather than gender. lowest fluency levels. Girls also exhibited greater reading fluency than boys. However, comparing at the same fluency level, girls and boys exhibited the same level of reading comprehension, suggesting that while reading fluency varies across gender, reading comprehension depends on reading fluency rather than gender. 42 46 5 Chapter 5: Analysis of Student Factors Associated with Better Reading Scores Chapter 5: Analysis of Student Factors Associated with Better We sought to explore the association between students’ characteristics and performance using data collected Reading in the student questionnaire. Students were Scores asked questions on their background such as their parents’ Wewhether literacy, sought they attended to explore kindergarten, the associationand whether between books for school they had characteristics students’ and and other books. We performance using data also include collected questions in the socio-economic student questionnaire. measuring Students status (SES) were such as asked asset questions ownership; on their however, the low background variation in the SESsuch as their indicators parents’ prevented us literacy, whether from creating they of an index attended kindergarten, SES. Below, and of is the distribution whether they had books for school and relevant variables among students in the sample. other books. We also include questions measuring socio-economic status (SES) such as asset ownership; however, the low variation in the SES indicators prevented us from creating an index of SES. Below, is the distribution of relevant variables among students in the sample. Table 23 - Characteristics of students in the sample along several student and family factors Table 23 - Characteristics of students in the sample along several student and family factors Mean (%) SD N Student speaks the same language at home as in the school 98 14 1,189 Student has a literate family member: Student has a literate mother 62 48 1,203 Student has a literate father 47 50 1,203 Student has both parents literate 39 49 1,203 Student has at least one literate sibling 59 49 1,203 Student has other literate family member (e.g., grandparent) 2 14 1,203 Student is in overage (as a proxy for repetition) 6 23 1,203 Student attended kindergarten before Class 1 64 48 1,188 Student has books at home 67 47 1,185 If yes to above, language in books at home are written: Tongan 59 49 1,203 English 14 35 1,203 Student was absent from school for more than 1 week 33 47 1,129 Student has the School Reader 84 37 1,187 Student reported s/he does homework 95 21 1,196 Student reported s/he gets help from a family member to do his/her homework: 88 33 1,133 From his/her mother 59 49 1,203 From his/her father 27 44 1,203 From his/her sibling 37 48 1,203 From another relative of his 1 9 1,203 Note: Missing data explains cases where N is less than 1,203. We explored how nine student characteristics contribute to performance in the following six sub-tests: 1) number of letters recognized per minute; 2) number of sounds recognized in a 43 We explored how nine student characteristics contribute to performance in the following six sub-tests: 1) number of letters recognized per minute; 2) number of sounds recognized in a word (out of 10 words); 3) number of letter sounds recognized per minute; 4) number of familiar words recognized per minute; 5) word of number invented (out of 10 words words); recognized 3) number ofminute; per and 6) number letter sounds recognizedof correct words 4) per minute; read in a connected number of passage familiarper minute. words In general,per recognized students minute;with 5) each numberof those factors performed of invented better on words recognized theminute; sub-tests, but per and these 6) numberwere differences of correct read in a connected wordsstatistically not always significant.passage per minute. Family literacy seems Into general, students affect boys more than with each of those factors performed better girls, and family literacy seems to strongly influence student performance in sound recognition not on the sub-tests, but these differences were in a word. always statistically significant. Family literacy seems to affect boys more than girls, Having a literate mother is associated with better performance for each level and for both boys and girls. and family literacy seems to strongly influence student performance in sound recognition in a However, the p-values presented below show that the differences are not statistically significant except for word. Having a literate mother is associated with better performance for each level and for boys’ performance in letter recognition and sound recognition in a word. both boys and girls. However, the p-values presented below show that the differences are Wenotusedstatistically significant sample weights except for all the for because analysis of the samplein boys’ performance letter recognition stratification present inand sound the data. 21 Tables recognition in a word. 5.1 – 5.10 compare the mean scores by class and gender of the six sub-tests for the group of students whoWehadused the sample weights factor –e.g., for all students the with a analysis because literate mother- of the group with sample ofstratification students whopresent did not in have the data. students the –e.g., factor 21 Tables 5.1 – 5.10 without a compare the mean literate mother. scores Table by class 24 below and gender summarizes of the six the statistical sub-tests found differences for the group of students who had the factor –e.g., students with a literate mother- in TEGRA results which are marked in tables 5.1 to 5.10 in bold (p<5%), and bold and italicized with the (p<1%). group of students who did not have the factor –e.g., students without a literate mother. Table 24 below summarizes the statistical differences found in TEGRA results which are marked in tables 5.1 to 5.10 in bold (p<5%), and bold and italicized (p<1%). Table 24 – Summary of ANOVA Results by Student Factors Table 24 – Summary of ANOVA Results by Student Factors Table Class Gender Statistical differences in better performance of boys in 5.1 - Literate mother No statistical differences sub-tests 1 and 2; no differences among girls Statistical differences in better performance of boys in 5.2 - Literate father Statistical differences in sub-test 2 for Class 1 students sub-tests 1,2, and 3; no differences among girls Statistical differences in better performance of boys in 5.3 - Both parents are literate Statistical differences in sub-test 2 for all class levels sub-tests 1,2,3 and 4; for girls only in sub-test 2 Statistical differences in better performance of boys in all 5.4 - Literate sibling Statistical differences in sub-test 2 for all class levels sub-tests; for girls, in sub-test 2 and 5 Statistical differences in sub-tests 1, 4,5,6 for Class 3 Statistical differences in better performance of boys in 5.5 - Books at home students sub-tests 1, 4 and 6; for girls in sub-test 1,4,5, and 6 Statistical differences in sub-tests 1,5 and 6 in Class 1; and Statistical differences in better performance of boys in all 5.6 - Does homework in sub-test 1,4,5,6 in Class 2; no statistical differences for sub-tests except sub-test 2; for girls, only in sub-test 1 Class 3 Statistical differences in better performance of boys in 5.7 - Family helps with homework No statistical differences sub-tests 1 , 4 and 6; no differences among girls Statistical differences in sub-test 2 for all class levels, and Statistical differences in worse performance of boys in 5.8 - Attended kindergarten sub-test 4 for Class 3 students sub-tests 2; for girls, in sub-test 4 Statistical differences in sub-tests 5 and 6 for Class 2 5.9 - Overage No statistical differences students; and sub-tests 4,5,6 in Class 3 students Statistical differences in sub-test 3 for Class 2 studnets; Statistical differences in worse performance of boys only 5.10 - Absenteesm and sub-tests 3 and 6 for Class 3 students in sub-test 3 21 Since simple t-tests would not allow us to use sample weights, we used a regression to determine the significance of student characteristics. The p-value on the coefficient of an unweighted regression is equivalent to the p-value of a two-sided t-test, so using p-values from a weighted regression would allow us to use the sample weights and test for the significance of the factors. 44 21 Since simple t-tests would not allow us to use sample weights, we used a regression to determine the significance of student characteristics. The p-value on the coefficient of an unweighted regression is equivalent to the p-value of a two-sided t-test, so using p-values from a weighted regression would allow us to use the Table 25 - Results from differences in average scores determined by factor (tables 5.1 to 5.10) Table 25 - Results from differences in average scores determined by factor (tables 5.1 to 5.10) Table 5.1 Student Factor 1 - Mother's literacy By Class By Gender Group with Factor Group without Factor Group with Factor Group without Factor Subtest Class 1 Class 2 Class 3 Total Class 1 Class 2 Class 3 Total Girls Boys Total Girls Boys Total Letter Recognition 37.8 59.6 69.4 55.7 35.9 58.1 69.5 54.1 58.4 53.0 55.7 59.2 48.6 54.1 Sound Recognition in a Word 3.3 4.5 5.2 4.4 2.7 3.9 5.1 3.9 4.4 4.3 4.4 4.2 3.5 3.9 Letter Sound Recognition 17.1 23.3 25.6 22.0 15.1 23.2 24.1 20.7 23.2 20.8 22.0 22.3 19.1 20.7 Familiar Word Recognition 9.0 19.5 26.7 18.5 9.1 18.6 29.6 18.7 20.9 16.0 18.5 22.5 14.7 18.7 Unfamiliar Word Recognition 8.4 17.5 23.1 16.4 8.1 17.2 24.2 16.3 18.2 14.5 16.4 19.4 12.9 16.3 Words in a Connected Passage 12.0 28.7 40.4 27.1 11.5 27.3 44.2 27.1 31.6 22.5 27.1 32.3 21.6 27.1 Note: bold numbers denote significant differences between groups at p < 5%; bold and italicized numbers denote significant differences between groups at p < 1%. Table 5.2 Student Factor 2 - Father's literacy By Class By Gender Group with Factor Group without Factor Group with Factor Group without Factor Subtest Class 1 Class 2 Class 3 Total Class 1 Class 2 Class 3 Total Girls Boys Total Girls Boys Total Letter Recognition 37.8 59.6 69.8 55.8 36.4 58.6 69.1 54.5 57.5 54.0 55.8 59.7 48.9 54.5 Sound Recognition in a Word 3.6 4.6 5.5 4.6 2.7 4.0 4.8 3.8 4.6 4.5 4.6 4.2 3.5 3.8 Letter Sound Recognition 16.3 23.6 25.9 22.0 16.3 22.9 24.3 21.2 21.6 22.3 22.0 23.9 18.2 21.2 Familiar Word Recognition 9.0 19.5 26.9 18.6 9.0 18.8 28.5 18.6 20.5 16.6 18.6 22.3 14.6 18.6 Unfamiliar Word Recognition 8.4 17.3 23.0 16.3 8.2 17.4 24.0 16.4 17.9 14.7 16.3 19.3 13.2 16.4 Words in a Connected Passage 12.4 29.8 40.8 27.7 11.2 26.8 42.7 26.6 32.0 23.5 27.7 31.8 20.9 26.6 Note: bold numbers denote significant differences between groups at p < 5%; bold and italicized numbers denote significant differences between groups at p < 1%. 45 47 46 Table 5.3 Student Factor 3 - Literacy in both parents By Class By Gender Group with Factor Group without Factor Group with Factor Group without Factor Subtest Class 1 Class 2 Class 3 Total Class 1 Class 2 Class 3 Total Girls Boys Total Girls Boys Total Letter Recognition 38.5 59.3 68.8 56.1 36.2 58.9 69.9 54.4 58.2 54.1 56.1 59.0 49.5 54.4 Sound Recognition in a Word 3.8 4.9 5.5 4.8 2.7 3.9 4.9 3.8 4.8 4.8 4.8 4.1 3.4 3.8 Letter Sound Recognition 17.5 23.6 26.8 22.8 15.6 23.0 23.8 20.7 22.5 23.1 22.8 23.0 18.2 20.7 Familiar Word Recognition 9.6 20.0 27.0 19.2 8.7 18.6 28.3 18.2 21.4 17.0 19.2 21.5 14.5 18.2 Unfamiliar Word Recognition 9.0 17.6 22.8 16.8 7.9 17.2 24.0 16.1 18.5 15.0 16.8 18.8 13.1 16.1 Words in a Connected Passage 12.8 30.6 40.3 28.5 11.2 26.6 42.9 26.3 33.2 23.8 28.5 31.1 21.1 26.3 Note: bold numbers denote significant differences between groups at p < 5%; bold and italicized numbers denote significant differences between groups at p < 1%. Table 5.4 Student Factor 4 - Literacy in at least one sibling By Class By Gender Group with Factor Group without Factor Group with Factor Group without Factor Subtest Class 1 Class 2 Class 3 Total Class 1 Class 2 Class 3 Total Girls Boys Total Girls Boys Total Letter Recognition 37.3 59.8 69.4 57.3 36.8 58.0 69.5 52.0 61.0 53.6 57.3 55.6 47.9 52.0 Sound Recognition in a Word 3.7 4.8 5.5 4.8 2.5 3.5 4.5 3.3 4.9 4.7 4.8 3.7 2.9 3.3 Letter Sound Recognition 17.4 24.0 24.9 22.5 15.3 22.2 25.5 20.1 23.4 21.7 22.5 22.1 17.9 20.1 Familiar Word Recognition 9.5 19.4 28.0 20.0 8.5 18.7 27.0 16.5 22.8 17.2 20.0 19.7 13.0 16.5 Unfamiliar Word Recognition 8.8 17.7 23.8 17.6 7.8 16.9 22.9 14.6 19.9 15.3 17.6 17.0 11.8 14.6 Words in a Connected Passage 12.0 28.7 41.9 29.1 11.5 27.4 41.6 24.3 33.7 24.5 29.1 29.3 18.6 24.3 Note: bold numbers denote significant differences between groups at p < 5%; bold and italicized numbers denote significant differences between groups at p < 1%. 48 Table 5.5 Student Factor 5 - Student has books at home By Class By Gender Group with Factor Group without Factor Group with Factor Group without Factor Subtest Class 1 Class 2 Class 3 Total Class 1 Class 2 Class 3 Total Girls Boys Total Girls Boys Total Letter Recognition 38.1 60.9 71.2 57.6 35.8 55.9 65.2 50.7 61.0 53.7 57.6 54.1 47.5 50.7 Sound Recognition in a Word 3.3 4.4 5.3 4.4 3.0 4.1 4.8 3.9 4.6 4.2 4.4 4.0 3.7 3.9 Letter Sound Recognition 17.0 24.1 25.7 22.5 15.4 21.9 23.9 20.0 23.6 21.3 22.5 21.4 18.6 20.0 Familiar Word Recognition 9.5 19.8 29.4 20.0 8.3 18.0 23.1 15.7 23.2 16.5 20.0 17.7 13.9 15.7 Unfamiliar Word Recognition 8.6 17.7 24.7 17.4 7.7 16.7 20.4 14.3 20.0 14.5 17.4 15.8 12.9 14.3 Words in a Connected Passage 12.7 29.2 44.5 29.6 10.3 26.2 34.6 22.4 34.9 23.6 29.6 25.3 19.7 22.4 Note: bold numbers denote significant differences between groups at p < 5%; bold and italicized numbers denote significant differences between groups at p < 1%. Table 5.6 Student Factor 6 - Student does homework By Class By Gender Group with Factor Group without Factor Group with Factor Group without Factor Subtest Class 1 Class 2 Class 3 Total Class 1 Class 2 Class 3 Total Girls Boys Total Girls Boys Total Letter Name Knowledge 38.0 60.2 69.6 56.1 26.4 36.3 65.1 37.0 59.4 52.7 56.1 46.2 28.9 37.0 Initial Sound Recognition 0.3 0.4 0.5 0.4 0.2 0.3 0.5 0.3 0.4 0.4 0.4 0.4 0.3 0.3 Letter Sound Recognition 16.4 23.6 25.2 21.8 16.2 17.4 22.6 17.8 23.0 20.6 21.8 21.8 14.2 17.8 Familiar Word Reading 9.2 19.6 27.6 18.9 7.2 11.2 32.3 13.2 21.7 15.9 18.9 17.1 9.7 13.2 Invented Word Reading 8.6 17.7 23.5 16.7 4.0 10.3 26.7 10.4 19.0 14.3 16.7 13.3 7.8 10.4 Words in Oral Reading Passage 12.2 28.8 41.7 27.7 6.2 15.5 46.9 17.0 32.3 22.8 27.7 23.0 11.7 17.0 Note: bold numbers denote significant differences between groups at p < 5%; bold and italicized numbers denote significant differences between groups at p < 1%. 47 49 48 Table 5.7 Student Factor 7 - Student receives help with homework at home By Class By Gender Group with Factor Group without Factor Group with Factor Group without Factor Subtest Class 1 Class 2 Class 3 Total Class 1 Class 2 Class 3 Total Girls Boys Total Girls Boys Total Letter Name Knowledge 37.7 60.0 69.6 56.6 38.6 62.4 71.8 52.8 59.2 53.7 56.6 62.1 45.9 52.8 Initial Sound Recognition 3.1 4.4 5.2 4.2 3.7 4.3 5.1 4.2 4.4 4.0 4.2 4.3 4.1 4.2 Letter Sound Recognition 16.5 23.7 25.3 22.1 15.8 21.8 23.2 19.2 23.3 20.8 22.1 19.7 18.9 19.2 Familiar Word Reading 9.0 19.7 27.4 19.2 10.1 18.8 33.0 17.6 21.7 16.4 19.2 23.5 13.1 17.6 Invented Word Reading 8.3 17.7 23.2 16.8 10.1 17.9 28.6 16.4 18.7 14.6 16.8 21.7 12.4 16.4 Words in Oral Reading Passage 11.9 28.7 41.3 28.1 13.2 30.1 48.8 25.8 32.2 23.5 28.1 35.2 18.8 25.8 Note: bold numbers denote significant differences between groups at p < 5%; bold and italicized numbers denote significant differences between groups at p < 1%. Table 5.8 Student Factor 8 - Student attended kindergarten By Class By Gender Group with Factor Group without Factor Group with Factor Group without Factor Subtest Class 1 Class 2 Class 3 Total Class 1 Class 2 Class 3 Total Girls Boys Total Girls Boys Total Letter Name Knowledge 36.1 58.6 68.3 54.1 39.5 60.3 71.2 57.5 57.1 50.9 54.1 62.0 52.8 57.5 Initial Sound Recognition 2.8 3.9 4.9 3.9 3.8 5.0 5.5 4.8 4.2 3.6 3.9 4.8 4.8 4.8 Letter Sound Recognition 15.3 22.6 24.5 20.7 18.6 24.6 26.2 23.2 21.6 19.8 20.7 25.3 21.1 23.2 Familiar Word Reading 9.0 18.7 26.3 17.8 9.5 20.2 29.9 20.2 20.2 15.3 17.8 24.0 16.3 20.2 Invented Word Reading 8.1 16.8 22.8 15.7 9.0 18.5 24.7 17.6 17.8 13.6 15.7 20.4 14.7 17.6 Words in Oral Reading Passage 11.6 27.1 40.9 26.1 12.6 30.5 43.2 29.2 30.6 21.5 26.1 34.5 23.8 29.2 Note: bold numbers denote significant differences between groups at p < 5%; bold and italicized numbers denote significant differences between groups at p < 1%. 50 Table 5.9 Student Factor 9 - Student is overage (as a proxy for repetition) By Class By Gender Group with Factor Group without Factor Group with Factor Group without Factor Subtest Class 1 Class 2 Class 3 Total Class 1 Class 2 Class 3 Total Girls Boys Total Girls Boys Total Letter Name Knowledge 47.0 49.9 64.1 53.3 36.3 59.5 69.7 55.2 51.9 54.2 53.3 59.0 51.1 55.2 Initial Sound Recognition 3.6 3.8 4.3 3.9 3.1 4.3 5.2 4.2 3.8 3.9 3.9 4.4 4.0 4.2 Letter Sound Recognition 28.6 27.8 20.4 25.7 15.4 23.0 25.4 21.3 31.5 22.2 25.7 22.5 20.0 21.3 Familiar Word Reading 14.2 13.9 19.7 15.9 8.6 19.4 28.2 18.7 18.2 14.4 15.9 21.6 15.6 18.7 Invented Word Reading 12.1 11.8 18.1 13.9 8.0 17.6 23.8 16.5 16.7 12.2 13.9 18.8 14.0 16.5 Words in Oral Reading Passage 18.5 18.1 31.2 22.4 11.3 28.7 42.4 27.4 25.6 20.5 22.4 32.2 22.3 27.4 Note: bold numbers denote significant differences between groups at p < 5%; bold and italicized numbers denote significant differences between groups at p < 1%. Table 5.10 Student Factor 10 - Student was absent for more than one week in the school year By Class By Gender Group with Factor Group without Factor Group with Factor Group without Factor Subtest Class 1 Class 2 Class 3 Total Class 1 Class 2 Class 3 Total Girls Boys Total Girls Boys Total Letter Name Knowledge 37.2 58.0 67.6 55.2 37.7 59.9 70.8 56.9 57.6 52.8 55.2 60.6 52.9 56.9 Initial Sound Recognition 3.2 4.1 5.0 4.1 3.1 4.5 5.3 4.3 4.2 4.1 4.1 4.6 4.1 4.3 Letter Sound Recognition 17.3 20.2 22.1 20.0 17.0 25.2 26.8 23.2 21.1 18.8 20.0 24.5 21.9 23.2 Familiar Word Reading 9.1 18.6 26.3 18.4 9.2 19.7 28.6 19.6 20.4 16.4 18.4 22.9 16.0 19.6 Invented Word Reading 7.6 16.9 22.4 16.1 8.8 17.8 24.3 17.3 17.9 14.2 16.1 19.8 14.6 17.3 Words in Oral Reading Passage 12.0 27.7 38.4 26.8 12.3 28.6 43.8 28.9 30.4 23.0 26.8 34.1 23.3 28.9 Note: bold numbers denote significant differences between groups at p < 5%; bold and italicized numbers denote significant differences between groups at p < 1%. 49 51 In summary, there is some evidence that students with literate parents, literate siblings, books at home, and help with homework perform better on tests of literacy, while students who attend kindergarten, are overage, and absent for more than one week in the school year perform worse on tests of literacy22. Family literacy seems to affect boys more than girls, and family literacy seems to strongly influence student performance in sound recognition in a word. Having at least a literate sibling is associated with better performance for initial sound recognition at each class level, and it seems to exert a stronger positive effect in the development of basic reading skills of boys than of girls. Having books at home is associated with better performance for girls, boys, and class 3 students, especially in the oral reading passage. Receiving help with homework at home appears to improve performance in boys’ recognition of letters and familiar word, and reading of connected words in a passage, but this may not improve literacy performance in girls. Students without kindergarten outperformed students who attended kindergarten on some sub-tests, indicating possible issues with the quality of kindergarten. The lower performance of overage students in class 2 and 3, but not in class 1, may be the effect of grade retention in later classes. Finally, there is some evidence that being absent for more than one week in a school year is associated with poorer literacy outcomes for boys and students in class 2 and class 3. 22 This finding results counterintuivie. Because TEGRA did not look into differences in teacher practice in reading among kindergarten teachers, this finding is inconclusive and requires further inviestigation. 50 6 Chapter 6: Analysis of Teacher Factors Associated with Better Chapter 6: Analysis Reading of Teacher Scores Factors Associated with Better Reading Scores We sought We sought toto explore explore the association the association between between teacherteacher characteristics characteristics –e.g., –e.g., years years of experience, of experience, professional professional qualification, qualification, in-service in-service training, training, etc. - and etc. - and student reading student outcomes reading using outcomes data collected in using data the teacher collected As teacher in the questionnaire. shown in questionnaire. As the table below, shown most in the teachers in table below, the sample mosthigh exhibit teachers in the teaching sample experience, hold teaching certificates, and are actively involved with Parent-Teacher Associations (PTAs) and with exhibit high teaching experience, hold teaching certificates, and are actively involved hold Parent-Teacher Associations (PTAs) and hold meetings with their students’ parents. Although meetings with their students’ parents. Although there is little consensus among sample teachers on how there is little consensus among sample teachers on how best to engage parental support, most best to engage parental support, most teachers agreed on the importance on getting family members to help teachers agreed on the importance on getting family members to help students with their students with their homework, to read with the students at home, and for parents to be more consistent in homework, to read with the students at home, and for parents to be more consistent in their their attendance to PTA meetings. Table 26 below summarizes the profile of teachers in the TEGRA sample. attendance to PTA meetings. Table 26 below summarizes the profile of teachers in the TEGRA sample. Table 26 Table ––Profile 26 Profile of Teachers of Teachers in in TEGRA TEGRA Mean SD N Years of experience (in total) 15.70 10.78 168 Years of experience (as certified teacher) 13.53 9.66 158 Percentage of teachers that hold a Trained Teacher Certification (0=no; 1=yes) 97% 0.18 170 Year in which s/he acquired this certification 1995 9.71 148 Difference between total and certified years of experience 2.59 5.55 150 Percentage that received in-service training in last 3 years (0=no; 1=yes) 50% 0.50 171 Percentage that received in-service reading training in last 3 years (0=no; 1=yes) 27% 0.44 171 In the last 3 years, percentage that received 1 in-service training in reading instruction (0=no; 1=yes) 23% 0.42 171 In the last 3 years, percentage received 2 in-service trainings in reading instruction (0=no; 1=yes) 8% 0.27 171 Average total number of hours spent in training in the last 3 years 92 144 29 Percentage of teachers that work in a school with a functioning PTA (0=no; 1=yes) 96% 0.21 169 Percentage of teachers that meet with the parents of his/her students (0=no; 1=yes) 95% 0.22 171 Percentage of teachers that have the recommended Tongan Reading Text (0=no; 1=yes) 66% 0.48 171 If teacher has the recommended reading text, percentage who use as part of their lessons (0=no; 1=yes) 92% 0.27 135 Percentage of teachers that have the teacher guide for reading 55% 0.50 170 instruction (0=no; 1=yes) 52 51 In order In to identify order to theteacher teachercharacteristics identify the characteristics associated associated with with better better student student reading reading outcomes, outcomes, we we conducted separate regression analyses for each factor (see table below). The dependent conducted separate regression analyses for each factor (see table below). The dependent variable was the variable was the average number of correct words read per minute in the oral reading passage – average number of correct words read per minute in the oral reading passage –i.e., scores from sub-test 6a. i.e., scores from sub-test 6a. The factors used in the analysis included 1) teachers’ experience The factors used in the analysis included 1) teachers’ experience level; 2) whether or not they were certified; level; 2) whether or not they were certified; 3) if they’d receive any in-service training in the last 3) if they’d receive any in-service training in the last 3 years; 4) if any of this training was related to reading 3 years; 4) if any of this training was related to reading instruction; 5) whether the teacher had instruction; 5) whether the teacher had the recommended Tongan reading text; 6) whether they used the the recommended Tongan reading text; 6) whether they used the recommended text; and 7) recommended whether they text; the 7) had and whether guide they had instructions. for reading the guide for reading We theninstructions. We then include class andinclude genderclass and in the gender in the regression regression analysis. analysis. Effect of Teacher Characteristics on Student Performance Effect Table 27of Teacher provides Characteristics results on analyses of seven regression Studentto Performance show the relationship of seven teacher characteristics on student Table performance, 27 provides using results the oral of seven reading fluency regression indicator analyses measured to show as the number the relationship ofof correct seven words teacher read per minute on characteristics student6a. in sub-test performance, Results on theusing the oral contribution ofreading fluency indicator teacher experience measured and training as on student the number are performance of inconclusive. correct words Tworead per factors minute showed in sub-test statistical 6a. results significant Results on the contribution (regressions of 5 and 6, marked with ** inexperience teacher whose on and training table 27). Students student teachers haveperformance the recommended are inconclusive. Tongan reading Twotextfactors read onshowed average statistical 4 more words significant per minuteresults (regressions than students whose5 and 6, marked teachers who do with ** in not have thetable text. 27). Studentsstudents whose Furthermore, teachers userecommended have the whose teachers the recommended Tongan Tonganreading text read readingtext on average read on average 74 more more words words perper minute minute than than students whose teachers who students whose teachers do not use the text.do not have the text. Furthermore, students whose teachers use the recommended Tongan reading text read on average 7 more words per minute than students whose teachers do not use the text. Table 27 - Regression Table analyses 27 - Regression teacher ofof analyses characteristics teacher characteristics on student on student literacy literacy performance performance Dependent variable = number of correct words read per minute in a passage Independent Variables Coefficient Reg 1 - Years of Experience -0.112 Reg 2 - Holds Teaching Certificate -0.050 Reg 3 - Received in-service training in last 3 years -0.221 Reg 4 - Received in-service reading training in last 3 years -0.580 Reg 5 - Teacher has recommended Tongan Reading Text 4.454 ** Reg 6 - Teacher uses Tongan Reading Text 6.581 ** Reg 7 - Teacher has teacher guide for reading instructions -0.460 ** p < .01 Effect of Teacher Expectations on Student Performance Effect of Teachers Teacher were Expectations also asked to report their on Student of expectations Performance student performance. Table 28 provides results of seven regression analyses to show the relationship of seven indicators of teacher Teachers were also asked to report their expectations of student performance. Table 28 provides results of expectations on student performance, measured as the number of correct words students read seven regression analyses to show the relationship of seven indicators of teacher expectations on student performance, measured as the number of correct words students read per minute from the assessment. Each regression analysis consists of two models: 1) a model of the independent variable only, and 2) a model 53 of the independent variable while controlling for gender. We determined the reference category as the median 52 expectation of all teachers for each indicator. For example, we identified class 2 as the reference category in regression 3 because the median response of teachers for students to understand stories they read was by the end of class 2. The positive coefficient of 2.70 in class 1 indicates that students’ whose teachers have higher expectation for them to understand stories a year earlier than the median expected time read on average 2.7 more words per minute relative to students whose teachers with average expectations for their students. The negative coefficient of -2.5 in Class 3 indicates that students whose teachers have lower expectation for them to understand stories a year later than the median expected time read on average 2.5 fewer words per minute relative to students whose teachers with average expectations for their students. It is important to note inconsistent patterns such as the negative coefficient associated with kindergarten in regression 3. Although it is not certain, this inconsistency could suggest poor quality kindergarten education in the country. Overall results of the regression analysis show that students performed worse when teachers had lower expectations, thought the task was not important, or expected students to be able to perform the tasks later. In addition, controlling for gender changes the magnitude of the coefficient, but the sign and significance levels remained the same in nearly all cases. However, the analyses did not provide significant results that teachers with high expectations are associated better student performance. Regression 1 shows that students whose teachers think students should read aloud a short passage with few mistakes by kindergarten read on average 26 fewer words per minute relative to students whose teachers think it should be done by class 3. Regression 2 shows that students whose teachers think students should write their name later than class 3 read on average 15 fewer words per minute relative to students whose teachers think it should be done by class 1. Regression 3 shows that students whose teachers think they should understand stories they read by kindergarten read on average 27 fewer words per minute relative to students whose teachers think it should be done by class 2. Regression 4 shows that students whose teachers think students should recognize and say letter names by class 2 read on average 8 fewer words per minute relative to students whose teachers think it should be done by class 1. Regression 5 does not contain statistically significant differences between when teachers think students should sound out invented words. Regression 6 shows that students whose teachers think students should understand stories they hear by class 3 read on average 4 fewer words per minute relative to students whose teachers think it should be done by class 1. Regression 7 shows that students whose teachers think students should recite the alphabet in class 3 read on average 5 fewer words per minute, while students whose teachers think reciting the alphabet is not important read an average 18 fewer words per minute relative to students whose teachers think it should be done by class 1. Teachers also reported similar expectations by class. Results indicated that on average, teachers expected students to be able to do the following by the end of class 1: write their name, recognize and say letter names, and recite the alphabet. Teachers expected students to be able to do the following by the end of class 2: read aloud with few mistakes, understand stories they read, sound out invented words, and understand stories they hear. 53 Table 28 - Regression analyses of average effects of teachers’ expectations on student literacy performance Dependent variable = number of correct words read per minute in a passage Controlling for Gender (F=1) Independent Variables Coefficient SD Coefficient SD Reg 1 - Grade students should read aloud a short passage with few mistakes Kindergarten -25.950 *** (1.284) -30.410 *** (1.582) Class 1 3.578 (1.939) 3.517 (1.892) Class 2 0.524 (1.823) 0.743 (1.784) Class 3 — — — — Not important -3.948 ** (1.284) 0.831 (1.343) Reg 2 - Grade students should be able to write their name Kindergarten -0.572 (1.988) -1.064 (1.949) Class 1 — — — — Class 2 -10.300 (9.483) -14.980 (9.536) Class 3 -14.550 *** (2.945) -14.070 *** (2.834) Reg 3 - Grade students should understand stories they read Kindergarten -27.040 *** (1.141) -31.740 *** (1.489) Class 1 2.700 (1.975) 2.428 (1.910) Class 2 — — — — Class 3 -2.537 (1.714) -2.635 (1.682) Reg 4 - Grade students should recognize and say letter names Kindergarten 1.587 (2.103) 1.441 (2.063) Class 1 — — — — Class 2 -7.530 ** (2.667) -7.698** (2.628) Reg 5 - Grade students should sound out unfamiliar words Kindergarten 0.931 (5.973) 0.557 (5.917) Class 1 1.044 (1.769) 0.831 (1.719) Class 2 — — — — Class 3 -2.666 (1.954) -2.850 (1.915) Reg 6 - Grade students should understand stories they hear Kindergarten 2.091 (4.800) 0.927 (4.546) Class 1 — — — — Class 2 0.660 (1.944) 0.588 (1.900) Class 3 -4.187 * (1.886) -4.119 * (1.856) 54 56 Dependent variable = number of correct words read per minute in a passage Controlling for Gender (F=1) Independent Variables Coefficient SD Coefficient SD Reg 7 - Grade students should recite alphabet Kindergarten 0.581 (2.411) 0.399 (2.285) Class 1 — — — — Class 2 -3.221 (2.787) -2.622 (2.688) Class 3 -4.979 * (2.076) -5.414 ** (2.091) Not important -17.530 *** (0.925) -22.190 *** (1.322) * p < 0.05, ** p < 0.01, *** p < 0.001 Effect of Teacher Instructional and Assessment Methods Effect of Teacher Instructional and Assessment Methods The final set of teacher analyses observed the relationship between instructional and assessment methods The final set of teacher analyses observed the relationship between instructional and assessment and student performance. Tables 29 and 30 provide results of 12 regression analyses of the frequency in methods and student performance. Tables 29 and 30 provide results of 12 regression analyses of which students and teachers performed twelve instructional and assessment methods within the course of the frequency in which students and teachers performed twelve instructional and assessment the week. The dependent variable was measured as the number of correct words students read per minute methods within the course of the week. The dependent variable was measured as the number of from the assessment. The reference category for all models is zero frequency (i.e., the task never occurs). correct words students read per minute from the assessment. The reference category for all models each Although regression is zero analysis frequency included (i.e., separate the task models never of theAlthough occurs). independent variable each alone and regression of the analysis independent while controlling variable models included separate for gender, the of the independent results alone variable indicate changes and of theinindependent the magnitude of the variable coefficient, but the sign while controlling for and significance gender, levels the results remained indicate the same changes inin thenearly all cases. magnitude of the coefficient, but the sign and significance levels remained the same in nearly all cases. In terms of the methods used during reading instruction, most teachers ask students to repeat sentences, In terms retell ofthey a story methods the read, out during used sound inventedreading instruction, words, learn meaningsmost teachers of new words,ask students aloud, orto read repeat perform readingretell sentences, silent a story between they 1-2 days read,to a week sound out invented daily. Most words, teachers do learn not require meanings students of text to copy newfrom words, the read aloud, board. or also Teachers perform silent provide reading written between and oral 1-2 days evaluations, a portfolios review week to anddaily. Most teachers projects, doand not copy books require students homework copy to in (see table 27 1). from the board. Teachers also provide written and oral text Annex evaluations, review portfolios and projects, copy books and homework (see table 27 in Annex Results 1). of the regression analyses in Table 28 provide evidence that several common instructional methods that teachers ask students are associated with higher levels of student literacy, while other methods are Results of associated the with regression lower analyses levels of student in Regression literacy. Table 28 3provide evidence shows that studentsthat several who retell common a story either instructional 2-4 methods days per week read teachers or daily that ask on average students 5 and 4 more are associated words with per minute higher relative respectively ofstudents levels to student literacy, while other methods are associated with lower levels of student literacy. Regression 3 who never retell a story they read. However, regression 2 shows that students who copy text from the shows that students who retell a story either 2-4 days per week or daily read on average 5 and 4 chalkboard either 1-2 days or 2-4 days per week read on average 4 and 10 fewer words per minute more words per minute respectively relative to students who never retell a story they read. respectively relative to students who never copy text from the chalkboard. Regression 5 also shows that However, regression 2 shows that students who copy text from the chalkboard either 1-2 days students who learn the meaning of new words either 1-2 days or 2-4 days per week read on average 7 and or 2-4 days per week read on average 4 and 10 fewer words per minute respectively relative to 10 fewer words students per minute who never copy relative to students text from whose teachers the chalkboard. never assign Regression 5 also the showstask. Finally, that regression students who 7 shows that students who perform silent reading 1-2 days per week read on average learn the meaning of new words either 1-2 days or 2-4 days per week read on average 7 and 10 22 fewer words per minute fewer relative words per students tominute who never relative perform silent to students whosereading. Regressions teachers 1, 4, and never assign the6 task. did not show Finally, regression statistically 7 shows differences significant that students who perform in student silent performance reading for 1-2 students days who per week repeated read onsounded sentences, average fewer words 22 invented out words, per andminute relative to students who never perform silent reading. Regressions read aloud. The regression analyses in Table 30 also identified several common assessment methods that are associated 57 with lower levels of student performance. Regression 8 shows that students of teachers who perform written 55 evaluations 1-2 days per week read on average 7 fewer words per minute relative to students whose teachers never perform written evaluations. Regression 11 shows that students of teachers who review copy books 2-4 days per week read on average 9 fewer words per minute relative to students of teachers who never review copy books. Finally, regression 12 shows that students of teachers who review homework read on average 10 fewer words per minute relative to students whose teachers never review homework. Regressions 9 and 10 did not show statistically significant differences in student performance of teachers who performed oral evaluation or reviewed portfolios. Table 29 - Regression analyses of average effects of frequency of instructional methods on student literacy performance Dependent variable = number of correct words read per minute in a passage Controlling for Gender (F=1) Independent Variables Coefficient SD Coefficient SD Reg 1 - The whole class repeated sentences the teacher said first 1-2 days 6.364 (3.302) 6.374 * (3.217) 2-4 days 2.594 (1.846) 2.460 (1.802) Daily -0.976 (2.191) -0.464 (2.116) Reg 2 - Students copied down text from the chalkboard 1-2 days -4.030 * (1.602) -3.910 * (1.555) 2-4 days -10.070 *** (2.815) -10.190 *** (2.622) Daily -2.858 (3.916) -3.776 (3.899) Reg 3 - Students retold a story that they had read 1-2 days -3.083 (3.528) -2.626 (3.406) 2-4 days 4.713 * (1.879) 4.830 ** (1.824) Daily 3.821 * (1.943) 3.451 (1.897) Reg 4 - Students sounded out unfamiliar words 1-2 days -0.041 (2.771) -0.446 (2.773) 2-4 days 1.555 (1.853) 1.598 (1.813) Daily 1.475 (1.909) 1.392 (1.856) Reg 5 - Students learned meanings of new words 1-2 days -6.984 *** (1.174) -1.939 (1.231) 2-4 days -9.925 *** (2.008) -9.620 *** (1.962) Daily -1.989 (1.632) -1.423 (1.588) Reg 6 - Students read aloud to teacher or to other students 1-2 days 1.590 (4.057) 1.549 (3.793) 2-4 days 0.619 (2.606) 0.607 (2.561) Daily -2.075 (1.689) -2.009 (1.635) Reg 7 - Students were assigned reading to do on their own during school time 1-2 days -22.150 *** (3.998) -23.990 *** (3.489) 2-4 days -2.688 (1.694) -2.966 (1.643) Daily 1.232 (1.906) 0.547 (1.849) * p < 0.05, ** p < 0.01, *** p < 0.001 Table 30 - Regression analyses of average effects of frequency of assessment methods on student literacy performance 56 Table 30 - Regression analyses of average effects of frequency of assessment methods on student literacy performance Dependent variable = number of correct words read per minute in a passage Controlling for Gender (F=1) Independent Variables Coefficient SD Coefficient SD Reg 8 - Teacher performed written evaluations 1-2 days -7.071 * (3.324) -6.987 * (3.241) 2-4 days -2.435 (3.329) -2.816 (3.380) Daily 2.501 (1.579) 2.439 (1.535) Reg 9 - Teacher performed oral evaluations 1-2 days -3.674 (2.675) -4.034 (2.598) 2-4 days -1.568 (2.183) -2.023 (2.178) Daily -0.753 (1.745) -0.816 (1.687) Reg 10 - Teacher reviewed portfolios and other projects 1-2 days 0.787 (1.635) 0.660 (1.603) 2-4 days 2.818 (2.369) 2.935 (2.242) Reg 11 - Teacher reviewed copy books 1-2 days 0.739 (3.099) 0.748 (3.075) 2-4 days -9.483 * (3.987) -9.030 * (4.159) Reg 12 - Teacher reviewed homework 1-2 days -2.640 (3.608) -2.274 (3.673) 2-4 days -9.630 *** (1.932) -10.200 *** (2.001) * p < 0.05, ** p < 0.01, *** p < 0.001 In conclusion, results of the analysis of teacher factors on student reading outcomes suggest that few In conclusion, teacher resultsbut characteristics the of a analysis number of of types teacher factors teacher on student expectations, reading instructional outcomes methods, suggest and that assessment few teacher methods characteristics in Tonga but have an effect a number on student types reading of teachermeasured performance, expectations, as the instructional average numbermethods, of words andminute per assessment methods that students in Tonga read have correctly. an effectteacher Surprisingly, on student reading education andperformance, measured training was not as associated the average with number of words student performance, use minute but the per that students of the Tongan read appears Reading Texts correctly. Surprisingly, better teacher to promote reading education The outcomes. andresults did was training not associated not identify with where possible areas student performance, high but the can useimprove teacher expectations of the Tongan student Reading Texts appears to promote better reading outcomes. The results did not identify performance, but low teacher expectations can hinder student performance in several areas, namely the possible areas where high teacher expectations can improve student performance, but low level in which teachers expect their students should be able to read aloud, write their names, understand teacher expectations can hinder student performance in several areas, namely the level in which stories they read and hear, recognize and say letters, and recite the alphabet. Although it is beyond the teachers expect their students should be able to read aloud, write their names, understand scope of this study, the negative relationship between kindergarten and student performance presents stories they read and hear, recognize and say letters, and recite the alphabet. Although it is suggests a need to improve the quality of kindergarten in Tonga. The study also found that while instructional beyond the scope of this study, the negative relationship between kindergarten and student methods such as performance asking students presents suggests retell to a need a story can improve to improve thestudent literacy, quality several tasksin of kindergarten can also hinder Tonga. The student performance, including asking students to copy from the chalkboard and silent reading. study also found that while instructional methods such as asking students to retell a story can improve student literacy, several tasks can also hinder student performance, including asking students to copy from the chalkboard and silent reading. 57 60 Learning the meaning of new words was also found to be negatively associated with student performance. This may not suggest that students should stop learning new words altogether, but a possible problem in how new words are taught, such as through rote memorization. The study did not find assessment methods that could improve student performance, but it did identify several methods that may hinder literacy, namely if teachers performed written evaluations or reviewed copy books. Reviewing homework was also negatively associated with student performance. Similar to learning new words, this may reflect a problem in the process of how teachers review homework, such as a poor review of homework or a large number of students failing to complete their homework, and hence, not engaged in reviewing homework. Finally, gender does not appear to strongly influence teacher’s expectations, instructional methods, and assessment methods. 58 7 Chapter 7 - Next Steps TEGRA results call for immediate attention to the way reading development is taking place in the country. As a diagnosis study, its main purpose was to (a) generate data on the extent reading deficits in the first cycle of primary education, (b) identify specific skills that could be hampering reading comprehension among Tongan students, and (c) identify student and teacher factors that contribute to better reading outcomes in Tonga. TEGRA results indicated that while most students are able to develop some of the basic skills needed to become effective readers, most students are not able to distinguish the sounds in words or understand the relationship between sounds and letters, whether in isolation or in the context of a word. As a result, few students are able to read fluently by the end of Class 3 to focus on the meaning of what they are reading. This in turn, leads to average low reading comprehension levels, already identified in previous STAT Class 4 assessments. Based on the data presented, specific recommendations to be considered are presented to improve the quality of reading instruction in Tongan schools: • Improve instruction in Tongan phonics and increase phonemic awareness levels among students. TEGRA data suggested students were either not receiving any instruction on the alphabetic principle or this training was not sufficient or appropriate to develop an understanding of letter-sound correspondences. Without the ability to distinguish the sounds in words –i.e., phonemic awareness- and a solid understanding of how letters and sounds go together to create words, students are bound to continue to struggle in their reading development paths and have a limited understanding of what they read. However, improvements in reading instruction should not focus on phonics exclusively but should be incorporated into a comprehensive system of reading instruction covering the fundamental skills tested, to support students develop sufficient reading skills as early and comprehensively as possible. • Better reading instruction calls for additional support to teachers. In order to improve reading instruction in Tonga, teachers will require additional support to have more informed reading expectations about reading development in the early grades. TEGRA results identified how incorrect expectations had a negative impact on student performance, especially when teachers believed specific fundamental skills should be developed in later years. Better learning expectations should be coupled with information the specific methods, classroom activities and assessment methods that can allow teachers improve their practice through clear instruction and monitoring indicators. More research will be needed on the specific way silent reading and the learning of new words –the two methods associated with better reading outcomes- take place in the classroom, to identify, on the one hand, the specific elements that make this methods work in the Tongan context place; an and on the other, what aspects of the teacher’s practice –i.e., planning, instruction, teaching materials, specific methods, etc. - are falling short to develop basic reading skills among children. 59 • Teacher support should be constant. TEGRA survey results showed how only 27% of the teachers had received any in-service training on reading instruction in the last three years. If less than 3 out of 10 teachers in the country benefit from learning about specific ways in which they can improve their practice, Tongan teachers will continue to practice their profession in isolation. Moreover, teachers need to become aware of the fundamentals of reading development in both pre- and in-service training. The current TIOE syllabus for primary teachers has no mention of grade-level methods or expected reading outcomes in the early grades. Specific reading instruction training should be a centerpiece of TESP II. • Increase the availability of books and other reading materials to children to read at home. Survey results showed how students who have reading books at home have better reading outcomes and are more likely to become fluent. Having access to books at home and the possibility to read and develop a sense of enjoyment will be critical to ensure students engage in reading on a regular basis. While English language books tend to be available, most reading materials in Tongan are limited and complex for beginning readers –e.g., newspapers and the bible. Production of reading books and stories in Tongan written by Tongans would be important not only to allow students to read child-appropriate stories but to preserve the rich oral tradition of Tongan story tellers. • Introduce policy actions that increase student exposure to literacy outside the school. Making more books available to students will not per se ensure better reading outcomes. Along with access to more and relevant reading materials in Tongan, children will need support to develop a reading habit beyond the requirements of the school curriculum. One way of improving this would be to develop community literacy programs where schools become focal point of literacy in the community. Building on the success created by the school-grants program, schools can encourage parents and literate siblings –a predictor of better reading scores in Tonga- to become more involved in the reading development of young children by participating in “peer readingsâ€? and activities reading clubs both in school and at home. • Establish reading benchmarks to monitor improvements across classes. As MEWAC moves on to establish an oral reading fluency benchmark for the early grades against which reading improvements can be monitored, it is important to consider that initial standards should be established at relatively low levels since not enough is currently known about when and at what rate Tongan children should progress when learning to read in their own language. One way to establish a benchmark would be to use it as a marker to track reduction in the percentage of student scoring zero across sub-tests –as opposed to establish fluency targets. Monitoring achievements of standards over time will eventually provide more information on the rate and the way in which average fluency develops in the Tongan language. • Support teachers to establish reading goals for individual students which they can monitor throughout the school year. In order for teachers and schools to be able to be held accountable for reading outcomes, teachers, school officials and parents need to understand what these standards mean and how each can support reading development in their own school in order 60 to reach national standards. School development plans should contain reading improvement goals as part of their minimum service standards, as well as a description of activities aimed at encouraging reading. Parents and the community as a whole should be brought into this effort. • More research is needed on how instruction methods and activities are carried out in the classroom. Many of the teacher factors traditionally associated to better reading outcomes such as teacher certification or learning of new words showed contradictory or counterintuitive results in Tonga. Although there are several potential explanations for this phenomenon, this calls for more research on how instruction takes place in the classroom. For example, average reading scores among students whose teachers promote the learning of new words are lower than those whose teachers rely less in this activity. A question that could be look further into would have to analyze how learning of new words takes place –e.g., through explicit decoding and association with similar vocabulary or by memorization- in order to recommend specific improvements to current instructional methods. • Last but not least, it is clear that more research is needed to better understand the factors that contribute to differences in reading performance between boys and girls. An analysis of the factors that contribute to these differences is beyond the scope of this survey. However, it is cause of great concern that boys are already in academic disadvantage after completing just one year of education. A better understanding of this phenomenon is critical to inform sector policies and increase the success of future reading development programs. 61 Bibliographical References Abadzi, H. 2006. Efficient learning for the poor: Insights from the Frontier of Cognitive Neuroscience. Washington, DC: The World Bank. Chabbott, C. (2006). Accelerating early grades reading in high priority EFA Countries: A desk review. From http://www.equip123.net/docs/E1-EGRinEFACountriesDeskStudy.pdf (Accessed on October 29, 2010) Chiappe, P., L. Siegel, and L. Wade-Woolley. 2002. Linguistic diversity and the development of reading skills: A longitudinal study. Scientific Studies of Reading 6(4): 369–400. Ehri, L. C., Nunes, S. R., Stahl, S. A., & Willows, D. M. (2001). Systematic phonics instruction helps students learn to read: Evidence from the National Reading Panel’s meta- analysis. Review of Educational Research, 71(3), 393-447. Fuchs, L., D. Fuchs, M.K. Hosp, and J. Jenkins. 2001. Oral Reading Fluency as an Indicator of Reading Competence: A Theoretical, Empirical, and Historical Analysis. Scientific Studies of Reading 5(3), 239–256. Hasbrouck, J., & Tindal, G. A. (2006). Oral reading fluency norms: A valuable assessment tool for reading teachers. International Reading Association, 636–644. Jiménez, J. E. and I. O’Shanahan Juan. 2008. Enseñanza de la lectura: de la teoría y la investigación a la práctica educativa. Revista Iberoamericana de Educació, 45(5): 1–22. Available at http://www.rieoei. org/2362.htm (accessed October 28, 2010). Linan-Thompson, S., and S. Vaughn. 2007. Research based methods of reading instruction for English language learners: Grades K-4. Alexandria, VA:Association for Supervision and Curriculum Development. Ministry of Education of Tonga. 1994. Language Education. English and Tongan Syllabus Classes 1-6 Form 1-2. (mimeo). MEWAC. 2004. Tonga Education Policy Framework 2004-2019 (mimeo). MEWAC. 2008. Tonga Standardize Test of Achievement: Results Report (mimeo). National Reading Panel. 2000. Teaching Children to Read: An Evidence-Based Assessment of the Scientific Research Literature on Reading and Its Implications for Reading Instruction. Washington, DC: National Institute of Child Health and Human Development. Available at http://www.nichd.nih.gov/publications/nrp/ report.cfm (accessed July 13, 2010). RTI. 2009. Early Grade Reading Assessment Toolkit. Prepared by RTI for The World Bank, Office of Human Development. Available at https://www.eddataglobal.org/documents/index.cfm?fuseaction=pubDetail& ID=149 (accessed September 03, 2010). RTI. 2010. Guyana Early Grade Reading Assessment: October 2008 Results. Prepared by RTI for The World Bank, Office of Human Development. Available at https://www.eddataglobal.org/ (accessed August 02, 2010). 62 Sprenger-Charolles, L. 2004. Linguistic Processes in Reading and Spelling: The Case of Alphabetic Writing Systems: English, French, German and Spanish. Pp. 43–66 in Handbook of Children’s Literacy. Edited by T. Nunes and P. Bryant. Dordrecht, the Netherlands: Kluwer Academic Publishers Share, D. L., Jorm, A., Maclearn, R., & Matthews, R. 1984. Sources of individual differences in reading acquisition. Journal of Education Psychology, 76, 1309-1324. Snow, C. E., Burns, M. S., & Griffin, P. (Eds.). 1998. Preventing reading difficulties in young children. Washington, DC: Committee on Preventing of Reading Difficulties in Young Children and National Academy Press. Stanovich, K. E. (2000). Progress in understanding reading: Scientific foundations and new frontiers. New York: Guilford Press. Stores, Christine. 2008. “Language, Literacy and Bilingualism. A Report Presented to MEWAC to support development of the National Language Policyâ€? Tonga Education Support Program – Curriculum Reform Project. (Mimeo). Tonga Institute of Education (TIoE). 2004. Diploma of Education (Primary). Curriculum Document. (Mimeo) 63 64 ANNEX 1/ TABLES ANNEX 1/ TABLES Table 31 -TEGRA Reliability Matrix Table 31 -TEGRA Reliability Matrix Phonemic Sight Words Invented Connected Letters Correct Sounds Correct Reading Listening Awareness – Correct Per Words Correct Words Correct Writing Items Per Minute Per Minute Comprehension Comprehension Initial Sound Minute Per Minute Per Minute (W) (LCPM) (SCPM) (RC) (LC) (PAIS) (SWCPM) (IWCPM) (CWCPM) LCPM 1 .409** .518** .699** .707** .673** .594** .408** .679** PAIS .409** 1 .489** .388** .440** .360** .334** .319** .409** SCPM .518** .489** 1 .458** .531** .435** .388** .274** .423** SWCPM .699** .388** .458** 1 .897** .898** .807** .394** .742** IWCPM .707** .440** .531** .897** 1 .879** .794** .387** .716** CWCPM .673** .360** .435** .898** .879** 1 .860** .380** .728** RC .594** .334** .388** .807** .794** .860** 1 .387** .656** LCPM .408** .319** .274** .394** .387** .380** .387** 1 .456** W .679** .409** .423** .742** .716** .728** .656** .456** 1 Statistics Average 53.83 0.39 20.33 17.77 15.66 26.27 0.23 0.55 0.48 Std Dev 25.36 0.33 16.72 14.57 12.54 23.51 0.31 0.29 0.29 Min 0 0 0 0 0 0 0 0 0 Max 130.43 1 89 103.45 66.67 147.5 1 1 1 68 Table 32 - ANOVA Results: Differences in Means across Sub-tests (ST) Table 32 - ANOVA Results: Differences in Means across Sub-tests (ST) Sum of Squares df Mean Square F Sig ST1 - Number of letters read correctly by cl a s s Between Groups 208623.945 2 104311.972 221.9 0.000 Wi thi n Groups 564103.442 1200 470.086 Tota l 772727.387 1202 642.868 by gender Between Groups 16505.7 1 16505.7 26.21 0.000 Wi thi n Groups 756221.686 1201 629.66 Tota l 772727.387 1202 642.868 ST2 - Number of sounds correctly identified in a word by cl a s s Between Groups 9.22 2 4.61 45.58 0.000 Wi thi n Groups 121.355 1200 0.101 Tota l 130.575 1202 0.109 by gender Between Groups 0.374 1 0.374 3.45 0.064 Wi thi n Groups 130.201 1201 0.108 Tota l 130.575 1202 0.109 ST3 - Number of sounds read correctly by cl a s s Between Groups 21615.451 2 10807.726 41.27 0.000 Wi thi n Groups 314268.224 1200 261.89 Tota l 335883.675 1202 279.437 by gender Between Groups 1480.027 1 1480.027 5.32 0.021 Wi thi n Groups 334403.648 1201 278.438 Tota l 335883.675 1202 279.437 ST4 - Number of familiar words read correctly by cl a s s Between Groups 68550.682 2 34275.341 219.89 0.000 Wi thi n Groups 187048.151 1200 155.873 Tota l 255598.833 1202 212.645 by gender Between Groups 9387.388 1 9387.388 45.79 0.000 Wi thi n Groups 246211.445 1201 205.005 Tota l 255598.833 1202 212.645 ST 5 - Number of unfamiliar words read correctly by cl a s s Between Groups 46224.003 2 23112.001 194.37 0.000 Wi thi n Groups 142688.979 1200 118.907 Tota l 188912.982 1202 157.166 by gender Between Groups 5771.712 1 5771.712 37.85 0.000 Wi thi n Groups 183141.27 1201 152.491 Tota l 188912.982 1202 157.166 69 65 Table 32 - ANOVA Results: Differences in Means across Sub-tests (ST) contd Sum of Squares df Mean Square F Sig ST – 6a Number of words read in a narrative passage by cl a s s Between Groups 177289.932 2 88644.966 218.48 0.000 Wi thi n Groups 486871.654 1200 405.726 Tota l 664161.586 1202 552.547 by gender Between Groups 26768.557 1 26768.557 50.44 0.000 Wi thi n Groups 637393.029 1201 530.719 Tota l 664161.586 1202 552.547 ST 6b - Reading comprehension (% of correct responses) by cl a s s Between Groups 26.331 2 13.165 173.77 0.000 Wi thi n Groups 90.916 1200 0.076 Tota l 117.246 1202 0.098 by gender Between Groups 3.019 1 3.019 31.75 0.000 Wi thi n Groups 114.227 1201 0.095 Tota l 117.246 1202 0.098 ST7 -Listening comprehension (% of correct responses) by cl a s s Between Groups 12.882 2 6.441 90.91 0.000 Wi thi n Groups 85.02 1200 0.071 Tota l 97.903 1202 0.081 by gender Between Groups 0.054 1 0.054 0.66 0.418 Wi thi n Groups 97.849 1201 0.081 Tota l 97.903 1202 0.081 ST8 - Writing (weighted score - spelling/writing) by cl a s s Between Groups 33.885 2 16.943 301.39 0.000 Wi thi n Groups 67.458 1200 0.056 Tota l 101.343 1202 0.084 by gender Between Groups 0.951 1 0.951 11.38 0.001 Wi thi n Groups 100.392 1201 0.084 Tota l 101.343 1202 0.084 Note: Pairwise tests by school type and region are not significant 66 70 Table 33 - Frequency Table of methods 33 - Frequency instruction ofof of methods instruction used by used by Tongan Tongan teachers teachers inearly the grades in the early grades Never 1-2days 2-4days Daily Mean The whole class repeated sentences that you said first. 0.086 0.312 0.225 0.377 SD 0.281 0.465 0.419 0.486 Students copied down text from the chalkboard. Mean 0.577 0.304 0.092 0.027 SD 0.495 0.461 0.290 0.161 Students retold a story that they had read. Mean 0.052 0.436 0.341 0.171 SD 0.222 0.497 0.475 0.378 Students sounded out invented words. Mean 0.046 0.266 0.273 0.415 SD 0.210 0.443 0.447 0.494 Students learned meanings of new words. Mean 0.003 0.188 0.380 0.429 SD 0.059 0.392 0.487 0.496 Students read aloud to teacher or to other students. Mean 0.005 0.112 0.331 0.551 SD 0.071 0.317 0.472 0.499 Students were assigned reading to do on their own Mean 0.029 0.230 0.365 0.376 during school time. SD 0.169 0.422 0.483 0.486 Written evaluations Mean 0.012 0.043 0.369 0.576 SD 0.109 0.204 0.484 0.496 Oral evaluations Mean 0.101 0.159 0.301 0.439 SD 0.303 0.366 0.460 0.498 Review of portfolios and other projects Mean 0.000 0.401 0.455 0.144 SD 0.000 0.492 0.499 0.353 Review of copy books Mean 0.000 0.010 0.017 0.973 SD 0.000 0.101 0.128 0.162 Review of homework Mean 0.000 0.022 0.083 0.895 SD 0.000 0.146 0.277 0.307 71 67 ANNEX 2 / READING INSTRUCTION IN TONGA Reading as described in the curriculum The curriculum identifies Reading in Tongan as a communication skill alongside Listening, Speaking, Writing and Spelling. (Silapa ‘o e Lea Faka-Tonga, 1969) The aim of this syllabus is to develop children’s skills in these areas and address the weaknesses that exist in any of these areas. Other additional skills that are related to reading are also included in the detailed description of skills to be taught at primary level. One such skill is sound identification. Students are to be taught the following: • Sounds of the letters of the Tongan alphabet • Sounds made by a range of birds • Sounds made by the cry of a variety of animals • Sounds of bottles and cans • Sounds of babies and mothers. With regards to listening, the curriculum highlights the ‘fragility’ of the aural tools and that communication may be severely affected if children have a problem in this area. Students are to be guided to be able to transmit and interpret sounds correctly. The curriculum highlights the urgent need to teach Tongan children to read well for here lays a window of opportunity to higher learning. The curriculum also states the need to ensure that all Tongan children learn to read well for with this knowledge, they will be able to solve problems and easily understand that which is unclear. The following items list possible reading texts and activities that may be used. • Reading from pictures • Reading word lists • Reading sentences • Reading paragraphs • Reading passages • Guessing Riddles • Poetry • Legends • Myths (of Origin) • Stories eg, Fables etc • Reading books etc (from MEWAC’s Curriculum Development Unit) 68 The Scope and Sequence Chart for Language Education identifies language forms and focus which pertain to both literary and non-literary forms and with a focus on number and nursery rhymes, everyday experiences (eg, going to the market) and songs and hymns of Tongan life Reading as described in the classroom Teachers plan an overview from the syllabus and at the beginning of each term, this overview is confirmed and weekly and daily planning is based on this. This overview covers, for the Tongan Language, activities such as Story Reading, Shared Reading, Guided Reading, Printing and Story Writing. Other development activities are also planned where all the skills listed in the syllabus, which are critical to reading, are required to be taught/ developed. In the Teachers’ Daily Planner Book, a time schedule is given for the teaching of the Tongan Language, beginning with Development Activities and leading into Story Reading – Shared and Guided, followed by Printing and Story Writing. All Tongan Language Instruction and activities are carried out in the morning. In reality, a lot of teachers select only what they want to teach. Comments from Education Officers reveal that teachers may plan an overview according to the syllabus and plan weekly and daily for activities but the actual delivery may not be according to the plan. One example is the use of speech training in these levels, used in the previous syllabus which included Oral English (Tate). ‘Older’ teachers found this very useful and continued to use this speech training. Younger teachers do not use this activity. The mismatch between planning and reality needs to be addressed. Materials used in the syllabus and Teacher’s Guide are sometimes not available so teachers either go without or they write their own stories and create their own teaching resources. Teachers are also assisted by the daily radio broadcast to primary schools, guiding teachers through various activities like listening comprehension, dictation and others. Instructional Methods The instructional methods as instructed by the syllabus state that both English and Tongan should be used to develop the skills identified as vital to one’s development and ability to communicate. A mostly student-centered approach is encouraged and teachers mostly carry out a class that starts with a demonstration, group work on a cooperative learning task and followed by individual work. Teachers are encouraged to identify their students’ individual learning styles and needs and to cater for this in their planning. A variety of teaching strategies are suggested in the syllabus so teachers can give differentiated instruction to their classes. Composite and multi-grade classes also have special needs and are also addressed in their planning. 69 Results from survey on language instruction The Scope and Sequence for Language Education states that “Tongan and English are compulsory subjects throughout all years of primary and secondary school. While Tongan is more important in early primary years, English becomes of equal importance by the time secondary school is reached. This ensures that the policy of bilingualism can achieve its purposes.â€? 70 ANNEX 3 / TONGAN LANGUAGE Although Tongan is the official language of Tonga, its role as such in the current social structure of the country is in question. However, it is still the dominant language, but mainly spoken and hardly written. English is taught in all schools. Tongan belongs to the Oceanic languages, a subgroup of the Malayo-Polynesian branch of the Austronesian language family. All Oceanic languages have numerous linguistic features in common but the closest relatives of Tongan are the other languages of the Polynesian subgroup. In Tonga, the only dialect diversity is with the language of Niuafo’ou, which historically belongs to the Samoic- Outlier languages. It has five vowels (a, e, i, o, u) which makes a phonemic difference between short and long vowels. There are 12 consonant with twelve phonemes (f, h, k, l, m, n, ng [velar nasal], o, p, s, t, v & ‘[glottal stop]). It is a very transparent language in regards to the relationship of its phonemes and its orthographic symbols. The syllable structure is Consonant – Vowel (CV). Tongan only allows open syllables and does not permit consonant clusters. Stress in usually on the penultimate vowel but it is only marked if it is shifted to the last (short) vowel as a definite accent. Policies & Preservation Since there is only less than one hundred and thousand people in this tiny island, in comparison to other dominant world languages, Tongan is near extinct from the sheer smallness of its number of speakers. There are other major powers at play, of course. Our smallness, our economic dependence on English- speaking countries play a big part. In addition, Language Policies in the Ministry of Education have not helped. There has been an insistence in teaching children in both Tongan and English in the primary level and when children go to secondary schools, all subjects (except for Tongan and Religious Studies/Scripture) are to be taught and examined in English. This extremely harmful language policy has persisted for nearly 5 decades now and has resulted in the elevation of English as an “elite languageâ€? while Tongan is regarded as inferior. Nevertheless, the Ministry of Education is very much aware of these issues and has passed a language policy in 2009 which emphasizes the necessity of teaching young Tongan children in Tongan first for it is in acquiring skills from the learning of the mother-tongue, that children can learn the second language better, using these important language learning skills. EGRA has come at a most opportune time. It reinforces our desire as language educators, to be vigilant in keeping our indigenous language robust. This survey will indicate the much-needed future directions for us to follow. [Some of linguistic information above is taken from Svenja Volkel’s PhD thesis, “Social Structure, space and possession in Tongan culture and languageâ€?. An ethnolinguistic study, Mainz, 2007]. 71 ANNEX 4 / DEVELOPMENT OF TEGRA INSTRUMENTS Due to the express desire of the TEGRA team to downplay using this assessment s a “testâ€?, we avoided at almost all times (with a few slips of course!) using the words “testâ€? or “sub-testâ€?. This was due to our understanding of how these words were used in our exam-oriented Tongan context. Instead we used the word “surveyâ€? for “testâ€? and “sectionsâ€? for “sub-testsâ€?. Introduction: The Tongan Alphabet The following letters make up the Tongan Alphabet: a e f h i k l m n ng o p s t u v ‘ There are five vowels: a e i o u There are twelve consonants: f h k l m n ng p s t v ‘ Section 1: Letter Name Knowledge (‘Ilo’i ‘o e ngaahi mata’itohi) A corpus of texts, written in the Tongan language and collected for linguistic analysis (such as for a forming a letter frequency table) did not exist. As it is a language from a predominantly oral culture, it was a challenge to set up this corpus. However, because of time constraints and the difficulty of getting electronic copies of these texts, we managed to collect texts that were representative of the Tongan reading material which almost all Tongans read or are most readily available to most Tongan homes. This corpus included the most popular local newspaper (also available for all Tongan communities in New Zealand, Australia and the United States), the latest Tongan translation of the Four Gospels from the Bible (Mathew, Mark, Luke & John), junior readers for Classes 1 – 3 and the latest Free Wesleyan Church hymn book. The FWC Church is the largest religious denomination in the country and their hymn book is widely used by all sectors. The above-mentioned electronic texts were analyzed according to the instructions in the EGRA toolkit. It can be deemed as quite a small corpus but when letter frequency tests were conducted with each text, there were very little variations from the table below, which was the final letter frequency used. 72 The above-mentioned electronic texts were analyzed according to the instructions in the EGRA toolkit. It can be deemed as quite a small corpus but when letter frequency tests were conducted with each text, there were very little variations from the table below, which was the final letter frequency used. Letters in the Tongan Language: Frequency of Use (%) Letters in the Tongan Language: Frequency of Use (%) A 18.6614 N 5.3543 E 10.8661 NG 2.3622 F 3.7007 O 9.2125 H 5.6692 P 2.1259 I 10.4724 S 1.8110 K 8.0314 T 3.0708 L 2.8346 U 4.8031 M 2.1259 V 0.4724 ‘ 7.559 Letters for Stimulus Material 1 (refer to Stimulus Material 1 - used for this section and also used in Section 3) Letters for Stimulus Material 1 (refer to Stimulus Material 1 - used for this section and also used were then selected based on the frequency with which letter occurred as shown in the table. These letters in Section 3) were then selected based on the frequency with which letter occurred as shown in were then randomly placed, 10 words in ten lines. the table. These letters were then randomly placed, 10 words in ten lines. Although one font was used, from enumerators’ feedback (due to their experience in Primary teaching), Although certain one letters as was font such and from used, the “hâ€? enumerators’ the glottal feedback stop (‘) needed (due font a different to their experience so that in Primary these letters had the teaching), of appearance certain letters how the such letters as the looked when“hâ€? and the teachers glottal wrote themstop (‘) needed and how a used students different them.font so that these letters had the appearance of how the letters looked when teachers wrote them and how students used them. Section 2: Phonemic Awareness (‘Ilo’i ‘a e ongo ‘o e ‘uluaki mata’itohi) We decided to use the second approach suggested in the EGRA toolkit, which was to focus on the Initial Sound Identification. This was to have students identify the first sound in a selection of common words. 78 From our experience as local language teachers and educators, we felt that this section would be challenging for our students. To ask students to identify and sound out each sound present in the word (which is the first approach of ‘phoneme segmentation’, would unduly pressurize the students. We thought that it would be much better for them to identify only the first sound in these common words. Simple words (in according to Tongan pronunciation) were selected from the corpus, especially from the junior readers. There was also a consideration of a progression of simple to slightly less simple syllables. For example, the first word was [niu], the consonant “nâ€? has a very distinct sound, while the vowel combination of the sixth word [ou] is slightly more challenging. A letter frequency test was also conducted for the ten words so that the letters reflect the letter frequency table. 73 Section 3: Letter Sound knowledge (‘Ilo’i ‘a e ongo ‘o e ngaahi mata’itohi) Although the TOOLKIT suggested that the same laminated page of letters used in the first Stimulus Material, be also used for this section, there was a slight deviation with the Tongan Instrument. This was with the consonant “fakau’aâ€? or the glottal stop (the last letter). To quote C.M. Churchward, a reknowned Tongan linguist, in the first page of his book “Tongan Grammarâ€? – “In non-technical language, it may be described as a miniature clearing of the throat!â€?). Although the glottal stop is taught in all schools as a separate consonant, it does not have a distinct phonetic sound attached it. It must be accompanied by a vowel because it is sounded only before vowels and words mean differently when it appears. Therefore, in Stimulus Material 3, in all spaces in which a glottal space appeared, a vowel was added to it and the five vowels appeared according to the letter frequency in which they had occurred in the letter frequency table in Section 1. Section 4: Familiar Word Reading (Lau ‘o e Ngaahi Lea Maheni) As with section 4, there were no official word lists. Nevertheless, there were informal “sight wordsâ€? which the curriculum language writers and Class 1 – Class 3 teachers used. We collected these sight words from primary school teachers in primary schools in Nuku’alofa (capital) and from MEWAC’s (Ministry of Education, Women Affairs & Culture) Curriculum Development Unit. The words selected from the list of sight words collected for Stimulus Material 4 for this section, was based on their representing certain areas which class 1 – 3 curriculum covered and also children’s life experience in Tonga at this age group. We conducted cross-checking amongst selected words (in rows & columns) so that there be no relationship between the words. Section 5: Invented Word Reading (Lau ‘o e ngaahi lea fa’u) A list was compiled and was generated in combinations of increasing difficulty. • Criteria for the selection of non-familiar words (vowel/ consonant combinations in the Tongan language and proposed variations for the EGRA test) • The glottal stop (‘) is considered to be a consonant so therefore, it may not be used at the end of a word or between a vowel and a consonant. • Two consonants may not stand together and words may not end in a consonant. • Words may contain up to four vowels together and even without a consonant. • The consonant ‘ng’ represents one sound only /Å‹/. 74 Tongan vowel and consonant combinations may include: • CVCV • CVVCV • CVVCVV • CVV • VVCV • VCV • VV • VVV • VVVV The selection of non-familiar words will be based on consonant-vowel combinations typical of the Tongan language. In the Stimulus, the students are presented with forms legal to the Tongan language and using letters in legitimate positions. Students are tested on their ability to decode these words and the sounds they produce, regardless of whether they have meaning or not. The words will be tested against the letter frequency count identified for Sub-Test 1 and will progress in difficulty. Section 6 Passage Reading and Comprehension (Laukonga mo e Mahino) A number of narratives were written using a variety of sentence structures, names of characters, vocabulary etc. They were also tested for their letter frequency to ensure that the narratives also contained about the same letter frequency likely to be found in any document written in Tongan. We were mindful to keep to the narrative structure of having a beginning, a dilemma and resolution at the end, retain cultural appropriateness and to keep a ‘local’ flavour at all times. A narrative was chosen which followed the structure and requirements outlined in the EGRA guide and told of an everyday experience that was familiar to children in Tonga. A variety of sentence structures were used as well as some complex vocabulary. Questions were based directly on the narrative and were both fact- based and requiring inference from the text. 75 Section 7 Listening Comprehension (Fanongo mo e Mahino) A number of very short narratives were constructed with accompanying questions. A short conversation between two friends was selected and questions for the students to respond to. The activity in the very short narrative was again familiar to students and the questions relatively easy to respond to. Section 8 Dictation (Tala Kae Tohi) A number of sentences were constructed for selection in varying degrees of semantic and lexical complexity. Semantic Structure applied to examples for Dictation: All sentences begin with a capital letter and end with a fullstop unless it is a question and exclamation which end with the conventional punctuation marks. Long vowels are marked by a macron (¯) or a double vowel (VV) and stress is marked by the symbol (´). A number of sentences were developed and follow basic Tongan sentence structure. This is the sentence constructed for ST8. Ha’u ke tau Å? ‘o kaukau tahi he ‘oku ‘afu. Come, let’s go for a swim because it’s hot. Command/Imperative sentence – Used widely and often in the Tongan context. Children hear these kinds of constructions every day. These types of sentences will always begin with an imperative. 76