72345 v2 ead in g ra d e R a r ly G uatu E en t Va n s s essm urv ey A e lin e S ) B a s EG RA (Van Anglophone Stream | R e s u lt s R e p o r t ©2012 International Bank for Reconstruction and Development / International Development Association or The World Bank 1818 H Street NW Washington, DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this work is subject to copyright. Because The World Bank encourages dissemination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. For permission to reproduce any part of this work for commercial purposes, please send a request with complete information to the Copyright Clearance Center Inc., 222 Rosewood Drive, Danvers, MA 01923, USA; telephone: 978-750-8400; fax: 978-750-4470; internet: www.copyright.com. All other queries on rights and licenses, including subsidiary rights, should be addressed to the Office of the Publisher, The World Bank, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2422; email: pubrights@worldbank.org Table of Contents Table of Contents........................................................................................................................................... 1 List of Tables.................................................................................................................................................. 2 List of Figures................................................................................................................................................. 5 ABSTRACT.................................................................................................................................................... 7 ............................................................................................................................... 8 ACKNOWLEDGEMENTS. ........................................................................................................................... 9 SUMMARY AND FINDINGS. ............................................................................................................. 10 Survey results and main findings. Factors contributing to greater reading fluency and comprehension for Anglophone students................. 11 At the student level............................................................................................................................ 11 ............................................................................................................................ 12 At the teacher level. From assessment to intervention: next steps........................................................................................... 14 Chapter 1- Introduction............................................................................................................................ 17 Structure of the Report............................................................................................................................ 18 Chapter 2: Survey Implementation.......................................................................................................... 19 Sample Design........................................................................................................................................ 19 .............................................................................................. 20 Development of the VANEGRA Instrument. ........................................................................................................................ 20 Fieldwork and Data Entry. Reliability of the Instrument...................................................................................................................... 21 Chapter 3: VANEGRA English Results. ................................................................................................... 24 ................................................................................................................... 24 Structure of the Assessment. Administration of the VANEGRA English Instrument................................................................................. 26 VANEGRA English Results per Sub-test.................................................................................................. 29 Sub-test 1 – Letter Name Knowledge................................................................................................ 29 ............................................................................................... 30 Sub-test 2 – Initial Sound Recognition. Sub-test 3 – Letter Sound Identification............................................................................................. 32 1 Sub-test 4 – Familiar Word Reading................................................................................................... 33 ................................................................................................. 34 Sub-test 5 – Invented Word Reading. ................................................................................................. 35 Sub-test 6a – Oral Passage Reading. ............................................................................................. 36 Sub-test 6b – Reading Comprehension. .............................................................................................. 38 Sub-test 7 – Listening Comprehension. Sub-test 8 – Dictation........................................................................................................................ 39 ............................................................................. 44 Differences in Performance by Grade and Gender. Summary of Assessment Results............................................................................................................ 46 Chapter 4: Performance in Oral Reading Fluency and Reading Comprehension. ............................ 47 Chapter 5: Analysis of Student Factors Associated with Better Reading Outcomes. ....................... 50 Chapter 6: Analysis of Teacher Factors Associated with Better Reading Outcomes........................ 56 ................. 56 Association of Teacher Characteristics and Teaching Activities with Student Performance. Effect of Teacher Expectations on Student Performance.................................................................... 60 Chapter 7 - Next Steps............................................................................................................................. 67 ........................................................................................................................... 70 Bibliographical References. ANNEX 1/ TABLES....................................................................................................................................... 72 List of Tables Table 1 Student Factors Impacting Reading Outcomes............................................................................... 11 Table 2 Teacher and School Characteristics Impacting Reading Outcomes................................................. 13 Table 3 VANEGRA English sample by region, grade level and gender.......................................................... 19 Table 4 Reliability of the VANEGRA English assessment.............................................................................. 21 Table 5 Reliability of the VANEGRA English assessment (zero scores removed)........................................... 21 Table 6 VANEGRA English Instrument Structure and Early Skills Tested....................................................... 26 Table 7 Sub-test 1 Letter Name Knowledge: Results by grade and gender................................................. 30 Table 8 Sub-test 2 Initial Sound Recognition: Results by Grade and gender................................................ 31 Table 9 Sub-test 3 Letter Sound Identification: Results by Grade and gender.............................................. 32 ................................................ 33 Table 10 Sub-test 4 Familiar Word Reading: Results by Grade and Gender. 2 Table 11 Sub-test 5 Invented Word Reading: Results by Grade and Gender............................................... 34 Table 12 Sub-test 6a Oral Passage Reading: Results by Grade and Gender............................................... 35 Table 13 Sub-test 6b Reading Comprehension: Results by Grade and Gender........................................... 37 ............................................................................. 38 Table 14 Percentage of Correct Answers in Sub-test 6b. Table 15 Sub-test 7 Listening Comprehension: Results by Grade and Gender............................................ 39 Table 16 Sub-test 8 Dictation: Results by Grade and Gender...................................................................... 40 ............ 41 Table 17 Sub-test 8 Dictation: Number of letters and full words (total and correct) written by grade. .......... 41 Table 18 Sub-test 8 Dictation: Number of letters and full words (total and correct) written by gender. Table 19 Distribution of Students by Fluency............................................................................................... 49 Table 20 Average fluency, accuracy and reading comprehension levels, by condition of fluency.................. 49 Table 21 Characteristics of students in the sample along several student and family factors........................ 50 Table 22 Regression Results Part A............................................................................................................. 51 ............................................................................................................ 52 Table 23 Regression Results Part B. ............................................................................................................ 53 Table 24 Regression Results Part C. ............................................................................................................ 54 Table 25 Regression Results Part D. Table 26 Profile of Anglphone Teachers in VANEGRA.................................................................................. 57 Table 27 Effects of Teacher and School Characteristics on Literacy Acquisition........................................... 59 Table 28 Teachers median expectations about reading outcomes............................................................... 61 Table 29 Regression analyses of average relationships of teachers’ expectations on fluency in reading....... 64 Table 30 Regression analyses of average relationships of teachers’ expectations ........................................................................................................... 65 on reading comprehension. Table 31 VANEGRA English Reliability Matrix............................................................................................... 72 ...................................................................................................... 73 Table 32 Cronbach’s Alpha: Grade One. Table 33 Cronbach’s Alpha: Grade Two....................................................................................................... 73 Table 34 Cronbach’s Alpha: Grade Three.................................................................................................... 74 ............................................................... 75 Table 35 ANOVA Results: Differences in Means across Sub-tests. Table 36 Summary of student factors affecting reading outcomes of Anglophone Ni-Vanuatu students....... 78 .................... 83 Table 37 Results from differences in average scores determined by factor (tables 5.1 to 5.24). 3 Table 38 School and Teacher Characteristics: Mean Subtest Scores........................................................... 95 Table 39 School and Teacher Characteristics: Mean Subtest Scores........................................................... 96 Table 40 School and Teacher Characteristics: Mean Subtest Scores.......................................................... 97 Table 41 School and Teacher Characteristics: Mean Subtest Scores.......................................................... 98 Table 42 School and Teacher Characteristics: Mean Subtest Scores.......................................................... 99 Table 43 School and Teacher Characteristics: Mean Subtest Scores......................................................... 100 Table 44 School and Teacher Characteristics: Mean Subtest Scores......................................................... 101 Table 45 School and Teacher Characteristics: Mean Subtest Scores......................................................... 102 Table 46 School and Teacher Characteristics: Mean Subtest Scores......................................................... 103 Table 47 School and Teacher Characteristics: Mean Subtest Scores......................................................... 104 Table 48 School and Teacher Characteristics: Mean Subtest Scores......................................................... 105 Table 49 School and Teacher Characteristics: Mean Subtest Scores......................................................... 106 Table 50 School and Teacher Characteristics: Mean Subtest Scores......................................................... 107 Table 51 School and Teacher Characteristics: Mean Subtest Scores......................................................... 108 Table 52 School and Teacher Characteristics: Mean Subtest Scores......................................................... 109 Table 53 School and Teacher Characteristics: Mean Subtest Scores......................................................... 110 Table 54 School and Teacher Characteristics: Mean Subtest Scores......................................................... 111 Table 55 School and Teacher Characteristics: Mean Subtest Scores......................................................... 112 Table 56 School and Teacher Characteristics: Mean Subtest Scores......................................................... 113 Table 57 School and Teacher Characteristics: Mean Subtest Scores......................................................... 114 Table 58 School and Teacher Characteristics: Mean Subtest Scores......................................................... 115 Table 59 School and Teacher Characteristics: Mean Subtest Scores......................................................... 116 Table 60 School and Teacher Characteristics: Mean Subtest Scores......................................................... 117 Table 61 School and Teacher Characteristics: Mean Subtest Scores......................................................... 118 4 List of Figures ........... 22 Figure 1 Letter Level, Word Level, Oral Reading Fluency and Reading Comprehension Performance. Figure 2 Oral Reading Fluency, Reading and Listening Comprehension........................................................ 22 Figure 3 Oral Reading Fluency, Reading Comprehension and Dictation Scores............................................. 23 Figure 4 Stages of Reading Development..................................................................................................... 24 Figure 5 Early Grader Reading Assessment Components............................................................................. 25 Figure 6 VANEGRA English: Zero-score students as a percentage of the sample as a whole........................ 28 Figure 7 VANEGRA English: Zero-score students as a percentage in the sample per grade.......................... 28 Figure 8 Distribution of student responses to the writing items (total sample)................................................ 42 Figure 9 Distribution of student responses to the writing items (per grade).................................................... 42 .................................................. 43 Figure 10 Distribution of student responses to spelling items (total sample). Figure 11 Distribution of student responses to the writing items (per grade).................................................. 43 Figure 12 Differences in performance between boys and girls by grade, measured by the number of correct familiar words read per minute (CFWPM)............................... 44 Figure 13 Differences in performance between boys and girls by grade, measured by the number of correct words read in a connected text per minute (CWCPM)........... 45 Figure 14 Differences in performance between boys and girls in dictation by grade, as a weighted score..... 46 Figure 15 Average Scores in Oral Reading Fluency (sub-test 6a) and Reading Comprehension (sub-test 6b)................................................................................... 47 Figure 16 Average Reading Comprehension Levels in Fluent Students (N=130)............................................ 48 5 6 ABSTRACT In August 2010, the Government of Vanuatu carried out early grade reading baseline assessments in Anglophone and Francophone schools with financial support from local education partners and technical assistance from the World Bank. The results of the Vanuatu Early Grade Reading Assessment (VANEGRA) are cause for concern. English language findings show that while most students develop some fundamental skills in grade levels 1, 2 and 3, by the end of Grade 3 only about 1 in 4 students are able to develop fluency in reading to understand most of the text they read. The VANEGRA survey also collected data on the attributes of students, teachers and schools. Factors that were shown to be predictors of better reading performance in the early grades include: speaking English at home, owning the reading textbook, having literate parents, having books at home, reading at school and at home, attending kindergarten, doing homework, and receiving help from a family member to do homework. Teacher certification and experience showed no statistical effect on student reading outcomes; however, attending in-service teacher training on reading showed statistical significance on both fluency development (measured in the number of words read correctly per minute) and reading comprehension. With regard to resources and school management, better student reading performance was statistically associated with the availability and use of school libraries, holding a meeting with parents at least once per month, and having and using the recommended Vanuatu reading texts. VANEGRA also asked about use of seven reading instructional activities. Many of these were associated with better reading outcomes. Lastly, VANEGRA asked about teacher expectations for students’ reading performance. Teachers reporting expectations below the median had a negative association with reading fluency and comprehension at statistically significant levels, however causation cannot be inferred. Based on the analysis presented, recommendations for improved reading instruction and greater parental involvement are presented at the end of the report. 7 ACKNOWLEDGEMENTS The Vanuatu Early Grade Reading Assessment (VANEGRA) baseline surveys in English and French are the result of the Government of Vanuatu’s commitment to improve reading levels in the country through a mid-term process that incorporates assessment data to improve reading instruction and promote greater parental and community involvement in students learning outcomes. VANEGRA diagnoses comprise the first step in the process by providing Ni-Vanuatu education officials with a system-level diagnosis of how well – and at what pace -- Ni-Vanuatu children in Anglophone and Francophone schools develop foundational skills needed to become literate. The reports were prepared by Myrna Machuca-Sierra (Education Specialist) and James A. Stevens (Senior Operations Officer) of the World Bank’s East Asia and the Pacific Education Unit (EASHE). Eleanor Wang (Junior Professional Associate, EASHE) supported data entry activities. Jose Ramon Laguna, Margaret Triyana and Steph de Silva provided support during data cleaning and validation, and the analysis of results. The report benefitted greatly from the comments made by Eduardo Velez-Bustillo (Education Sector Manager, EASHE), Stephen D. Close (Human Development Specialist, EASHE), Warwick Elley (University of Canterbury, New Zealand), Cedric Croft (Consultant, Ministry of Education Vanuatu) and Barbara Thorton (International Development Consultant). In a technical workshop following the presentation of preliminary results, staff from the Ministry of Education’s Curriculum Development Unit and the In-Service Unit provided excellent insights and recommendations on how best to advance a reading improvement agenda that benefits from VanEGRA results. Errors or omissions are the authors’ sole responsibility. The authors would like to express their gratitude to the Ministry of Education of Vanuatu, in particular to the Hon. Min. Charlot Salwai, and Mr. Roy Obed (Acting Director General of Education), for their leadership in this project. The VANEGRA Survey received great support from senior officials including Mr. Charley Robert (Acting PEO Curriculum and Assessment), and Mr. Donald Wotu (Acting Senior Education Advisor, Examinations and Assessments Unit). In particular, the authors would like to acknowledge the work of the VANEGRA Anglophone and Francophone core team members, enumerators and supervisors whose hard work and dedication made the English and French VANEGRA surveys possible: Alvin Tari, Ansen Veremaeto, Asanat Tasale, Antoinette Bihu, Bill Bule, Bris Mermer, Buddy Bule, Carmel Melsul, Clemontine Etul, Collin Jacob, Daniel Kohea, Daniel Norlan, Dolores Ngelgen, Dolores Virelala, Don Joseph, Dorneth Kalo, Edmon Hillary, Edward Ben, Enoch Leon, Fred Ottiman, George Josiah, Georgeline John, Gladys Esecher, Gossip Miken, Hapina Kapotua, Harkuk Vocor, Imbert Tevi, Jeffry Ruben, Jenny Sanga, Jerome Ludvaune, Jesica Gambetta, Joseph Buleru, Joshian Molvurai, Kalmaire Morrison, Katchiri Tanga, Kathrine Naliupis, Leah Viro, Lenah Tambe, Lidcha Nanuman, Lucian Bires, Marie Assumpsion, Marie Manu, Marie Tavussi Moli, Marie-Pierre Malere, Mele Socopoe, Michelle Atuary, Particia Mabontare, Patrick Esecher, Paul Michael, Paul Tabi, Paul Thompson, Peter Jacob, Peter Patison, Prescilla Olul, Presley Gaiala, Rachel Henry, Redina Api, Rossie Rihu, Samuel Kaltoutak, Seth Niavie, Silas Boas, Simon Bulekap, Simon Namol, Stangley Lanson, Steven Yawiko, Suthy Lunabek, Tania Melenamou, Thomas Butu, Timothy Lokai, Yamei Johnson. Last but not least, the authors would like to thank the 1,282 Anglophone students and the 1,293 Francophone students who enthusiastically participated in the survey. To all, tenkiu tumas.   8 SUMMARY AND FINDINGS This report summarizes the results of the English-language early grade reading assessment conducted in August 2010 by staff from the Vanuatu Ministry of Education. There were 1,282 students were assessed from 40 randomly-selected Anglophone primary schools in Vanuatu. A separate survey was carried out in French-language schools at the same time. Financial support for the survey was provided by local education partners through the Vanuatu Education Road Map (VERM). Technical assistance and management support was provided by the World Bank. The assessment is part of a global initiative aimed at helping countries measure how well children are learning to read in the early grades of primary education. It aims to help educators develop local knowledge about the specific skills students are struggling with and the factors that appear to contribute to reading development in their schools. Equipped with such evidence, education stakeholders can come together to devise response strategies to improve reading instruction, monitor student’s reading progression, and promote greater parental and community involvement to ensure all children develop the skills needed to become effective readers. The VANEGRA English assessment consisted of eight modules or sub-tests covering basic reading skills such as phonemic awareness, recognition of letter names and sounds, automatic word reading, decoding, oral reading fluency, and comprehension – measured in terms of both reading and listening to short narrative passages. A short dictation exercise was included to test early writing skills such as spelling, orientation to text, spacing, capitalization, and punctuation. The VANEGRA English student test was complemented by a student contextual interview which collected information about socioeconomic characteristics, such as availability of reading books at home and literacy prevalence among family members. The survey also included a teacher questionnaire which gathered data on the qualifications of Anglophone teachers, their expectations about reading outcomes and the frequency with which they use methods of reading instruction and assessment. Although the VANEGRA English instrument followed a standardized process of adaptation to the local context, results are meant to be used to diagnose gaps in reading instruction, and not for cross-country comparison. The survey seeks to provide a baseline standard of reading fluency in English. A version with assessment instructions in Bislama was also produced to accommodate students whose limited proficiency in English could have limited their understanding of the instructions in each sub-test. The reliability of the instrument to capture reading abilities in Vanuatu was estimated using Cronbach’s alpha, a common measure of reliability of survey instruments, at a coefficient of 0.94 - in a scale going from 0 to 1, the minimum Cronbach coefficient acceptable in research is 0.7. The analysis of VANEGRA English student data included descriptive statistics (means and standard deviations) to measure average levels in basic reading skills; an analysis of variance was carried out to determine the statistical significance of differences in average scores between groups with and without factors identified as predictors of reading. Also, regression analyses were carried out to estimate average oral reading fluency and associated reading comprehension levels for the sample as a whole. Lastly, statistical analyses were carried out to estimate the relationship between teacher characteristic and student reading outcomes. The most relevant results are presented below, followed by a discussion of policy implications for reading instruction and teacher professional development in the country. 9 Survey results and main findings As students progress across grades, they develop better competency in all basic reading skills; however, these gains do not lead to demonstrated reading fluency (words read correctly per minute) and improved comprehension for most Grade 3 students. As Anglophone students progress from Grade 1 to 3, average competency in all sub-tests improves with the largest gains observed at the end of Grade 2. Yet, low mastery of the relationship between letters and sounds appears to be one of the main reasons why students struggle to read words both in isolation and in the oral reading passage. Poor word-level reading abilities may be hindering the development of fluency in reading which could explain why at the end of Grade 3, only about 1 in 4 students (24% of Grade 3 students) is able to read at the fluency level needed to understand most of the text they read. In the Anglophone sample, less than 1% of Grade 1 students are fluent, increasing to 5% of Grade 2 students and 24% of Grade 3 students. Among fluent students (N=130), 18% are in Grade 2 and 80% are in Grade 3, indicating the critical role Grade 3 plays in the development of reading fluency among beginning readers. Low scores are partly due to the number of students for whom the test had to be discontinued because they lacked the minimum knowledge tested. In VANEGRA English, a particular subtest could be discontinued if the student was unable to read the minimum number of letters or words needed. Early- stop cases are allowed in all sub-tests except listening comprehension and dictation. Students assessed as early-stop cases -i.e., zero-score-- serve as a measure of the number of students with the lowest score possible. In VANEGRA English, for the sample as a whole, the proportion of zero-score students is above 30% in sub-tests requiring word-level reading skills (familiar word reading, invented word reading, and oral passage reading). In sub-test 5 alone, 59% of the students in the sample were unable to successfully decode the first 5 invented words in the exercise, which suggests most students struggle to match letters and sounds to create words. Achieving oral fluency in reading is crucial to improve reading comprehension. As students achieve automaticity on the “mechanics� of reading –i.e., matching letters and sounds to make up words and sentences-- they develop fluency in reading, allowing them to read longer texts and focus on the meaning of the text. An analysis of oral reading fluency and reading comprehension among Anglophone Ni-Vanuatu students showed that students achieve greater levels of reading comprehension when they read at an average rate of at least 45 correct words per minute (CWCPM). At this standard, 10% of the sample, or 130 out of 1,282 Anglophone students tested, could be considered fluent in reading. The differences in oral reading fluency and comprehension between fluent and less-than-fluent are striking: while fluent students read almost all the narrative passage (58 out of 60 words) at an average fluency of 71 correct words per minute and understand about 83% of the text they read, less-than-fluent students read on average about 20 out of 60 words in the passage at an average fluency of 10 correct words per minute, which allows an average comprehension of only 13% of the text. Finally, VANEGRA English results showed that in many sub-tests, girls performed better than boys and appeared to progress faster in some –though not all- of the skills tested. Girls’ scores were statistically better than boys in 5 out of 8 sub-tests (letter name knowledge, letter sound knowledge, familiar word reading, oral passage reading, and dictation). Non-statistically significant results in the remaining three 10 sub-tests suggest that, on average, boys and girls struggle equally to decode invented words (decoding skills), understand stories they hear (listening comprehension) and isolate sounds of letters in the context of words (phonemic awareness skills). In addition, average differences in performance by gender resulting from the interaction of gender and grade showed that after controlling for the grade effect, gender differences are statistically significant only in three sub-tests in Grades 2 and 3 (familiar word reading, oral reading passage and writing). These differences suggest girls transition into word-level fluency and reading fluency faster than boys, which may contribute to a larger number of girls achieving fluency -- 11% or 75 of the 645 girls in the sample versus 7% or 55 of the 637 boys in the sample. Girls comprised 60% of the fluent students in the sample. Factors contributing to greater reading fluency and comprehension for Anglophone students At the student level VANEGRA included a series of questions about student characteristics and behaviors which could be associated with reading abilities. For example, as one may expect, better performance on most VANEGRA English sub-tests were associated with students who reported having literate parents, students who reported having books at home, students whose teachers read aloud in the classroom, students whose parents read with them at home, and students who reported doing homework and receiving help from their parents on homework. These associations, however, vary by grade and gender. Following, is a summary of characteristics and behaviors that were found to have the most impact on student reading outcomes: Table Table 1- 1- Student Student Factors Factors Associated to Associated toBetter BetterReading Outcomes Reading Outcomes Factor Impact Estimated Difference Speaks On average, students who speak English at home scored about 2 words per minute more in the English at Positive reading fluency measure and score about 5% better in the listening comprehension sub-test than Home those who reported not to speak English at home. Teacher who On average, students whose teachers read aloud as part of their teaching activities showed 4% Positive Reads Aloud better listening comprehension than those whose teacher does not read aloud to the class. Books On average, students who reported having books at home scored 5 words per minute more in available at Positive the reading fluency measure, about 6% better in reading comprehension and about 8% better in home listening comprehension than those who reported not owning books at home. Having a literate mother or a sibling appeared to show the strongest associations with better Literacy in reading outcomes – almost 3 more words per minute for each in the reading fluency measure. Positive the family Students with literate mothers or siblings also showed better results in reading and listening comprehension of up to 15% when compared to students who lacked either. On average, students who reported being absent more than one week during the school year Absent >1 Negative scored 3 words per minute less in the reading fluency measure, 3% and 4% less in the reading Week and listening comprehension measures, respectively. Attended On average, students who reported having attended kindergarten scored 2.6 words per minute Positive Kindegartern more in the reading fluency measure than those who reported not having attended it. 11 At the teacher level VANEGRA also explored the association between teacher characteristics and student performance using data collected through the teacher questionnaire on experience, certification, methods of instruction and assessment, and learning expectations. The most interesting finding was that teacher experience had no significant association with oral reading fluency, while certification in the form of a Certificate of Primary Teaching had a significant negative association. However, in-service teacher training on general content or reading instruction in the last 3 years translated into positive and statistically significant differences on student reading outcomes. The teacher characteristics and behaviors significantly associated with student reading outcomes are given in the table below. The estimated difference reports the average difference in scores between students whose teacher reported any of those factors versus those whose teacher did not.   12 Table 2 Table and and 2 - Teacher - Teacher School School Characteristics Associated Characteristics Associated to Better Reading to Better Outcomes Reading Outcomes Factor Impact Estimated Difference Average gain of 5 CWPM in reading fluency, about 6% better Students copied down text Positive reading comprehension scores, and 5% improvement in from the chalkboard dictation scores Average gain of almost 2 CWPM in reading fluency, 1% better Students retold a story that reading comprehension and 6% better listening Positive they had read comprehension scores, as well as 5% improvement in dictation scores Average gain of 2 CWPM in reading fluency, 5% better Students sounded out reading comprehension and 9% better listening Positive unfamiliar words comprehension scores, as well as 6% improvement in dictation scores Average gain of 16 CWPM in reading fluency, 17% better reading comprehension and 18% better listening Students Read Aloud Positive comprehension scores, and about 24% improvement in dictation scores Average gain of almost 8 CWPM in reading fluency, 8% better Students assigned reading on reading comprehension and almost 13% better listening Positive their own comprehension scores, as well as 11% improvement in dictation scores Average gain of 6 CWPM in reading fluency, almost 7% better reading comprehension and almost 13% better listening Library Present Positive comprehension scores, as well as 5% improvement in dictation scores Average gain of 9 CWPM in reading fluency, about 8% better reading comprehension and 7% better listening Supervision in Library Positive comprehension scores, as well as 7% improvement in dictation scores Average gain of almost 4 CWPM in reading fluency, 6% better reading comprehension and almost 4% better listening Reading Corner Present Positive comprehension scores, as well as 3% improvement in dictation scores Average gain of almost 2 CWPM in reading fluency, almost 4% better reading comprehension and 13% better listening PTA Functioning Positive comprehension scores, as well as 5% improvement in dictation scores Average gain of almost 4 CWPM in reading fluency, 4% better reading comprehension and 3% better listening Meeting with Parents Positive comprehension scores, as well as 4% improvement in dictation scores Average gain of almost 8 CWPM in reading fluency, 8% better School has Recommended reading comprehension and 6% better listening Positive Reading Texts comprehension scores, and almost 7% improvement in dictation scores Average decline of 14 CWPM in reading fluency, 18 % poorer Negative (Certificate reading comprehension and almost 19% poorer listening Certification in Primary comprehension scores, and almost 18% decline in dictation Education vs None) scores Average gain of almost 4 CWPM in reading fluency, about 3% better reading comprehension and almost 1% better listening Inservice Attendance Positive comprehension scores, as well as 3% improvement in dictation scores Average gain of 11 CWPM in reading fluency, 13% better Teacher experience (recent reading comprehension and 11% better listening graduates vs next group by Positive comprehension scores, as well as 16% improvement in experience) dictation scores Note: CWCPM stands for “Correct Words in Connected Text Per Minute� 13 13 From assessment to intervention: next steps VANEGRA English survey results call for an immediate response to improve reading instruction to ensure Anglophone students are equipped with the knowledge required to become skilled readers. Specific recommendations include: • Improve the focus and structure of reading instruction to promote greater fluency in reading by the end of Grade 3. Research has shown that developing fluency in reading is crucial to help students become effective readers in the first years of primary education. As students approach reading at a speed of about 45-60 words per minute, the reader becomes better able to focus on the meaning of the text rather than on individual letters and words. As shown by VANEGRA English results, students reading at least 45 correct words per minute were able to understand about 83% of the text they read. However, only 1 in 10 Anglophone students is able to reach this fluency level at the end of Grade 3. Two factors could contribute to explain these results. On the one hand, the low levels of letter recognition skills and poor knowledge of phonetic principles observed in the VANEGRA suggest current instruction falls short in developing a solid foundation for the development of reading fluency. On the other, poor English skills on students entering the Anglophone stream may contribute to delay the development of pre-reading skills in Grades 1 and 2 as students struggle to develop language and reading skills simultaneously. As the MoE sets forth to implement the new K-12 curricula, it is fundamental that instructional improvements in the early grades take into account the linguistic diversity of the country and provide adequate strategies to prepare students for reading development in a secondary language (L2). In addition, the new curriculum opens up the opportunity to improve instruction of letter and word-level reading skills to promote a better sequenced instruction of basic reading skills. • Ensure teachers working in the early grades have the knowledge to improve their practice to impact the reading outcomes of their students. In order to improve reading instruction in Vanuatu, teachers will have to improve their knowledge of reading instruction to improve classroom practice. Although most teachers in VANEGRA had fairly realistic expectations about the reading outcomes of their students, average scores per sub-tests in the assessment fell short of fulfilling these expectations. Moreover, the fact that some instructional methods typically associated with better reading outcomes showed no statistical significance in the Vanuatu context, while others showed counterintuitive effects on student outcomes, suggests the need to review how these activities are carried out in the classroom to better understand the possible factors that are hindering their effectiveness. This is also true of the average effects of teacher methods observed on student outcomes. • Support the reading instruction skills of as many teachers working with beginning readers as possible. Data from the teacher questionnaire showed that only 29% of the teachers in the sample participated in general in-service training courses and only 20% had attended in-service training on reading in the last two years. If only 2 in 10 teachers in the country benefit from learning about specific ways in which they can improve their practice, Ni-Vanuatu teachers will continue to practice their profession in isolation. This would be particularly critical for the most experienced teachers –i.e., those in the >50th and 75th percentiles of experience- for whom experience tended to negatively 14 affect the reading outcome of their students. The role of the newly created In-Service Unit (ISU) at the MoE will be critical to further develop teacher knowledge and practice for reading instruction. As such, it is recommended that VANEGRA findings inform the development of the lesson plans and materials and that ISU staff works in close collaboration with curriculum developers to ensure teachers understand the new curriculum goals and receive support on how to achieve them. • Establish reference reading standards to monitor reading development in the early grades. As the MoE moves on to establish an oral reading fluency standard under VERM, it is important to consider that these indicators should be considered reference standards and not high-stakes benchmarks that would jeopardize additional funding or the promotion of teachers. Since these reference standards are drawn from baseline data, additional measures will be needed in subsequent years to learn about the rate at which Ni-Vanuatu students develop reading abilities. In this sense, reading standards should not be seen as high-stakes but an essential piece to monitor reading progression in the classroom. In order to set up national reference standards to monitor system- level quality improvements, it would be best to use the percentage of zero-score students in selected sub-tests as a marker and track reductions in the shares at least biannually. Monitoring achievements over time will eventually provide more information on the rate and the way in which average fluency develops among Anglophone students. A modified version of the test could be used to screen students during the school year that may be in need of additional support. • Help teachers translate national reference standards into easy-to-assess, easy-to-monitor reading goals to monitor the reading progression of their students during the school year. In order for teachers and schools to be able to be held accountable for reading outcomes, teachers, school officials and parents need to understand what these standards mean and how each can support reading development in their own school. School development plans should contain reading improvement goals as part of their minimum service standards, as well as a description of activities aimed at encouraging reading. Parents and the community as a whole should be brought into this effort. • Introduce policy actions that increase student exposure to literacy outside the school. VANEGRA English results showed how students who have reading books at home have better reading outcomes and are more likely to become fluent readers. The effect was positive for both boys and girls and for all basic reading skills. Thus, it is advisable that the MoE promotes increased student access to books at home. However, making more books available to students will not per se ensure better reading outcomes. Along with access to more reading materials, Ni-Vanuatu children will need support to develop a reading habit beyond the requirements of the school curriculum. One way of achieving this would be ensure the books being procured by the ongoing Book Flood program are not only grade-appropriate but that they are accessible to students in and outside of the classroom. Since 74% of the teachers reported having access to a school library, an adequate book-borrowing scheme carries the potential to expose students to print on a more regular basis. Another way of increasing exposure to literacy would be to develop community literacy programs where schools become a focal point of literacy in the community. Teachers and community leaders can start up reading clubs and reading competitions to further promote a reading culture among beginning and more experienced readers. 15 • Promote strategies to assure greater parental and community involvement in the reading development process of children. Research shows that the earlier the parental involvement, the more powerful and long-lasting the effects will be both in terms of academic and behavioral outcomes of children. Moreover, research also shows that the most effective form of parental involvement includes those where parents participated in learning activities at home. As seen in VANEGRA results, holding PTA meetings at least once a term translated into one of the largest positive effects on reading outcomes next to access to books and guided reading by teachers. What is more, having parents read at home with their children also showed positive effects on student reading outcomes. If parents and schools communicate regularly on the academic progress of children, parents tend to monitor school and classroom activities, and coordinate efforts with teachers such as helping with homeworks and carrying out extracurricular activities. For this to happen, it is important that parents and schools commit jointly to the reading development process of children. In addition to providing information on the academic progress of children, schools can advise on –and even facilitate- different ways in which parents can promote reading at home. If reading outcomes are to improve in the country, reading development must be seen as a joint enterprise that extends beyond the teacher and the school classroom environment. • Finally, it is clear that more research is needed to better understand the factors that contribute to differences in reading performance between boys and girls. An analysis of the factors that contribute to these differences is beyond the scope of this survey. However, VANEGRA English data showed that boys and girls finish Grade 1 at similar levels of performance in reading, but in Grade 2 girls move into word-level reading faster than boys and the difference increases at the end of Grade 3. A better understanding of this phenomenon is critical to inform sector policies and increase the success of future reading development programs. 16 1 Chapter 1- Introduction Since 2007, the Vanuatu Ministry of Education (MoE) has administered the Vanuatu Standardized Test of Achievement (VANSTA), a national assessment to monitor literacy and numeracy skills of students in Grades 4 and 6 of primary education. VANSTA’s 2007 and 2009 results revealed that a large share of students is failing to achieve reading comprehension and writing outcomes expected at their grade level (SPBEA, 2009). While VANSTA provides an indication that many students are not reading at levels deemed appropriate for Grades 4 and 6, it does not provide detailed findings about problems in the primary system that led to poor reading performance in Grades 1 through 3. To complement VANSTA results, the Vanuatu MoE sought to learn if students in Grades 1 to 3 are developing the basic reading skills needed to read fluently and understand what they read. If VANSTA scores are the result of low competence in basic reading skills, it is necessary to identify gaps in instruction where additional instruction and inputs may be needed. In response, the Vanuatu MoE, local education stakeholders including donors,1 and the World Bank joined efforts to conduct a national assessment of basic reading skills in Anglophone and Francophone schools, using adapted versions of the Early Grade Reading Assessment (EGRA) tool appropriate to the Vanuatu context. The assessments are part of a global initiative aimed at helping countries measure how well children are learning to read in the early grades of primary education. From June 2nd to August 26th 2010, a team of Anglophone and Francophone Ni-Vanuatu reading and language specialists, consultants, and staff from the MoE worked together with the World Bank to develop, trial and administer the Vanuatu Early Grade Reading Assessment (VANEGRA) Surveys before the end of the 2010 school year. The purpose of the VANEGRA surveys was three-fold: 1. To develop a baseline survey of basic reading skills and temporary reference standards to monitor reading performance in schools and system wide; 2. To build local capacity to replicate early grade reading assessments in the future; and 3. To work with local education stakeholders to interpret VANEGRA findings and analyze their policy and sector investment implications. In particular, the Vanuatu assessments aimed to answer the following questions: • What are the basic reading skills acquired by Anglophone and Francophone Ni-Vanuatu students in Grades 1, 2 and 3? • What are the reading fluency levels at which Ni-Vanuatu students reach high enough levels of comprehension to understand what they read? • What are the factors that influence the acquisition of reading skills among Ni-Vanuatu students? 1 VANEGRA surveys received financial support from pooling partners (AusAID, NZAID, the European Union, UNICEF) in the Vanuatu Education Partners Group. 17 To answer these questions, the latest English and French versions of the EGRA tool were adapted to the Vanuatu context. Because the EGRA tool is an orally-administered test – i.e. carried out as an interview - it is suitable to be administered to young children whose reading and writing skills have not fully developed. Given Vanuatu’s linguistic diversity and the difficulties reading assessment in a secondary language (L2) impose on students with poor language competence; we included two additional versions of the VANEGRA instruments (Bislama-English and Bislama-French). These contained instructions in Bislama to accommodate students whose limited proficiency in English could have limited their understanding of the instructions for each activity.2 The VANEGRA tools comprised three instruments: (1) a diagnostic instrument assessing basic reading, listening and writing skills among Grade 1, Grade 2, and Grade 3 students; (2) a student contextual interview gathering information on the student’s background, administered to all participating students; and (3) a teacher questionnaire regarding teacher characteristics, expectations, and assessment and instruction methods, answered by all Grade 1, Grade 2 and Grade 3 teachers in the sample schools. Each set of instruments was developed in English and French for their administration in Anglophone and Francophone schools, respectively. This report summarizes the main findings from the VANEGRA English survey and provides policy recommendations to inform sector discussions and literacy improvements in Vanuatu. Equipped with information about the specific skills students are struggling with and the factors that appear to contribute to reading development in their country, education stakeholders in Vanuatu can come together to develop response strategies to improve reading instruction and monitor student progression, in order to ensure all children develop the skills needed to become effective readers. Structure of the Report Chapter 1 briefly presents the purpose of the survey and how VANEGRA results are expected to improve learning outcomes in Vanuatu. Chapter 2 summarizes the overall implementation of the survey, in particular, the process followed to develop the VANEGRA English instrument. Chapter 3 presents the main results from each of the sub-tests administered. Chapter 4 presents the analysis of oral fluency and reading comprehension levels and a discussion about the establishment of a reference standard for oral reading fluency in the country. Chapter 5 and Chapter 6 present the results of the analysis of student and teacher factors associated with reading acquisition among Anglophone students. Finally, Chapter 7 draws conclusions about the key results to present possible recommendations for improving the quality of reading instruction in Anglophone schools. 2 Literacy acquisition in a secondary language (L2) is a complex process even if one does not consider the effect of oral competence in the subject’s primary language (L1). Research on reading and writing suggest literacy acquisition in a secondary language (L2) may produce varying outcomes depending on the nature of literacy in the primary language (L1) and/or the extent to which it has been mastered (e.g. Alderson, 1984; Carrell, 1991; Carson, 1991). Many students will confront literacy acquisition in L2 with good foundational skills in L1 whereas others will do without sufficient oral competence to support literacy acquisition in L2. Since literacy acquisition in Vanuatu is instructed in L2, VANEGRA results should be interpreted as a measure of reading outcomes in L2. 18 2 Chapter 2: Survey Implementation3 In order to build local capacity to replicate early grade assessments in the future, the Ministry of Education of Vanuatu requested technical assistance from the World Bank during survey preparation and administration. Between June 2nd to August 26th and November 3rd to the 17th, 2011, the World Bank provided in-country support to selected Ministry staff to undertake the sample design, adapt the VANEGRA instruments in English and French, facilitate the training of enumerators and supervisors, coordinate survey logistics during the pilot and fieldwork, and carry out test marking and data entry. Sample Design On June 2nd – 31st, 2011, a series of preparatory meetings took place to discuss the scope and purpose of the survey. With advice from the World Bank, the Vanuatu MoE chose a nationally representative sample with contrast groups according to Grade level (Grades 1, 2, and 3). The final sample design did not incorporate contrast groups by school type and regions, thus survey results can only suggest estimates by Grade and gender.4 The target population was defined as students enrolled in Grades 1 to 3 in primary schools implementing the official curriculum. Using enrollment data from the Vanuatu Education Management Information System (EMIS), a sample of 40 schools was selected using a stratified random design with proportional allocation based on school type –government or government-assisted-, region,5 and school size to ensure all school types and regions would have a probability of selection equal to their actual distribution in the country. The final sample consisted of 1,282 students, 645 girls and 637 boys (see Table 3). Table 3 - VANEGRA English sample by region, grade level and gender Table 3 - VANEGRA English sample by region, grade level and gender 1st Grade 2nd Grade 3rd Grade Province Total Male Female Total Male Female Total Male Female Total Malampa 34 40 74 35 38 73 36 38 74 221 Penama 34 32 66 33 33 66 33 33 66 198 Sanma 34 38 72 35 37 72 36 36 72 216 Shefa 54 55 109 54 57 111 61 49 110 330 Tafea 43 41 84 41 43 84 41 41 83 250 Torba 12 10 22 11 11 22 11 12 23 67 Total 211 216 427 209 219 428 218 210 428 1,282 Development of the VANEGRA Instrument 3 The term “survey instruments� used refers to the set of VANEGRA documents administered to students and teachers. The term Due to differences “assessment� in language, refers to the VANEGRA culture, diagnostic and instrument expectations consisting about of 8 sub-tests learning or sections. Theoutcomes, theisEGRA term “EGRA tool� used as tool is to reference adapted and piloted the latest English, to Spanish French and context fit the versions of each which country have been adapted where applied. From in EGRA-participating June countries to fit 20 the to thlocal context. 30 , 2011, teams of four Anglophone and four Francophone reading, language and assessment th 4 In EGRA minimum sample sizes, some 400 children are needed for any combination of contrast groups of interest. For example, a specialists sample from comparing the MoE male-urban woulddeveloped draft require some 400, versions as would of themale-rural, female-urban, instruments which thus, and female-rural; were piloted in distinguishing by Vila Port and gender from locality July would 12th a require –sample 14th, 2011. Minor size of some 1600, changes were baseline whereas a simple incorporated to the per grade would pilot require instruments only 400 per grade. In order to meet VANEGRA logistical costs and timeline, a decision was made to include only gender and grade (≥1,200 students) as groups. readability and clarity of questions in the narrative passages. From August 2 to to improve contrast nd 5 11th, four training workshops - 2 for Anglophone and 2 for Francophone enumerators - were Due to geographical and cost limitations, only the main island in each of the provinces was surveyed. held simultaneously in Port Vila and Luganville. These workshops were attended by 19 temporary and retired teachers as well as zone curriculum advisors from the six provinces in the country. The decision to hold four parallel training sessions was made to reduce transport Development of the VANEGRA Instrument Due to differences in language, culture, and expectations about learning outcomes, the EGRA tool is adapted and piloted to fit the context of each country where applied. From June 20th to 30th, 2011, teams of four Anglophone and four Francophone reading, language and assessment specialists from the MoE developed draft versions of the instruments which were piloted in Port Vila from July 12th – 14th, 2011. Minor changes were incorporated to the pilot instruments to improve readability and clarity of questions in the narrative passages. From August 2nd to 11th, four training workshops - 2 for Anglophone and 2 for Francophone enumerators - were held simultaneously in Port Vila and Luganville. These workshops were attended by temporary and retired teachers as well as zone curriculum advisors from the six provinces in the country. The decision to hold four parallel training sessions was made to reduce transport costs and develop capacity among VITE/CDU and EAU staff to conduct similar trainings in the future. On August 10th, each training session held an enumerator practicum in selected schools in Luganville and Port Vila. After the practicums, each team had feedback meetings where great emphasis was placed on the importance of ensuring that forms were completed fully, clearly, and correctly by every enumerator. Inter- rater reliability – i.e. the ability of enumerators to administer the assessment correctly and consistently - was calculated with results of 85% and above for all sub-tests except Sub-test 3 –Letter Sound Knowledge – for which a 72% rate of reliability was achieved. Fieldwork and Data Entry Data collection took place between August 11th and 26th 2011. There were 32 enumerators, 3 VANEGRA Anglophone trainers and 1 fieldwork coordinator who carried out data collection in 40 sample schools in the islands of Gaua (Torba Province), Pentecost (Penama), Malekula (Malampa), Éfaté (Shefa), Santo (Sanma), and Tanna (Tafea). Complete survey documents were brought back to Port Vila by the VANEGRA trainers and fieldwork coordinators for marking. Data entry took place between November 3rd and 17th 2010. There were several instances of ‘out of range responses’ from the teacher and student questionnaires that were coded as missing data in the analysis. For the 98 teachers surveyed in English-speaking schools, item response rates were above 82%. For the students, item response rates were above 86%. Teacher responses were matched to their students based on school, section and grade where section and grade –e.g. grade 1 section A - indicate the particular student-teacher pair within a school. Participation was very high, only one student out of 1283 declined to participate. Reliability of the Instrument Cronbach’s alpha was calculated for the VANEGRA instrument to measure the reliability of the test. Results showed a strong internal consistency with a coefficient of 0.94. As a rule of thumb, an alpha coefficient of 0.80 is considered good and 0.7 is typically the minimum acceptable. The alphas for each sub-test were close to 1, suggesting high reliability across VANEGRA sub-tests (see Table 34 in Annex 1 for the VANEGRA intercorrelation matrix). 20 Reliability of the Instrument Cronbach’s alpha was calculated for the VANEGRA instrument to measure the reliability of the test. Results showed a strong internal consistency with a coefficient of 0.94. As a rule of thumb, an alpha coefficient of 0.80 is considered good and 0.7 is typically the minimum acceptable. The alphas for each sub-test were close to 1, suggesting high reliability across VANEGRA sub-tests (see Table 34 in Annex 1 for the VANEGRA intercorrelation matrix). Table 4- Table 4 Reliability of the - Reliabilityof VANEGRA the VANEGRA English English assessment assessment average item-test item-retest Item Obs Sign inter item alpha correlation correlation covariance Correct Letters Per Minute 1282 + 0.85 0.80 0.65 0.94 Phonemic Awareness 1282 + 0.66 0.57 0.70 0.95 Correct Letters Sounds Per Minute 1282 + 0.85 0.80 0.65 0.94 Correct Words Per Minute 1281 + 0.92 0.90 0.63 0.93 Correct Non Words Per Minute 1282 + 0.86 0.81 0.65 0.94 Oral Reading Fluency 1282 + 0.93 0.90 0.63 0.93 Reading Comprehension 1282 + 0.89 0.85 0.64 0.93 Listening Comprehension 1282 + 0.68 0.60 0.70 0.95 Writing 1282 + 0.88 0.84 0.64 0.94 Test scale 0.66 0.94 However, However, due due the to to high the proportion high of scores of zero proportion in the data, zero scores in the was necessary it data, to determine it was necessary the reliability to determine of thethe instrument reliability ofwhen these scores when the instrument were excluded to avoid so as were these scores so correlation a spurious excluded as to avoid in the data. Table a spurious 5 indicates the correlation reliability in the data. of the instrument Table under 5 indicates this assumption. the reliability The overall reliability of the instrument under this of the data is 0.93, assumption. which is still excellent. Cronbach’s alpha was also calculated for each grade and these tables are available The overall reliability of the data is 0.93, which is still excellent. Cronbach’s alpha was also in the Annex. calculated for each grade and these tables are available in the Annex. Table Reliability 5Reliability Table 5 of of the the VANEGRA VANEGRA English English assessment assessment (zero (zero scores scores removed) removed) average item-test item-test Item Obs Sign inter item alpha correlation correlation covariance Correct Letters Per 947 + 0.80 0.73 0.59 0.92 Minute/convert Initial sound correct 1283 + 0.67 0.64 0.64 0.93 21 Correct Letters Sounds Per 1186 + 0.84 0.58 0.58 0.92 Minute Correct Words Per Minute 794 + 0.90 0.57 0.57 0.91 Correct Non Words Per Minute 524 + 0.86 0.59 0.59 0.92 Oral Reading Fluency 769 + 0.91 0.57 0.57 0.92 Reading Comprehension total 1283 + 0.88 0.56 0.56 0.91 Listening Comprehension total 1283 + 0.70 0.63 0.63 0.93 Writing 1283 + 0.87 0.57 0.57 0.91 Test scale 0.59 0.93 The acquisition of reading skills is individual in the manner in which they are acquired. Some skills are The acquisition of reading skills is individual in the manner in which they are acquired. Some attained simultaneously in some children, while others proceed in a more linear fashion. However, there are skills are attained simultaneously in some children, while others proceed in a more linear some average progressions that can be observed across the sample. Figure 1 indicates letter level, word fashion. However, there are some average progressions that can be observed across the sample. Figure 1 indicates letter level, word level, reading fluency and comprehension attainment by 21 percentile in the VANEGRA English sample. The graph indicates that the various skills are acquired together, though in this sample, certain skills such as decoding unfamiliar words are Listening Comprehension total 1283 + 0.70 0.63 0.63 0.93 Writing 1283 + 0.87 0.57 0.57 0.91 Test scale 0.59 0.93 The acquisition of reading skills is individual in the manner in which they are acquired. Some skills are attained simultaneously in some children, while others proceed in a more linear fashion. However, level, reading fluencythere are some average and comprehension progressions attainment that can by percentile be observed in the VANEGRAacross Englishthe sample. sample. The Figure thatletter 1 indicates graph indicates level, word the various level, skills are reading acquired fluency together, and comprehension though attainment in this sample, certain by as skills such percentile in the VANEGRA decoding unfamiliar Englishat words are acquired sample. The graph a later stage indicates compared thatwords to familiar the various skills and letter are recognition. acquired together, though in this sample, certain skills such as decoding unfamiliar words are acquired at a later stage compared to familiar words and letter recognition. Figure 1 Letter Level, Word Level, Oral Reading Fluency and Reading Comprehension Performance Figure 1 Letter Level, Word Level, Oral Reading Fluency and Reading Comprehension Performance 120 1.2 100 1 Correct Letters Per Minute 80 0.8 Correct Words Per Axis Title 60 0.6 Minute Correct Non Words Per 40 0.4 Minute Oral Reading Fluency 20 0.2 0 0 Reading Comprehension 0.1 0.5 0.9 0.01 0.05 0.25 0.75 0.95 0.99 Percentiles 22 Figure 2 indicates Figure 2 indicates the progression the progression oral with offluency of oral reading reading readingfluency with and listening reading and comprehension. listening In this sample, comprehension. on average, listeningIn this sample, comprehension on average, is attained listening ahead of comprehension reading comprehension is attained and oral ahead of reading fluency. reading comprehension and oral reading fluency. Figure 2 Oral Reading Fluency, Reading and Listening Comprehension Figure 2 Oral Reading Fluency, Reading and Listening Comprehension 120 1.2 100 1 80 0.8 Oral Reading Fluency Axis Title 60 0.6 Reading 40 0.4 Comprehension Listening 20 0.2 Comprehension 0 0 0.1 0.5 0.9 0.01 0.05 0.25 0.75 0.95 0.99 Percentiles Figure 3 indicates the progression of reading comprehension, oral reading fluency and dictation skills. 22 It shows that dictation skills are acquired at a very similar rate to reading comprehension in this sample and shows the relationship between the stabilization of memory through writing and the skills required to comprehend text that is read. 60 0.6 Axis Reading 40 0.4 Comprehension Listening 20 0.2 Comprehension 0 0 0.1 0.5 0.9 0.01 0.05 0.25 0.75 0.95 0.99 Percentiles Figure 3 indicates the progression of reading comprehension, oral reading fluency and dictation skills. It shows that dictation skills are acquired at a very similar rate to reading comprehension in this sample Figure 3 indicates the progression of reading comprehension, oral reading fluency and dictation and shows the relationship between the stabilization of memory through writing and the skills required to skills. It shows that dictation skills are acquired at a very similar rate to reading comprehension comprehend text that is read. in this sample and shows the relationship between the stabilization of memory through writing and the skills required to comprehend text that is read. Figure 3 Oral Reading Fluency, Reading Comprehension and Dictation Scores Figure 3 Oral Reading Fluency, Reading Comprehension and Dictation Scores 120 1.2 100 1 80 0.8 Oral Reading Fluency Axis Title 60 0.6 Reading 40 0.4 Comprehension Dictation 20 0.2 0 0 0.1 0.5 0.9 0.01 0.05 0.25 0.75 0.95 0.99 Percentiles In summary, the VANEGRA English instrument has good reliatbility, even when zero-score cases (discussed in Chatper 3) are removed. The results of the survey indicate that the basic literacy skills are acquired 23 together and, although they are acquired at different rates and in different progressions amongst students, some average relationships can be observed across the sample. 23 3 Chapter 3: VANEGRA English Results Chapter 3: VANEGRA English Results Structure of the Assessment Structure As of the Assessment has been confirmed by scholars working to understand reading acquisition in multiple languages, As has and (Jimenez beenO’Shanahan confirmed Juan, 2008; Linan-Thompson by scholars and Vaughn, working to understand 2007; Abadzi, reading 2006; inSprenger- acquisition multiple languages, Charolles, (Jimenez 2004; and Chiappe O’Shanahan et al., Juan, 2002), being 2008; able to Linan-Thompson read and well requires a grasp of Vaughn, 2007; five basic skills inAbadzi, almost 2006; any Sprenger-Charolles, alphabetic 2004; language in which Chiappe print et al., 2002), can be decoded being able into sounds to read (National well Panel, Reading requires a grasp of 2000): five basic skills in almost any alphabetic language in which print can be decoded into sounds • phonemic awareness–focusing on, manipulating, breaking apart, and putting together sounds orally; (National Reading Panel, 2000): • phonemic • phonics–linking awareness –focusing written letters on, manipulating, to their sounds breaking and forming spelling apart, and putting together patterns; sounds orally; phonics • • fluency –linking –achieving written speed, lettersand accuracy, to their sounds expression and forming spelling patterns; in reading; • fluency–achieving speed, accuracy, and expression in reading; • vocabulary–knowing words (both oral and written) and their meaning; and • vocabulary–knowing words (both oral and written) and their meaning; and comprehension • • comprehension –understanding –understanding the concepts the concepts read orread or heard. heard. Though Though not not all all children children develop develop their reading their reading abilities abilities in the in the same waysame way or pace, or literature the pace, the literature shows that shows all that readers all readers progress progress through through a series a series of phases of phases or stages or stages –sometimes –sometimes simultaneously- simultaneously- in their reading in their reading development (RTI, 2010). process (RTI, 2010). development process Figure Figure 4 - Stages 4 - Stages of Reading Development ofReading Development Source: RTI, 2010. Source: RTI, 2010. Once children learn to apply the foundational reading skills with a certain level of reflex or automaticity, they Once children learn to apply the foundational reading skills with a certain level of reflex or can move beyond the task of decoding a text (Stage 1) to begin deriving its meaning (Stage 2). As children automaticity, they can move beyond the task of decoding a text (Stage 1) to begin deriving its learn sounds that link to form words, they can begin connecting those sounds to printed words and the idea meaning (Stage 2). As children learn sounds that link to form words, they can begin connecting behind those words. Then they can link words to form sentences, paragraphs, and stories. In other words, those sounds to printed words and the idea behind those words. Then they can link words to 24 25 form sentences, paragraphs, and stories. In other words, children transition from learning to children transition from learning to read to reading to learn (Stage 3 and beyond). Comprehension is the read to reading to learn (Stage 3 and beyond). Comprehension is the ultimate prize—the ultimate differenceprize—the between difference “readingbetween “reading it� and it� and “getting “getting it� (RTI), it� (RTI), 2010). The2010). The structure structure of the EGRA of the EGRA tool in tool in different international applications used this knowledge as a reference point different international applications used this knowledge as a reference point during the during the adaptation of the of contexts tool to local adaptation the tool (Figure 5). to local contexts (Figure 5). Figure 5 - Early Figure Grader 5 - Early Reading Grader ReadingAssessment Components Assessment Components Source:RTI, Source: RTI,2010. 2010. In the context of a linguistically diverse nation such as Vanuatu, students may not only be learning to read in In the context of a linguistically diverse nation such as Vanuatu, students may not only be the early grades: they may be doing so while acquiring a new language. This new language (English) may in learning to read in the early grades: they may be doing so while acquiring a new language. This fact new belanguage or fourth language. their third (English) may in fact linguistic The be challenges their third younglanguage. or fourth students are Thefacing should challenges linguistic be kept in mind youngwhen arethe interpreting students resultsshould facing of the VANEGRA be kept assessment. in mind when interpreting the results of the VANEGRA assessment. In particular, the VANEGRA English assessment focused on measuring these skills plus basic listening and writing skills through eight modules or sub-tests: (1) letter name knowledge; (2) identification of initial sounds In particular, the VANEGRA English assessment focused on measuring these skills plus basic in words; (3) letter sound knowledge; (4) familiar word reading; (5) invented word reading; (6) oral reading listening and writing skills through eight modules or sub-tests: (1) letter name knowledge; (2) fluency with comprehension; identification (7) listening of initial sounds comprehension; in words; and (8) (3) letter sound dictation (see knowledge; Annex (4) 2 for familiar a copy word of the reading; VANEGRA English (5) invented instrument). word reading;Table(6) oral shows 6 below reading these skills how fluency to each of the VANEGRA relate comprehension; with English (7) listening components, measures comprehension; and and indicators. (8) dictation (see Annex 2 for a copy of the VANEGRA English instrument). Table 6 below shows how these skills relate to each of the VANEGRA English components, measures and indicators.   26 25 Table 6 - VANEGRA Table English 6 - VANEGRA Instrument English Structure Instrument Structure and and Early Early Skills Skills Tested Tested Skill demonstrated by students’ Measure and Indicator Sub-test Early reading skill ability to: Provides the name of upper- and Letter name fluency in terms of 1. Letter name Letter recognition lower-case letters distributed in correct letter names identified per knowledge random order minute (CLPM) Segment words into 2 to 5 phonemes Phoneme segmentation as the 2. Identification of initial Phonemic number of sounds correctly identified Identify words with different sounds awareness beginning or ending phoneme Provides the sound of upper- and Letter name fluency in terms of 3. Letter sound Phonics lowercase letters distributed in correct letter sounds identified per knowledge random order minute (CLPM) Familiar word fluency in terms of Read simple and common one- and 4. Familiar word reading Word reading correct familiar words read per two-syllable words minute (CFWPM) Make grapheme-phoneme Invented word fluency in terms of 5. Invented word Alphabetic correspondences (GPCs) through the correct invented words read per reading principle reading of simple invented –i.e., minute (CUWPM) invented- words to test decoding skills Oral reading fluency in terms of Oral reading Read a text with little effort and at a correct words read per minute in a fluency sufficient rate narrative passage (CWCPM) 6. Oral reading fluency Respond correctly to different types Response to questions after reading with comprehension Reading of questions, including literal and a story as a percentage of correct comprehension inferential questions about the text answers they have read Respond correctly to different types Response to questions after hearing 7. Listening Listening of questions including literal and a story as a percentage of correct comprehension comprehension inferential questions about the text answers the enumerator reads to them Write, spell, and use grammar properly through a dictation exercise, Alphabetic Write, spell, and use grammar determined by the percentage of 8. Dictation principle properly through a dictation exercise overall early writing skills (spelling and basic conventions), weighted score. Note: Adapted Note: Adaptedby bythe theauthors, based authors, on RTI, based on 2009 RTI, and 2009Linan-Thompson, 2010. and Linan-Thompson, 2010. Administration of the VANEGRA English Instrument Administration of the VANEGRA English Instrument The VANEGRA assessment was administered via face-to-face interviews between an enumerator6 and a The VANEGRA assessment was administered via face-to-face interviews between an student. Each interview lasted 20 to 25 minutes from the onset of the test to completion of the student enumerator and a student. Each interview lasted 20 to 25 minutes from the onset of the test to 6 background questionnaire. In six of the eight sub-tests in the VANEGRA instrument, students had 60 completion of the student background questionnaire. In six of the eight sub-tests in the seconds VANEGRA to complete the sub-test instrument, in order students hadto assess 60 automaticity seconds in a given to complete the skill. sub-testTo be insuccessful order to readers, assess basic reading competencies have to be automatic. Fluency measures assess not only automaticity in a given skill. To be successful readers, basic reading competencies have to whether or not bea child knows something, automatic. but whether Fluency measures they have assess not internalized only whetherthe knowledge and can or not a child process knows the information something, but automatically (Linan-Thompson, 2007). Time-limitation allows proper comparison of fluency across slow 6 The term readers and enumerator used iswho fast readers in reference may to same register the purposely trained scores interviewers at different administer periods early 2009). grade reading of time (RTI, assessments. 27 6 The term enumerator is used in reference to purposely trained interviewers administer early grade reading assessments. 26 In VANEGRA, student scores in time-limited sub-tests were calculated as the number of correct items – i.e. letter names, letter sounds, or words - read per minute. If a student completes all of the words before the time expires, the time of completion is recorded and the number of items correctly read per minute is estimated on that time period.7 Selected sub-tests applied an “early-stop rule� to discontinue the administration of a sub-test if students were unable to correctly respond to any of the items in the first of ten lines (Sub-tests, 1,3,4,5, and 6a), or if their responses for the first five items were incorrect (Sub-test 2).8 In this situation, the enumerator was asked to mark the box that read “Check this box if the exercise was discontinued because the child has no correct answers in the first line� and to proceed to the next sub-test in the test (RTI, 2010).9 The justification to discontinue a sub-test has two reasons. First, the early stop rule helps avoid frustration among students whose dispirit may affect their performance in subsequent sub-tests. Second, the early stop rule is also an approximation of zero-scores throughout the test: as in most psychometric tests, it is assumed that students who fail the first initial items will fail the remainder of the test, especially if test items become progressively more difficult, as in the case of VANEGRA. Sub-tests 7 and 8 did not apply the “early stop rule� so results in these sub-tests relate to the total sample of students and report percentage of correct answers over the total number of items in the sub-test. For each sub-test in VANEGRA, results are presented first in terms of the percentage of zero-score students, as well as the average score tested. The reason for this is that the percentage of zero-score students represents students who showed no evidence of the skill tested whereas the average score represents the average rate of acquisition of a given skill. In VANEGRA English, across sub-tests and grades, the largest proportions of early-stop cases appeared in sub-test 5 (invented word reading), sub-test 6a (oral passage reading), and sub-test 4 (familiar word reading). The sub-test that showed the lowest percentage of early- stop cases was sub-test 1 (letter sounds knowledge) where the test had to be discontinued only for 7% of the students in the sample. 7 Correct Items Per Minute = (Total items read – Total items incorrect) / [(60 – Time remaining on stopwatch) / 60] (RTI, 2009) 8 The rule was established to avoid frustrating students who did not have the skill or did not understand the task of the sub-test (RTI, 2010). 9 See Annex 2 for a copy of the VANEGRA English instrument. 27 Figure 6– VANEGRA Figure Figure 6– English: 6– VANEGRA VANEGRA Zero-score English: English: students Zero-score Zero-score studentsas students as apercentage as a a percentage percentage of of sample of the the the sample sample as as a as a whole a whole whole Total Sample S1. Letter S1. Letter Names Names 26% 26% S2. Initial S2. Initial Sounds Sounds 35% 35% S3. Letter S3. Letter Sounds Sounds 7% 7% S4. Familiar S4. Word Reading Familiar Word Reading 38% 38% S5. Invented S5. Word Reading Invented Word Reading 59% 59% S6a. Oral S6a. Reading Passage Oral Reading Passage 40% 40% As expected, the assessment had to be discontinued for a larger percentage of Grade 1 students (see As expected, Figure the 4 below). assessment This had to notably number declined be discontinued for in Grades 2 a larger and percentage 3, yet more of Grade than a third of the 1 students students in (see Figure Grade 3 were below). 4unable This number to decode declined a single notably word, and in Grades one in ten 2 and was unable to 3, yet read more the first than a third 15 words of in the the students oral in Grade reading passage. 3 were Slightly unable more thanto decode one in tena single was word, unable and the to read onefirst five was in ten unable familiar toin read words the the first 15 words in the oral reading passage. Slightly more than one in ten was unable to read activity. the first five familiar words in the activity. Figure 7– VANEGRA Figure English: 7– VANEGRA Zero-score English: students Zero-score students as percentage as aapercentage the sample in sample in the per grade per grade Figure 7– VANEGRA English: Zero-score students as a percentage in the sample per grade By Grade 85% 85% 73% 73% 69% 69% 57% 59% 59% 57% 48% 48% 37% 37% 33% 33% 33% 33% 34% 34% 22% 22% 16% 16% 17% 17% 13% 13% 8% 10% 10% 8% 5% 5% 1% 1% S1. Letter S1. NamesS2. Letter Names Initial Sounds S2. Initial Sounds S3. S3. Letter Letter S4. Familiar S4. Familiar S5. Invented S5. Invented S6a. Oral S6a. Oral Sounds Sounds Word Reading Word Reading Word Word Reading Reading Passage Reading Reading Passage Grade 1 Grade 1 Grade 2 Grade 2 Grade 3 Grade 3 29 28 VANEGRA English Results per Sub-test VANEGRA English results show reading gains across the three grades tested. Across grades, Anglophone students appear to have a basic knowledge of letter names and sounds, which progressively consolidates for almost all of them at the end of Grade 3 as seen by the low proportion of Grade 3 students scoring zero in sub-test 1. However, the average rate of progress is slow which may be hindering the development of word-level reading skills, oral reading fluency and comprehension. Looking at differences in performance between boys and girls, VANEGRA English results showed evidence of gender differences in some sub-tests in favor of girls, though the size and significance of the effect varies per skill tested. Gender differences are generally not statistically significant for Grade 1 students, but the gender effect is more pronounced in later grades, and these differences are statistically significant for familiar word reading, narrative passage reading and writing (see Table 16 for regression output on grade and gender interactions and Figures 9, 10 and 11 for differences in performance by grade and gender). For each of the sub-tests below, average results are presented for the sample as a whole and per grade, as well as without the proportion of zero-score students. There are two reasons behind this decision. On the one hand, some researchers argue that in cases where there is a large presence of zero-score students, overall means tend to underestimate the true average score of the population. On the other hand, the use of average means without zero-score students tends to overestimate the true average score. Since one of the purposes of this survey is to inform policy decisions over the establishment of temporary reference standards of oral reading fluency in the country, we believe it is important to present both results in order to inform policy discussions and future decisions over where and how to establish adequate reference reading standards for Vanuatu. Sub-test 1 – Letter Name Knowledge10 The test of letter name knowledge is the most basic of assessments of student reading preparedness (and risk). Letter name knowledge is a consistent predictor of reading development for native speakers of English, French, and other alphabetic languages (Chiappe, Siegel, & Wade-Woolley, 2002). It has also proved to be a useful indicator for non-native speakers (Chiappe, 2006). In the sub-test of letter name knowledge, students are asked to provide the names (not the sounds) of all of the letters they could read, within a one-minute period. The full set of letters of the English alphabet was listed in random order, 10 letters to a row, for a total of 100 letters. Letters were selected based on the frequency with which they occur in English. Randomization was used to prevent students from reciting a memorized alphabet—that is, to test for actual automaticity of letter recognition and translation of print to sound. VANEGRA English students showed a positive performance on recognizing letter names, defined by the indicator “number of correct letter names identified per minute� (CLPM). Table 7 shows average scores in sub-test 1 by Grade and gender, and overall scores excluding zero-scoring students for those unable to recognize any letters. On average, students read an average of 19 letters per minute (26 letters attempted 10 Throughout this section, sub-test description is based on the Early Grade Reading Assessment Toolkit (RTI, 2009). 29 Letters were selected based on the frequency with which they occur in English. Randomization was used to prevent students from reciting a memorized alphabet—that is, to test for actual automaticity of letter recognition and translation of print to sound. VANEGRA English students showed a positive performance on recognizing letter names, defined by the indicator “number of correct letter names identified per minute� (CLPM). Table 7 shows average scores in sub-test 1 by Grade and gender, and overall scores excluding zero- scoring students for those unable to recognize any letters. On average, students read an and 19 letters read correctly). About a quarter of the students scored zero on this section. Zero-score average of 19 letters per minute (26 letters attempted and 19 letters read correctly). About a students reduce the average mean by 7 letter names correctly identified (from 26 down to 19). 32% of quarter of the students scored zero on this section. Zero-score students reduce the average students mean by read between 7 letter names1 and 20 letters correctly correctly, identified (fromand 33% 26 downread 19). 32%20 to between of and 50 letter readnames students between per 1 and The minute. 20 remaining 10% read letters correctly, more and 33% than 50 letters read between per minute 50 letterThe 20 andcorrectly. nameslargest perincrease minute. in letter The remaining name 10%takes knowledge more readplace than 50 between letters Grade per Grade 1 and minute correctly. 2 with Thedifference an average largest increase in names of 11 letter letter name9knowledge versus between Grade takes place 2 and between Grade Grade 3. Overall, 1 are girls and Grade more 2 with accurate an average readers of letterdifference names (20 of 11 letters letter names versus 9 between Grade 2 and Grade 3. Overall, girls are more accurate readers correctly identified) than boys (18 letter names) but this difference is not statistically significant across grades. of letter names (20 letters correctly identified) than boys (18 letter names) but this difference is not statistically significant across grades. Table 7- Sub-test Table 1 Letter 7- Sub-test Name 1 Letter NameKnowledge: Results Knowledge: Results by by grade grade and gender and gender 95% Confidence Subtest 1 - Number of correct letter names interval N Mean SD Min Max identified per minute (CLPM) Lower Upper bound Bound CLPM - overall 1,282 19 20 0 97 18 20 CLPM - minus zero score students 947 26 19 1 97 25 27 Grade CLPM - overall 427 9 13 0 89 8 10 Grade 1 CLPM - minus zero score students 221 17 14 1 89 15 18 CLPM - overall 427 20 20 0 92 18 21 Grade 2 CLPM - minus zero score students 331 25 19 1 92 23 27 CLPM - overall 428 29 20 0 97 27 31 Grade 3 CLPM - minus zero score students 395 32 19 1 97 30 34 Gender CLPM - overall 645 20 21 0 92 18 22 Girls CLPM - minus zero score students 479 27 20 1 92 25 29 CLPM - overall 637 18 19 0 97 16 19 Boys CLPM - minus zero score students 468 24 18 1 97 23 26 Sub-test 2 – Initial Sound Recognition InSub-test each ofSound 2 – Initial order to read, Recognition us must turn the letters we see into sounds, sounds into words, and words into In orderSuccessfully meaning. to read, each of us must managing turn the this process letters requires we the see to ability into sounds, work sounds in reverse; into that is, words, students and should words into meaning. Successfully managing this process requires the ability to work in reverse; also grasp that words are composed of individual sounds and understand the process of separating (and manipulating) words into sounds (Snow et al., 1998). The ability to identify sounds in words, to separate words 31 into sounds, and to manipulate those sounds is termed phonemic awareness, found to play an important role in reading acquisition and the number one predictor of success in reading, better than socioeconomic status, preschool attendance, or reading time in the home (Share, Jorm, Maclearn, & Matthews, 1984). Testing for and remediating this skill is thus important for later reading development. Thus far, the EGRA tool has piloted an assessment of phonemic awareness in two different ways: using phoneme segmentation and identification of onset and rime sounds (first and last sounds). Phoneme 30 Maclearn, & Matthews, 1984). Testing for and remediating this skill is thus important for later reading development. Thus far, the EGRA tool has piloted an assessment of phonemic awareness in two different ways: using phoneme segmentation and identification of onset and rime sounds (first and segmentation – i.e. the division of words into phonemes - is one of the most complex skills of phonological last sounds). Phoneme segmentation – i.e. the division of words into phonemes - is one of awareness the most and should be complex emphasized skills in the early of phonological grades and awareness (Linan-Thompson & Vaughn, in should be emphasized 2007). It is also the early grades one of (Linan-Thompson the most predictive of later Vaughn,skills. & learning 2007). It is Thus also far, one ofsegmentation phoneme the most predictive has proved of later difficult to learning administer. skills. The VANEGRA far, phoneme Thus instrument a simpler has segmentation selected task proved difficult – i.e. initial sound to administer.- to identification The assess VANEGRA instrument selected a simpler task – i.e. initial sound identification student’s ability in phoneme segmentation. A set of 10 English familiar words was selected from a list - to assess student’s of words in phoneme abilityused commonly segmentation. in children books, community of 10 A set life English and schoolfamiliar words texts books was selected available in Vanuatu. from a list of words commonly used in children books, community life and school texts Students were asked to identify the initial sound in each of the words. The enumerator read each word books available in Vanuatu. Students were asked to identify the initial sound in each of the aloud twice before words. asking the student The enumerator to identify read each word the sound aloud twice and recording before askingthe the answer as correct, student incorrect, to identify no answer in each the sound anditem in the sub-test. recording the answer as correct, incorrect, no answer in each item in the sub- test. Scores are defined as the average number of initial sounds identified from a list of 10 one- and two- syllable words Scores are included the exercise. in the defined as average numberStudent ofperformance initial soundsin this sub-test identified showed from a list one of 10ofone- the best improvements across grades and two-syllable wordswith students included in identifying up to Student the exercise. 7 initial sounds at the end performance of Grade in this 3. Boys sub-test showed and girls one of similarly performed improvements the beston across the identification grades of initial with students sounds. identifying Both boys up to and girls read 57 initial initial sounds sounds correctly outat 10,end ofthe andof49% Gradeof 3. themBoys and girls correctly read performed more than similarly 5 sounds on correctly. the identification of The differences initial sounds. Both boys and girls read 5 initial sounds correctly out of 10, and 49% between boys and girls in each grade are small (less than 1) and they are not statistically significant. of them correctly read more than 5 sounds correctly. The differences between boys and girls in each grade are small (less than 1) and they are not statistically significant. Table 8 - Sub-test 2 Initial Sound Recognition: Results by Grade and gender Sub-test 2 Initial Sound Recognition: Results by Grade and gender 95% Confidence interval N Mean SD Min Max Subtest 2 -Number of correct initial Lower Upper sounds identified bound Bound Overall 1,282 5 4 0 10 4 5 Minus zero score students 832 7 3 1 10 7 7 Grade Overall 427 3 4 0 10 2 3 Grade 1 Minus zero score students 185 6 3 1 10 6 6 Overall 427 5 4 0 10 5 5 Grade 2 Minus zero score students 286 7 3 1 10 7 8 Overall 428 7 4 0 10 6 7 Grade 3 Minus zero score students 361 8 2 1 10 8 8 Gender Overall 645 5 4 0 10 4 5 Girls Minus zero score students 418 7 3 1 10 7 7 Overall 637 5 4 0 10 4 5 Boys Minus zero score students 414 7 3 1 10 7 7 32 31 Sub-test 3 – Letter Sound Identification Knowledge of how letters correspond to sounds is another critical skill children must master to become successful readers. Letter-sound correspondences are typically taught through phonics-based approaches. In this sub-test, students were asked to provide the sounds (not the names) of as many letters they could identify within a one-minute period. The full set of letters in the English alphabet was listed in random order, 10 letter sounds to a row, for a total of 100 letter sounds. Scores in Sub-test 3 are defined as the number of correct letter sounds identified per minute (CSPM). On average, students correctly read 42 letter sounds per minute, with 47 letter sounds attempted. Excluding zero-score students, the average is 46 correct letter sounds per minute (CSPM). Letter sound knowledge showed important improvements across grades, from about 20 correct letter sounds per minute in Grade 1 to 64 correct letter sounds per minute at the end of Grade 3. Girls performed slightly better with an average of 45 CSPM versus 40 CSPM read by boys.   Table 9 – Sub-test Table 3 Letter 8 – Sub-test SoundIdentification: Sound 3 Letter Results Identification: Results byby Grade Grade and and gender. gender. 95% Confidence interval N Mean SD Min Max Subtest 3 - Number of correct letter sounds Lower Upper identified per minute (CSPM) bound Bound CSPM - Overall 1,282 42 31 0 146 41 44 CSPM - Minus zero score students 1,186 46 29 1 146 44 48 Grade CSPM - Overall 427 22 21 0 98 20 24 Grade 1 CSPM - Minus zero score students 356 26 21 1 98 24 28 CSPM - Overall 427 43 27 0 132 41 46 Grade 2 CSPM - Minus zero score students 407 45 26 1 132 43 48 CSPM - Overall 428 64 28 0 146 61 67 Grade 3 CSPM - Minus zero score students 423 65 27 3 146 62 68 Gender CSPM - Overall 645 45 31 0 146 42 47 Girls CSPM - Minus zero score students 607 47 30 1 146 45 50 CSPM - Overall 637 40 30 0 131 37 42 Boys CSPM - Minus zero score students 579 44 29 1 131 42 47 Sub-test 4 – Familiar Word Reading Children who are able to read words that are familiar to them often do that by automatic recognition. Automated word recognition in reading allows a beginning reader to ‘read’ a familiar word not by its letters, but as a whole. For this assessment, high-frequency familiar words were selected from early grade reading materials, storybooks, and school visits to primary schools to ask teachers in Grades 1 to 3 about the vocabulary used during their lessons. Words were arranged horizontally, in good separation from each other, written in a familiar (lower case) font, comprising 10 rows, five familiar words per line. 32 Scores in sub-test 4 are defined as the number of correct familiar words read per minute (CFWPM). VANEGRA English scores in sub-test 4 showed a low automaticity in word reading, Sub-test 4 – Familiar Word Reading Children who are able to read words that are familiar to them often do that by automatic recognition. Automated word recognition in reading allows a beginning reader to ‘read’ a familiar word not by its letters, but as a whole. For this assessment, high-frequency familiar words were selected from early grade reading materials, storybooks, and school visits to primary schools to ask teachers in Grades 1 to 3 about the vocabulary used during their lessons. Words were arranged horizontally, in good separation from each other, written in a familiar (lower case) font, comprising 10 rows, five familiar words per line. Scores in sub-test 4 are defined as the number of correct familiar words read per minute (CFWPM). VANEGRA English scores in sub-test 4 showed a low automaticity in word reading, an ability closely associated to word reading in the oral reading passage and ultimately, comprehension. On average, students were able to read about 11 familiar words correctly in a minute. Excluding zero-scores, the average score increases to 19 familiar words. At the end of Grade 1, students are able to read about 2 familiar words per minute. This average increases to 10 and 23 at the end of Grade 2 and Grade 3, respectively. On average, girls read more familiar words per minute than boys. This gender difference is statistically significant only for Grade 2 and Grade 3.   Table 10 – Sub-test Table 44 9 – Sub-test Familiar Word Familiar Reading: Word Reading: Results Results by Grade by Grade and Gender and Gender 95% Confidence Subtest 4 - Number of correct familiar words interval N Mean SD Min Max read per minute (CFWPM) Lower Upper bound Bound CFWPM – Overall 1,281 11 17 0 136 10 12 CFWPM - Minus zero score students 794 19 19 1 136 18 20 Grade CFWPM – Overall 427 2 6 0 107 1 3 Grade 1 CFWPM - Minus zero score students 134 6 9 1 107 5 8 CFWPM – Overall 426 10 13 0 83 9 11 Grade 2 CFWPM - Minus zero score students 287 15 14 1 83 13 16 CFWPM – Overall 428 23 22 0 136 21 25 Grade 3 CFWPM - Minus zero score students 373 27 21 1 136 25 29 Gender CFWPM – Overall 645 13 18 0 115 11 14 Girls CFWPM - Minus zero score students 415 20 20 1 115 18 22 CFWPM – Overall 636 10 16 0 136 9 11 Boys CFWPM - Minus zero score students 379 18 18 1 136 16 19 Sub-test 5 – Invented Word Reading Invented word reading is a measure of decoding ability. Many children in the early grades learn to memorize or recognize by sight a broad range of words. Children’s decoding skills are often assessed using reading lists of invented words that cannot typically be read by sight 33 recognition. This allows for a purer measure of word recognition and decoding skills than does reading comprehension paragraphs, as children are unable to guess the next word from the Sub-test 5 – Invented Word Reading Invented word reading is a measure of decoding ability. Many children in the early grades learn to memorize or recognize by sight a broad range of words. Children’s decoding skills are often assessed using reading lists of invented words that cannot typically be read by sight recognition. This allows for a purer measure of word recognition and decoding skills than does reading comprehension paragraphs, as children are unable to guess the next word from the context. This sub-test included 50 one- and two-syllable invented words, five per row, with the vowel-consonant patterns of letters typical in the English language. Scores in Sub-test 5 are calculated as the number of correct invented words read per minute (CIWPM). VANEGRA English results in this sub-test showed a lower performance than the average scores in familiar word reading. Students were able to correctly read an average of 6 invented words per minute, with 11 words attempted and 6 correctly read on average. Excluding zero-score students, the average score increases to 16 invented words read correctly. At the end of Grade 1, students were able to read about 8 invented words correctly per minute. This average increases to 17 and 24 at the end of Grade 2 and Grade 3, respectively. There were no statistically significant differences between boys and girl’s decoding skills. Table 11 – Sub-test Table 5 Invented 10 – Sub-test Word 5 Invented Word Reading: Results Reading: Results by by Grade Grade and Gender and Gender 95% Confidence Subtest 5 - Number of correct invented interval N Mean SD Min Max words read per minute (CIWPM) Lower Upper bound Bound CIWPM – Overall 1,282 6 11 0 75 5 7 CIWPM - Minus zero score students 524 16 13 1 75 14 17 Grade CIWPM – Overall 427 1 5 0 75 1 2 Grade 1 CIWPM - Minus zero score students 65 8 9 1 75 5 10 CIWPM – Overall 427 6 10 0 51 5 6 Grade 2 CIWPM - Minus zero score students 175 14 11 1 51 12 15 CIWPM – Overall 428 12 14 0 72 11 13 Grade 3 CIWPM - Minus zero score students 284 19 14 1 72 17 21 Gender CIWPM – Overall 645 7 12 0 72 6 7 Girls CIWPM - Minus zero score students 273 16 14 1 72 15 18 CIWPM – Overall 637 6 11 0 75 5 6 Boys CIWPM - Minus zero score students 251 15 13 1 75 13 16 34 Sub-test 6a – Oral Passage Reading Oral reading fluency is a measure of overall reading competence: the ability to translate letters into sounds, unify sounds into words, process connections, relate text to meaning, and make Sub-test 6a – Oral Passage Reading Oral reading fluency is a measure of overall reading competence: the ability to translate letters into sounds, unify sounds into words, process connections, relate text to meaning, and make inferences to fill in missing information (Hasbrouck & Tindal, 2006). As skilled readers translate text into spoken language, they combine these tasks in a seemingly effortless manner. Because oral reading fluency captures this complex process, it can be used to characterize overall reading skill. Poor performance on a reading comprehension tool would suggest that the student had trouble with decoding, with reading fluently enough to comprehend, or with vocabulary. Sub-test 6a produced a 60-word narrative passage from children’s reading materials. The narrative passage began where the characters are introduced, a middle section containing some dilemma, and an ending section with an action resolving the dilemma. The passage provided the basis for the comprehension questions presented in sub-test 6b. Scores in sub-test 6a are calculated as the number of correct words read in the 60-word oral reading passage. On average, students were able to read 15 correct words read in a connected text per minute (CWCPM). Excluding zero-score students, the average score increases to 26 end of Grade CWCPM. At the1,end students of Gradewere able towere 1, students 12 correct readable words to read 12 of words correct the oral reading of the passage oral reading in a passage minute. This in a minute. average This averageincreases to28 increasesto and 42 28and 42 at at the endof the end ofGrade Grade 22andand Grade Grade 3, respectively. 3, respectively. Girls Girls outperformed outperformed boys in boys in reading a narrative readingpassage, a narrative passage, and this and difference this difference is statistically is statistically significant in Grades 2 significant in Grades 2 and 3 (a discussion of the gender and 3 (a discussion of the gender effect is produced below). effect is produced below). Table 12 – Sub-test Table 6a 6a 11 – Sub-test Oral Passage Oral Reading: Passage Reading: Results Results by Grade by Grade and Gender and Gender 95% Confidence Subtest 6a - Number of words correct words interval N Mean SD Min Max read in a connected text per minute (CWCPM) Lower Upper bound Bound CWCPM - Overall 1,282 15 22 0 161 14 17 CWCPM - Minus zero score students 769 26 24 1 161 25 28 Grade CWCPM - Overall 427 3 8 0 136 2 4 Grade 1 CWCPM - Minus zero score students 116 10 12 1 136 8 12 CWCPM - Overall 427 13 17 0 107 12 15 Grade 2 CWCPM - Minus zero score students 269 21 18 1 107 19 23 CWCPM - Overall 428 32 27 0 161 29 34 Grade 3 CWCPM - Minus zero score students 384 36 26 1 161 33 39 Gender CWCPM - Overall 645 17 24 0 161 15 19 Girls CWCPM - Minus zero score students 402 28 25 1 161 26 31 CWCPM - Overall 637 14 21 0 136 12 15 Boys CWCPM - Minus zero score students 367 25 23 1 136 22 27 Sub-test 6b – Reading Comprehension 35 Average reading comprehension levels in Sub-test 6b are largely explained by the poor performance in the oral reading passage. Without sufficient skills to read into the text, students Sub-test 6b – Reading Comprehension Average reading comprehension levels in Sub-test 6b are largely explained by the poor performance in the oral reading passage. Without sufficient skills to read into the text, students focus on reading words one-by-one, sometimes letter-by-letter. By the time they reach the end of the text, students have already forgotten what they read first. Scores in Sub-test 6b are calculated as the percentage of correct answers in the Sub-test -5 questions in total. The number of questions a student received depended on the number of words read in sub-test 6a, so that students had to respond only to questions related to the segment of the text they read. The early-stop marker was placed at 15 words – i.e. first row in the text - so that those unable to correctly read any of the first 15 words received a zero-score in this task. Students that read the first 15 words with at least one word read correctly were allowed to continue with the exercise. Those reading between 15 and 21 words received 1 question. Those reading between 21 and 31 received 2 questions, and those reading between 31 and 45 received 3 reading comprehension questions. Students reading between 45 and 51 received 4 questions and those reading above 51 words were asked all 5 reading comprehension questions. On average, students showed an average reading comprehension of 19% of the text they read. Excluding zero-score students (623 students), the average scores increase to 49% which suggests that students who completed the exercise, on average, understood about half of the text they read. At the end of Grade 1, the average student understands about 2% of the text. In Grade 2, the average increases considerably to 16% and by Grade 3, the average student understands about 42% of the text, demonstrated by correctly answering the questions received. Boys scored an average of 18%; they attempted to answer an average of 1 question out of 5. Girls scored an average of 21%; they attempted to answer 2 questions on average. Overall, boys and girls in Grade 1 have a similar level of reading comprehension. However, girls outperform boys in Grades 2 and 3 with differences that are statistically significant (seeTable 13). A closer look at differences in comprehension between students who were asked to answer questions based on the oral reading passage show important differences in the share of students in each category and the average comprehension based on questions asked. Table 13 shows the distribution of correct answers depending on the number of questions asked. 36 reading comprehension questions. Students reading between 45 and 51 received 4 questions and those reading above 51 words were asked all 5 reading comprehension questions. Table 13 – Sub-test Table 6b6b 12 – Sub-test Reading ReadingComprehension: Results Comprehension: Results by Grade by Grade and Gender and Gender 95% Confidence Subtest 6b - Percentage of overall interval reading comprehension in a N Mean SD Min Max Lower Upper connected text bound Bound Percentage of correct 1,282 19 30 0 100 18 21 answers - Overall Percentage of correct answers - Minus zero 529 49 28 20 100 47 51 score students Grade Percentage of correct 427 2 9 0 100 1 3 answers - Overall Grade 1 Percentage of correct answers - Minus zero 33 29 17 20 100 23 35 score students Percentage of correct 427 16 25 0 100 13 18 answers - Overall Grade 2 Percentage of correct answers - Minus zero 169 40 25 20 100 36 44 score students Percentage of correct 428 42 34 0 100 38 45 answers - Overall Grade 3 Percentage of correct answers - Minus zero 327 56 27 20 100 53 59 score students Gender Percentage of correct 645 21 31 0 100 18 23 answers - Overall Girls Percentage of correct answers - Minus zero 276 51 29 20 100 47 54 score students Percentage of correct 637 18 28 0 100 16 20 answers - Overall Boys Percentage of correct answers - Minus zero 253 47 27 20 100 44 50 score students On average, students showed an average reading comprehension of 19% of the text they read. Excluding zero-score students (623 students), the average scores increase to 49% which suggests that students who completed the exercise, on average, understood about half of the text they read. At the end of Grade 1, the average student understands about 2% of the text. In Grade 2, the average increases considerably to 16% and by Grade 3, the average student understands about 42% of the text, demonstrated by correctly answering the questions received. Boys scored an average of 18%; they attempted to answer an average of 1 question out of 5. Girls scored an 38 37 average of 21%; they attempted to answer 2 questions on average. Overall, boys and girls in Grade 1 have a similar level of reading comprehension. However, girls outperform boys in Grades 2 and 3 with differences that are statistically significant (seeTable 12). A closer look at differences in comprehension between students who were asked to answer questions based on the oral reading passage show important differences in the share of students in each category and the average comprehension based on questions asked. Table 12 shows the distribution of correct answers depending on the number of questions asked. Table 14 - Percentage of Correct Answers in Sub-test 6b Table 13 - Percentage of Correct Answers in Sub-test 6b # questions asked # questions correct 0 1 2 3 4 5 Total 0 623 87 24 16 0 3 753 100% 68% 14% 11% 0% 2% 59% 1 41 74 38 8 6 167 32% 44% 26% 11% 4% 13% 2 70 58 12 5 145 42% 39% 17% 4% 11% 3 37 23 18 78 25% 32% 13% 6% 4 28 40 68 39% 28% 5% 5 71 71 50% 6% Total 623 128 168 149 71 143 1,282 Note: Percentage values reflect column percentages. Note: Percentage values reflect column percentages. As seen in column “0�, 623 students were not given any questions – zero questions asked – As seen in column “0�, 623 students were not given any questions – zero questions asked – because they because they could not read correctly any of the first 15 words in the passage. Of the 128 could not read correctly any of the first 15 words in the passage. Of the 128 students that correctly read students that correctly read between 1 – 15 words in the first segment of the oral reading passage,132% between words – 15(41 in the first students) were segment able to the oral the ofanswer reading passage, question 32% (41 correctly. students) On average, were the able more to fluent the answer correctly. On question students in reading areaverage, the more – i.e. those that fluent in reading received 3 or students - the that are – i.e. those more questions received better their 3 or more questions - the better their comprehension, as they are able to provide comprehension, as they are able to provide correct answers to most, if not all, the questions correct answers to most, received. if not all, theThe proportion questions of students received. in the 0 The proportion ofand 1 correct students in theare less1 0 and than 15% correct for are columns less than 15%4 and for 5. Among columns the 4 and most 5. Among fluent (column the most 5), fluent 78% are (column 5),able 78%to understand are 80% or80% able to understand more or of theof text more thethey text (28% readread they with (28% 4/5 with correct 4/5 answers correct answers and and 50% 50% with with correct answersto correct answers toall all questions). questions). Sub-test 7 – Listening Comprehension Sub-test – Listening 7of Assessment Comprehension listening comprehension does not involve any reading from the student but involves the processing of oral language information only. Testing of listening comprehension Assessment of listening comprehension does not involve any reading from the student but involves the separately from reading comprehension is important due to the different ways in which processing of oral language information only. Testing of listening comprehension separately from reading learners approach, process, and respond to text. More importantly, listening comprehension is comprehension is important due to the different ways in which learners approach, process, and respond 39text. More importantly, listening comprehension is an important contributor to reading comprehension, to which tends to increase with reading acquisition. The narrative passage in VANEGRA’s Sub-test 7 was about 30 words long and narrated an activity or event familiar to Ni-Vanuatu children. Students then responded to oral comprehension questions – 5 in total. Scores in sub-test 7 are calculated as the number of correct answers in the sub-test. On average, students in the sample responded to 2 of the 5 questions correctly (Table 16). Excluding zero-score students, the average student correctly answered 3 questions. Students in Grade 3 performed the best with an average of 3 correct answers (60% average comprehension), followed by Grade 2 students with 2 correct answers 38 The narrative passage in VANEGRA’s Sub-test 7 was about 30 words long and narrated an activity or event familiar to Ni-Vanuatu children. Students then responded to oral comprehension questions – 5 in total. Scores in sub-test 7 are calculated as the number of correct answers in the sub-test. On average, students in the sample responded to 2 of the 5 questions correctly (Table 15). Excluding zero-score students, the average student correctly answered 3 questions. Students in Grade 3 performed the best with an average of 3 correct answers (60% average comprehension), followed by Grade 2 students with 2 correct answers (40% average comprehension), and then Grade 1 students with only 1 correct answer (20% (40% average comprehension), and then Grade 1 students with only 1 correct answer (20% average average comprehension). There were no statistically significant differences in the performance comprehension). There were no statistically significant differences in the performance of boys and girls on of boys and girls on listening comprehension. listening comprehension. Table 15 - Sub-test Table 7 Listening 14- Sub-test 7 ListeningComprehension: Results Comprehension: Results by Grade by Grade and Gender and Gender 95% Confidence Subtest 7 - Number of listening interval comprehension questions correctly N Mean SD Min Max Lower Upper answered bound Bound Overall 1,282 2 2 0 5 2 2 Minus zero score students 925 3 1 1 5 3 3 Grade Overall 427 1 1 0 5 1 1 Grade 1 Minus zero score students 224 2 1 1 5 2 2 Overall 427 2 2 0 5 2 2 Grade 2 Minus zero score students 326 3 1 1 5 3 3 Overall 428 3 2 0 5 3 3 Grade 3 Minus zero score students 375 4 1 1 5 3 4 Gender Overall 645 2 2 0 5 2 2 Girls Minus zero score students 455 3 1 1 5 3 3 Overall 637 2 2 0 5 2 2 Boys Minus zero score students 470 3 1 1 5 3 3 Sub-test 8 – Dictation Sub-test 8 – Dictation assessmentis Dictation assessment Dictation isfrequently frequentlyused usedbybyteachers teachers to to test both test bothoral comprehension oral comprehension and and writing skills. writing Students’ skills. hear sounds ability to ability Students’ and to hear correctly sounds andwrite the letters correctly and the write words corresponding letters and words to corresponding the sounds they to the hear sounds theytheir demonstrates hear with thetheir demonstrates success success alphabetic with the principle. alphabetic The dictation principle. sentence in The dictation the VANEGRA sentence English was the in 10 VANEGRA words long (“Sit English on the matwas and 10 words have some long food (“ Sit water� on the). mat and and have Students some received food and a weighted score capturing the accuracy for vowel and consonant sounds, spelling, spacing and direction of text, 40 capitalization, and punctuation. In addition, we estimated the number of letters and full words written, the percentage of those that were correctly written, and the number of pictograms used by children to represent the sentence given.11 Scores in Sub-test 8 are calculated using weights to create a variable with a maximum score of 100%.12 For easier interpretation of writing scores, we converted students’ raw dictation scores into a weighted average of their performance on each question. Each word for the spelling component received 10 points if 11 A pictogram is a pictorial representation of words used in Vanuatu to “smooth� student’s transition into alphabetic-based writing. Although not a standardized, there is a basic set of pictograms commonly used to represent the most common words and verbs used in the country. Even though pictograms are mostly used in Francophone schools in the teaching of reading and writing in Grades 1 and 2; the decision was made to include this option in English assessment as pilot data revealed some Anglophone schools also rely on this practice. 39 answered correctly and 5 if the response was partially correct. Spacing and capitalization received 3 points each, while the correct direction and use of the full stop received 2 points each for a total of 100 points possible for the entire section. On average, students wrote 2 out of 8 items correctly, with a weighted score of 25 out of 100. Excluding students unable to write anything, the average score increases to 3 out of 8 correct items. As in Sub-test 1 and 7, student performance in dictation was positive in terms of scores and overall progression across grades. Students in Grade 3 performed the best with an average score of 4 out of 8 items (49% weighted score), followed by Grade 2 students with an average score 2 out of 8 items (24% weighted score) and Grade 1 students with an average of 1 out of 8 items (6% weighted score). Out of the 34 letters in the sentence, students were able to write an average of 13 letters, 12 of which were written correctly. Out of the 10 words in the sentence, students wrote an average of 3 full words, 2 of which were spelled correctly, and about 1 word phonetically. The proportion of students using pictograms was very low, even for Grade 1 students. Table 16 shows evidence of progression across grades in all of the items estimated: the number of letters and full words increases as students go from Grade 1 (4 correct letters and no correct full words written) to Grade 3 (23 correct letters and 5 correct full words). Girls outperformed boys in dictation, and this difference is statistically significant in grades 2 and 3. Boys wrote an average of 12 letters, almost all of them correctly. Girls wrote an average of 13 letters, almost all of them correct. Boys wrote an average of 3 full words, 2 correctly written. Girls wrote an average of 4 full words, with 3 correctly written. More boys wrote words phonetically but both boys and girls wrote less than 1 word phonetically on average. Boys and girls on average wrote less than 1 word as pictogram. 16 –Sub-test TableTable 88 15 –Sub-test Dictation: Results Dictation: Results by by Grade Grade andand Gender Gender 95% Confidence Subtest 8 –Score for overall early writing interval N Mean SD Min Max skills (spelling and basic conventions) Lower Upper bound Bound Overall 1,282 2 2 0 8 2 2 Minus zero score students 918 3 2 1 8 3 3 Grade Overall 427 1 1 0 7 1 1 Grade 1 Minus zero score students 216 1 1 1 7 1 2 Overall 427 2 2 0 8 2 2 Grade 2 Minus zero score students 317 3 2 1 8 3 3 Overall 428 4 2 0 8 4 4 Grade 3 Minus zero score students 385 4 2 1 8 4 4 Gender Overall 645 2 2 0 8 2 2 Girls Minus zero score students 469 3 2 1 8 3 3 Overall 637 2 2 0 8 2 2 Boys Minus zero score students 449 3 2 1 8 3 3 12 Due to similarities in the components used in the VANEGRA instrument, the weights for this score follow the score used in the Guyana EGRA test. Table 16 –Sub-test 8 Dictation: Number of letters and full words (total and correct) written by grade 40 Grade 1 Grade 2 Grade 3 Total Mean SD Mean SD Mean SD Mean SD Number of letters written 4 7 12 13 23 12 13 13 Minus zero score students 317 3 2 1 8 3 3 Overall 428 4 2 0 8 4 4 Grade 3 Minus zero score students 385 4 2 1 8 4 4 Gender Overall 645 2 2 0 8 2 2 Girls Minus zero score students 469 3 2 1 8 3 3 Overall 637 2 2 0 8 2 2 Boys Minus zero score students 449 3 2 1 8 3 3 Table 17 –Sub-test 8 Dictation: Number of letters and full words (total and correct) written by grade Table 16 –Sub-test 8 Dictation: Number of letters and full words (total and correct) written by grade Grade 1 Grade 2 Grade 3 Total Mean SD Mean SD Mean SD Mean SD Number of letters written 4 7 12 13 23 12 13 13 Number of correct letters written 4 7 12 13 23 12 13 13 Number of full words 1 2 3 4 7 4 3 4 Number of full correct words written 0 1 2 3 5 4 4 3 Number of words written phonetically 0 1 1 1 1 1 1 1 Number of words written as a pictogram 0 1 0 1 0 1 0 1 Boys Boyswrote wrote average an an of 12 average letters, of almost 12 letters, all of them almost correctly. all of Girls wrote an them correctly. average Girls wroteof an 13 letters, average almost of 13 all of them letters, correct. almost Boys all wrote correct. of them an average of 3 wrote Boys full words, 2 correctly an average of 3 full Girls written. write words, 2 an average written. correctly of 4 full Girls with words, write an average 3 correctly of More written. 4 full wrote with words, boys words3 correctly written. phonetically, More but both boys and boys wrote girls wrote words less than 1phonetically, but on word phonetically both boys and average. girls Boys and wrote less girls on than wrote average 1 word phonetically than 1 word on less average. Boys and as pictogram. girls on average wrote less than 1 word as pictogram. For the writing components assessed in the sub-test, the rate of student non-response is quite high, but the rate decreases as students progress from grade 1 to later grades. 28% of students in the sample (342 students) did not write anything at all, most of them in Grade 1. Excluding students who produced a blank For the writing components assessed in the sub-test, the rate of student non-response is quite dictation section, 50% correctly used capitalization, 49% used spacing correctly, 95% wrote in the right high, but the rate decreases as students progress from grade 1 to later grades. 28% of students direction, and 9% correctly used the full stop. in the sample (342 students) did not write anything at all, most of them in Grade 1. Excluding students 42 who produced a blank dictation section, 50% correctly used capitalization, 49% used spacing correctly, 95% wrote in the right direction, and 9% correctly used the full stop. Table 18 –Sub-test 8 Dictation: Number of letters and full words (total and correct) written by gender Table 17 –Sub-test 8 Dictation: Number of letters and full words (total and correct) written by gender Boys Girls Mean SD Mean SD Number of letters written 12 13 13 13 Number of correct letters written 12 13 13 13 Number of full words 3 4 4 4 Number of full correct words written 2 3 3 3 Number of words written phonetically 1 1 1 1 Number of words written as a pictogram 0 1 0 1 While most students acquired an understanding of the orientation to write as early as Grade 1 (92% correct While most students acquired an understanding of the orientation to write as early as Grade 1 answer), other skills such as spacing and capitalization tend to consolidate as late as Grade 3. The use of (92% correct answer), other skills such as spacing and capitalization tend to consolidate as late the Grade as full stop, 3. the other on The use hand, appeared of the not on full stop, to develop in most the other students hand, acrossnot appeared grades (Figures 8 in and to develop 9).13 most students across grades (Figures 8 and 9). 13 Figure 8– Distribution of student responses to the writing items (total sample) 13 Anwers During the pilot of the instrument, the team observed to Writing that enumerators Items were not dictating the final point to students, as it is often the way teachers behave during dictation lessons. NR Incorrect Partial Correct 41 9% Number of full correct words written 2 3 3 3 Number of words written phonetically 1 1 1 1 Number of words written as a pictogram 0 1 0 1 While most students acquired an understanding of the orientation to write as early as Grade 1 (92% correct answer), other skills such as spacing and capitalization tend to consolidate as late as Grade 3. The use of the full stop, on the other hand, appeared not to develop in most students across grades (Figures 8 and 9). 13 Figure 8 – Distribution of student responses to the writing items (total sample) Figure 8– Distribution of student responses to the writing items (total sample) Anwers to Writing Items NR Incorrect Partial Correct 9% 49% 50% 35% 95% 14% 17% 4% 56% 33% 33% 2% 3% Direction Spacing Capitalization Full Stop 9 – Distribution FigureFigure of student 9– Distribution responses of student to responses to the the writing writing items items (per (per grade) grade) Correct Writing Items by Grade Direction Spacing 13 During the pilot of the instrument, the team observed Capitalization Full Stop that enumerators were not dictating the final point to students, as it is often the way teachers behave during dictation lessons. 43 95% 97% 92% 72% 73% 46% 46% 14% 18% 14% 8% 3% Grade 1 Grade 2 Grade 3 For the spelling component, students were marked based on their spelling of “have�, “some�, “food� and For the spelling component, students were marked based on their spelling of “have�, “some�, “water�. Non-response is also high for this component, with the rate decreasing as students progress from “food� and “water�. Non-response is also high for this component, with the rate decreasing as Grade 1 to later grades. Excluding students who left every dictation question blank, 20% correctly spelled students progress from Grade 1 to later grades. Excluding students who left every dictation the word “have�, 30% correctly spelled the word “some�, 39% correctly spelled the word “food� and 19% question blank, 20% correctly spelled the word “have�, 30% correctly spelled the word “some�, correctly spelled the 39% correctly word “water�. spelled Performance the word “food� in spelling and 19% showed steady itemscorrectly progression spelled the wordacross grades, “water�. with gains the largestin Performance observed spelling at the items end ofsteady showed Grade 3 (Figures 10 and progression 11).grades, with the largest gains across observed at the end of Grade 3 (Figures 10 and 11). Figure 10– Distribution of student responses to spelling items (total sample) 42 Spelling For the spelling component, students were marked based on their spelling of “have�, “some�, “food� and “water�. Non-response is also high for this component, with the rate decreasing as students progress from Grade 1 to later grades. Excluding students who left every dictation question blank, 20% correctly spelled the word “have�, 30% correctly spelled the word “some�, 39% correctly spelled the word “food� and 19% correctly spelled the word “water�. Performance in spelling items showed steady progression across grades, with the largest gains observed at the end of Grade 3 (Figures 10 and 11). Figure 10 –10– Figure Distribution student ofof Distribution responses student responses to spelling to spelling items items (total(total sample) sample) Spelling NR Incorrect Partial Correct 20% 19% 30% 39% 15% 16% 8% 13% 10% 5% 5% 4% 57% 51% 52% 54% "have" "some" "food" "water" 44 Figure 11– Distribution of student responses to the writing items (per grade) Figure 11– Distribution of student responses to the writing items (per grade) Correct Items in Spelling Grade 1 Grade 2 Grade 3 63% 52% 32% 32% 33% 24% 17% 14% 7% 4% 3% 1% "have" "some" "food" "water" Differences in Performance by Grade and Gender As previously stated, VANEGRA English results showed evidence of gender differences in some sub-tests in favor of girls, though the size and significance of the effect varies per skill tested. In order to identify the skills and the grades where gender differences take place, we explored performance differences by gender for each grade and included an interaction term between gender and grade to identify if the variation in significance observed across skills also varied across grades. We observed that differences in performance by gender start to appear in Grades 2 and 3 at statistically significant levels for sub-test 5 (familiar word reading), sub-test 6a (oral reading passage) and sub-test 8 (dictation). These differences suggest that although boys 43 and girls performance at the end of Grade 1 is similar, girls step into word-level reading faster than boys, as observed in the how boy’s performance lag as grade increases. Figures 9, 10, and Differences in Performance by Grade and Gender As previously stated, VANEGRA English results showed evidence of gender differences in some sub-tests in favor of girls, though the size and significance of the effect varies per skill tested. In order to identify the skills and the grades where gender differences take place, we explored performance differences by gender for each grade and included an interaction term between gender and grade to identify if the variation in significance observed across skills also varied across grades. We observed that differences in performance by gender start to appear in Grades 2 and 3 at statistically significant levels for sub-test 5 (familiar word reading), sub-test 6a (oral reading passage) and sub-test 8 (dictation). These differences suggest that although boys and girls performance at the end of Grade 1 is similar, girls step into word-level reading faster than boys, as observed in the how boy’s performance lag as grade increases. Figures 9, 10, and 11 below show gender differences in the three sub-tests where statistical significance was observed; table 17 shows the regression output. Familiar Word Reading Girls read more familiar words per minute than boys, and this difference is statistically significant in later grades. Girls read an average of 13 familiar words per minute, they attempted to read 18 and they correctly identified 12. Boys read an average of 10 words per minute, they attempted to read 16 and correctly read 9. The performance of boys and girls is comparable in grade 1. In Grade 2, girls read an average of 3 additional familiar words and in Grade 3 girls read an additional 5 words per minute on average. The differences in Grades 2 and 3 are statistically significant.   Figure 12– Differences in performance between boys and girls by grade, measured by the Figure 12– Differences in performance between boys and girls by grade, measured by the number of correct number of correct familiar words read per minute (CFWPM) familiar words read per minute (CFWPM) Familiar Word Reading Boys Girls 27 22 12 9 2 2 Grade 1 Grade 2 Grade 3 Oral Passage Reading Girls outperformed boys in reading a narrative passage, and this difference is statistically significant in later grades. Overall, boys read an average of 14 words per minute; they attempted to read 23 and correctly answered 13. Girls read an average of 17 words per minute, they attempted to read 25 and read 15 correctly. There is no statistically significant difference 44 between boys and girls’ performance in Grade 1, but the gender difference is significant in Grades 2 and 3. Girls read an average of 4 additional words in a narrative passage and girls in grade 3 read an additional 8 words per minute. These results suggest girls move into word 22 12 9 2 2 Grade 1 Grade 2 Grade 3 Oral Passage Reading Oral Passage Reading Girls Girls outperformed outperformed boysboys in reading in reading a narrative a narrative passage, passage, and and this this difference difference is statistically is statistically significant in later significant grades. Overall, inboys later grades. read Overall, an average boys of 14 words read per anthey minute; average attemptedof to 14 words read 23 and per minute; correctly they answered attempted to read 23 and correctly answered 13. Girls read an average of 17 words per 13. Girls read an average of 17 words per minute, they attempted to read 25 and read 15 correctly. There is no minute, they attempted to read 25 and read 15 correctly. There is no statistically significant difference statistically significant difference between boys and girls’ performance in Grade 1, but the gender difference is between boys and girls’ performance in Grade 1, but the gender difference is significant in significant in Grades 2 and 3. Girls read an average of 4 additional words in a narrative passage and girls in grade Grades 2 and 3. Girls read an average of 4 additional words in a narrative passage and girls in 3 read an additional 8 words per minute. These results suggest girls move into word reading faster than boys. grade 3 read an additional 8 words per minute. These results suggest girls move into word reading faster than boys. Figure 13– Differences in performance between boys and girls by grade, measured by the Figure 13– Differences in performance between boys and girls by grade, measured by the number of correct number of correct words read in a connected text per minute (CWCPM) words read in a connected text per minute (CWCPM) Oral Reading Passage Boys Girls 37 15 29 3 11 3 Grade 1 Grade 2 Grade 3 46 45 Dictation (Weighted score) Girls outperformed boys in dictation, and this difference is statistically significant in Grades 2 and 3. The difference in grade 1 is small and not statistically significant (about 1%), but girls scored about 7% higher on both the raw percentage score and weighted score in Grade 2. On average, girls scored almost 7% higher Dictation (Weighted score) on the outperformed Girls raw score, and boys 6% higher on the weighted in dictation, score and this in Gradeis difference 3. These results statistically suggest that significant boys and in Grades 2 and start girls out difference 3. The with approximately in gradethe1same writing is small skills and not Grade 1, but girls in statistically writing skills acquire (about significant 1%),faster than but girls boys in Grades do about scored 7% 2 and 3. on both the raw percentage score and weighted score in Grade 2. On higher average, girls scored almost 7% higher on the raw score, and 6% higher on the weighted score in Grade 3. These results suggest that boys and girls start out with approximately the same writing skills in Grade 1, but girls acquire writing skills faster than boys do in Grades 2 and 3. Figure 14– Differences in performance between boys and girls in dictation by grade, boys andscore as a weighted Figure 14– Differences in performance between girls in dictation by grade, as a weighted score Dictation (Weighted score) Boys Girls 53 46 30 23 9 10 Grade 1 Grade 2 Grade 3 A regression analysis was performed in order to determine the student-based factors impacting on the A regression analysis was performed in order to determine the student-based factors impacting acquisition of literacy skills. The dependent variables are the subtest scores while the independent variables on the acquisition of literacy skills. The dependent variables are the subtest scores while the include student-specific factors such as familial literacy, socio-economic status as indicated by ownership independent variables include student-specific factors such as familial literacy, socio-economic of the textbook status and reading as indicated materials and by ownership the textbook of the langauges and spoken at home reading in addition materials andto the gender. This is langauges spoken atat discussed inin length home Chatper 5. addition to gender. This is discussed at length in Chatper 5. Summary of Summary AssessmentResults of Assessment Results VANEGRA VANEGRA English English results results showshow reading reading gains gains acrossacross thegrades the three three grades tested. tested. As Anglophone As Anglophone students students move frommove Gradefrom they 1 Grade 1 to 3, to 3, they improve their improve their competence in competence all skill tested. inHowever, all skill tested. However, the average rate of the average progress rate is slow of progress which is slow to could contribute which could explaining contribute the to explaining slow development the slow of word-level development reading skills, oral of word-level reading skills, oral reading fluency and reading comprehension. Looking at reading fluency and reading comprehension. Looking at differences in performance between boys and girls, differences in performance between boys and girls, assessment results showed that boys and assessment results showed that boys and girls perform similarly in Grade 1, but girls outperform boys in the girls perform similarly in Grade 1, but girls outperform boys in the rest of the skills at rest of the skills at statistically significant levels in sub-test 5 (familiar word reading), sub-test 6a (oral passage statistically significant levels in sub-test 5 (familiar word reading), sub-test 6a (oral passage reading), and sub-test 8 (dictation). reading), and sub-test 8 (dictation). 46 Chapter 4: Performance in Oral Reading Fluency 4 and Reading Comprehension Chapter 4: Performance in Oral Reading Fluency and Reading Comprehension As stated before, oral reading fluency is a measure of overall reading competence: the ability to translate letters into sounds, unify sounds into words, process connections, relate text to As stated before, oral reading fluency is a measure of overall reading competence: the ability to translate meaning, and make inferences to fill in missing information (Hasbruck and Tindal, idem; letters into sounds, unify sounds into words, process connections, relate text to meaning, and make Fuschs et al, 2001). This is not to say, however, that oral reading fluency is the only predictor of inferences to fill in missing information (Hasbruck and Tindal, idem; Fuschs et al, 2001). This is not to say, reading comprehension among readers. Because oral reading fluency captures this complex however,and process it isreading that oral fluency strongly is the only associated predictor with reading comprehension bothoflistening and readingamong readers. Because comprehension, it is oral reading fluency captures this complex process frequently used as a marker of overall reading ability. and it is strongly associated with both listening and reading comprehension, it is frequently used as a marker of overall reading ability. Before VANEGRA, oral reading fluency had not been measured in Vanuatu. Thus, in order to Before VANEGRA, inform oral reading the establishment of fluency not been measured had standards reference in grade for early Vanuatu. Thus, in reading order VERM, to informwe under the sought to identify establishment the oral of reference reading standards fluency for early readingwhere grade levels Ni-Vanuatu under VERM, beginning we sought to identifyreaders the oral demonstrate high reading fluency reading levels comprehension where Ni-Vanuatu levels. beginning readers demonstrate high reading comprehension levels. VANEGRA English VANEGRA English results results showed showed evidence evidence of a positive of a positive correlation correlation between between oral reading oral reading fluency (sub- test 6a) (sub-test fluency 6a) and and reading reading comprehension comprehension (sub-test 6b): (sub-test 6b): better better fluency fluency in reading in reading appears appears to contribute to to contribute better to better comprehension in comprehension reading, although reading, inthis although relationship is notthis relationship linear. is not In Figure 15 below,linear. In average Figure 15 below, average reading comprehension levels of 75% and above start reading comprehension levels of 75% and above start at a minimum fluency level of 35 CWCPM. at a minimum fluency level of 35 CWCPM. Figure 15 –Average Scores in Oral Reading Fluency (sub-test 6a) and Reading Figure 15 –Average Scores in Oral ReadingComprehension (sub-test Fluency (sub-test 6a) 6b) Comprehension (sub-test 6b) and Reading Using a reference standard of 45 CWCPM to classify students as “fluent readers�, students are able to read Using a reference standard of 45 CWCPM to classify students as “fluent readers�, students are between 46 and 161 correct words in a connected text per minute (CWCPM) with an average fluency of 71 able to read between 46 and 161 correct words in a connected text per minute (CWCPM) with CWPM. At this level, students clustered around top comprehension levels – i.e. correct answers to 4 and 5 questions received - and less than a third demonstrated average reading comprehension levels below 80% 48 - i.e. correct answers to 4 or fewer questions. It should be noted that due to the few students that can be classified as fluent readers, the sample size is necessarily considerably lower than has been used hitherto. 47 an average fluency of 71 CWPM. At this level, students clustered around top comprehension levels – i.e. correct answers to 4 and 5 questions received - and less than a third demonstrated average reading comprehension levels below 80% - i.e. correct answers to 4 or fewer questions. It should be noted that due to the few students that can be classified as fluent readers, the sample size is necessarily considerably lower than has been used hitherto. Figure 16 – Average Reading Comprehension Levels in Fluent Students (N=130) Figure 16 – Average Reading Comprehension Levels in Fluent Students (N=130) Using a reference standard of 45 CWCPM to classify students as “fluent readers�, fluent students account Using a reference standard of 45 CWCPM to classify students as “fluent readers�, fluent for 9% of the sample. Differences by grade suggest fluency in reading emerges at the end of Grade 3 but students account for 9% of the sample. Differences by grade suggest fluency in reading emerges only among 23% of the Grade 3 students. Of the 130 fluent students, 81% were in Grade 3, 18% in Grade at the end of Grade 3 but only among 23% of the Grade 3 students. Of the 130 fluent students, 2, 81%and 1% onlyin were in Grade Grade 1. in In the 3, 18% sample Grade as only 2, and a whole, 11% 1% in of the Grade 1. girls andsample In the only 7% the as boys could a whole, 11% be of considered fluent in reading. Among fluent students, 60% were girls. the girls and only 7% the boys could be considered fluent in reading. Among fluent students, 60% were girls. Differences between fluent and less than fluent students are considerable in both accuracy and fluency. As expected,between Differences important differences fluent in both and less than fluent and fluency reading students comprehension are considerable emerge in both between accuracythese and fluency. two groups.As expected, While important less-than-fluent differences students in reading – i.e. those both fluency betweenand reading 0 and comprehension 44 CWCPM - attempted emerge to between read about theseintwo 21 words groups. a minute andWhile less-than-fluent got about students half of them correct, – i.e. for an oralthose reading reading between fluency average 0 and of fluent-students 44 CWCPM 10 CWCPM, attempted to read attempted toabout 21words read 58 words in got and a minute and 55 words got about correct (95% half of them accuracy), for correct, an oral reading for an fluency oral reading fluency average average–of of 71 CWCPM a 10 CWCPM, difference fluent of 61 students CWCPM in theattempted topassage. read 58 oral reading words In terms and got 55 of average words reading correct (95% less-than-fluent comprehension, accuracy), for students an oral showed reading fluency a fluency average level of 71 that allowed CWCPM them – a difference to receive reaching in of 61 CWCPM only one question, an the oral comprehension average reading passage. In terms of 13%. of average Conversely, reading fluent students comprehension, less-than-fluent students showed a fluency level that allowed them to receive received all questions and were able to understand about 83% of the text. only one question, reaching an average comprehension of 13%. Conversely, fluent students received all questions and were able to understand about 83% of the text. 49 48 Table 19 - Distribution of Students by Fluency Table 18- Distribution of Students by Fluency Table Less than Students of Students 18- Distribution Fluent Fluency by Students Fluent Less0 Reading than Fluent to 44 wordsStudents per Fluent >45 correct Students words per Reading 0 to 44 words per minute correct words per >45minute Freqeuncy minute Proportion minute Proportion SD N Overall Freqeuncy 1,152 Proportion 0.90 130 Proportion 0.10 0.008SD 1,282 N Overall Grade1 4251,152 1.000.90 2 130 0.000.10 0.008 0.003 4271,282 Grade1 Grade2 403 425 0.941.00 24 2 0.060.00 0.003 0.011 427 427 Grade2 Grade3 324 403 0.760.94 104 24 0.240.06 0.011 0.021 428 427 Grade3 Boys 590 324 0.930.76 47 104 0.070.24 0.021 0.010 637 428 Boys Girls 567 590 0.890.93 70 47 0.110.07 0.010 0.012 637 637 Girls 567 0.89 70 0.11 0.012 637 Boys | Grade 1 210 1.00 1 0.00 0.005 211 Boys Boys | Grade | Grade 2 1 202 210 0.971.00 6 1 0.030.00 0.005 0.012 208 211 Boys Boys | Grade | Grade 3 2 177 202 0.810.97 41 6 0.190.03 0.012 0.026 218 208 Boys | Grade Girls | Grade 1 3 215 177 1.000.81 1 41 0.000.19 0.026 0.005 216 218 GirlsGirls | Grade | Grade 2 1 203 215 0.931.00 16 1 0.070.00 0.005 0.018 219 216 GirlsGirls | Grade | Grade 3 2 154 203 0.730.93 56 16 0.270.07 0.018 0.031 210 219 Girls | Grade 3 154 0.73 56 0.27 0.031 210 Table Table 2019 -- fluency,accuracy Average fluency, Average accuracy reading andand comprehension reading levels, by comprehension condition by of levels, fluency of fluency condition Table 19 - Average fluency, accuracy and reading comprehension levels, by condition of fluency Less than Fluent (N=1,152) Fluent (N=130) Less than Fluent (N=1,152) Mean sd mean Fluent (N=130) sd Overall: > 45 words Mean 0.90 0.01 sd mean 0.10 0.01 sd Overall: CWCPM > 45 words 10 0.90 13 0.01 71 0.10 22 0.01 CWCPM Words attempted 21 10 10 13 58 71 2 22 Words Words attempted read (raw score) 10 21 13 10 55 58 4 2 Words read (raw score) 10 13 55 4 Reading comprehension 13 21 83 23 comprehension Readingattempted Questions 1 13 1 21 5 83 1 23 Questions attempted 1 1 5 1 In summary, there is much variation in reading fluency and comprehension among Ni-Vanuatu In summary, In students. Greater summary, there there isis oral much much readingvariation variation fluency in reading in reading fluency associated is fluency andwithcomprehension and comprehension higher levels among among of Ni-Vanuatu Ni-Vanuatu reading students. students. comprehension, Greater with fluency Greater oral reading oral reading fluent students— is associated9% fluency with of is associated levels of able the sample higher reading with higher to comprehension, read at least 45 levels correct with of reading fluent words students— comprehension, minute per 9% of the– sample reachingwith able fluent antoaverage students— read at fluency 9% of 71 CWCPM, least 45 correct of the sample words perwhich able minuteis to read at – associated least reaching an with 45 average correct average an fluency words of 71 per minute – understanding ofreaching 83% of an average the text they fluency read. of 71 By CWCPM, contrast, which is associated less-than-fluent students with– anthose i.e. average CWCPM, which is associated with an average understanding of 83% of the text they read. By contrast, understanding reading of between 0students and 83% 44of the text they CWCPM read. By contrast, less-than-fluent students – i.e. those less-than-fluent – read i.e. those -reading at a fluency between level 0 and 44 of 10 CWCPM CWCPM - read at with an average a fluency level of 10 reading between understanding of only0 and 16% 44 ofCWCPM the text - read they at a fluency read. The level suggest results of 10 CWCPMthat with an reading average fluency CWCPM with an average understanding of only 16% of the text they read. The results suggest that reading understanding of only 16% of the text they read. The results differs across both grades and gender. In particular, the results provide evidence that schooling suggest that reading fluency fluency differs across both grades and gender. In particular, the results provide evidence that schooling across both differsfluency increases levels, grades and gender. as students in Grade In particular, the results 3 demonstrated the provide evidence highest fluency that schooling levels while increases increases fluency fluency levels, levels,as students as students in Grade in 3 Grade demonstrated 3 demonstratedthe highest the fluency highest levels fluency while students levels whilein students in Grade 1 demonstrated the lowest fluency levels. Girls also exhibited greater reading Grade 1 demonstrated students the lowest fluency levels. Girls also exhibited greater reading fluency than boys, only fluency thanin Grade boys, only 1 demonstrated in Grades 2 and the 3.lowest fluency levels. Girls also exhibited greater reading in Grades fluency 2 and than 3. only in Grades 2 and 3. boys, 50 49 50 5 Chapter 5: Analysis of Student Factors Associated with Better Reading Outcomes Chapter 5: Analysis of Student Factors Associated with Better We sought to explore the associationReading Outcomes between students’ characteristics and performance using data collected We sought tostudent in the explorequestionnaire. Students the association were students’ between asked a series of questions and characteristics on their backgroundusing performance such data as collected their parents’ in whether questionnaire. the student literacy, Students were they attended kindergarten, asked athey and whether series hadof questions books on their for school and books. such as their parents’ literacy, whether they attended kindergarten, and whether background other they had books for school and other books. Table 21 - Characteristics of students in the sample along several student and family factors Table 20 - Characteristics of students in the sample along several student and family factors Mean SD N Age of students in the sample (in years) 8.6 1.5 1,281 Student speaks English at home 28% 45% 1,255 Student speaks Bislama or Vernacular at home 88% 33% 1,251 Student owns the school textbook 86% 34% 1,234 Student has a teacher that reads aloud to him/her 97% 17% 1,250 Student has reading materials at home: 62% 49% 1,259 In English 47% 50% 1,282 In French 1% 8% 1,282 In Bislama 13% 34% 1,282 In other language 1% 10% 1,282 Student has a literate family member: 90% 30% 1,243 Student has a literate mother 44% 50% 1,282 Student has a literate father 41% 49% 1,282 Student has both parents literate 28% 45% 1,282 Student has at least one literate sibling 45% 50% 1,282 Student has other literate family member 7% 26% 1,282 Student does homework and receives help from a family 93% 26% 1,267 member: From his/her mother 49% 50% 1,173 From his/her father 27% 44% 1,173 From his/her sibling 30% 46% 1,173 From another relative 4% 20% 1,173 Someone has a family member that reads with him/her 89% 31% 1,255 at home Mother 36% 48% 1,111 Father 24% 43% 1,111 Sibling 43% 50% 1,111 Other 8% 27% 1,111 Student attended kindergarten before Grade 1 94% 24% 1,247 Student was absent from school for more than 1 week 51% 50% 1,205 Student is in overage (as a proxy for repetition) 50% 50% 1,282 Note: Note: Missing Missing data data explains explains cases cases wherewhere N is N is less less than than 1,283. 1,283. 51 50 Inorder In orderto to determine determine the factors the factors that that are are statistically statistically significant significant in in the the acquisition acquisition of reading of skills, a reading regression skills, a was analysis regression analysis performed was on each subtests.on performed of the each The of the variable dependent subtests.is The dependent the subtest variable scores, is while the the subtest scores, independent while variables the independent are student-specific variables factors such asare student-specific gender, grade, family factors such literacy and aslanguage gender, the grade, family spoken literacy at home. Tables and language the24 22, 23, spoken and 25 give at home. the results Tables of the 22, 23, 24 and 25 give the results regressions. of the regressions. Table - Regression 22 21 Table Results Regression Results A A Part Part Sub-Test 1 Sub-test 2 Sub-test 3 Sub-test 4 Sub-test 5 Number of CLPM initial CSPM CFWPM CIWPM sounds 1.2165 -0.0985 2.8469 0.2975 -0.3175 Gender 0.5050 0.8240 0.3150 0.8200 0.7330 9.1389 2.4111 19.0239 5.5804 2.9700 Grade 2 0.0000 0.0000 0.0000 0.0000 0.0000 18.7601 3.8015 36.9030 16.2510 8.0740 Grade 3 0.0000 0.0000 0.0000 0.0000 0.0000 3.2736 -0.2257 3.0574 3.6495 2.0015 Girl in Grade 2 0.1510 0.6680 0.3630 0.0110 0.0610 2.6349 -0.0295 4.2897 6.1566 3.3628 Girl in Grade 3 0.3010 0.9540 0.2320 0.0070 0.0270 3.1563 0.4302 3.9110 1.4591 1.4189 English at Home 0.0100 0.0880 0.0220 0.1460 0.0460 -3.3532 -0.1572 -3.6583 -3.3585 -1.4229 Bislama at Home 0.0200 0.6200 0.0580 0.0040 0.0850 1.4665 -0.0156 2.0396 0.2896 0.6090 Owns Textbook 0.2520 0.9590 0.2940 0.7510 0.3540 1.8138 0.8570 2.4437 2.4602 1.2652 Teacher Reads Aloud 0.4200 0.0470 0.4510 0.0970 0.1540 Books available at 2.8097 0.3739 4.8824 4.0831 2.3352 home 0.0090 0.1190 0.0020 0.0000 0.0000 3.0349 0.7212 6.4336 1.6655 0.6289 Mother Literate 0.0310 0.0300 0.0040 0.1630 0.4280 1.9805 0.5782 -0.0060 1.6736 1.3848 Father Literate 0.1600 0.0850 0.9980 0.1690 0.0840 3.5314 1.0405 6.4533 1.8298 0.7997 Sibling Literate 0.0110 0.0010 0.0020 0.1100 0.3010 52 51 Table 23 Regression Results Part B Table 22 Regression Results Part B   Sub-Test 1 Sub-test 2 Sub-test 3 Sub-test 4 Sub-test 5 Number CLPM of initial CSPM CFWPM CIWPM sounds 3.3710 1.8537 -1.1863 0.4142 -0.3150 Other family member literate 0.1710 0.0000 0.7300 0.8210 0.7980 2.9029 0.1088 -1.0878 0.4073 1.2227 Interaction: Mother literate, female 0.1880 0.8140 0.7280 0.8150 0.3280 -0.6888 0.7788 5.1713 -1.0405 -1.3402 Interaction: Father literate, female 0.7600 0.1000 0.1060 0.5630 0.2970 -2.0577 0.0116 -3.1315 0.0376 -0.1697 Interaction: Sibling literate, female 0.3210 0.9780 0.2920 0.9820 0.8840 Interaction: Other family member -4.7821 -0.6832 2.5874 -1.7661 -1.0421 literate, female 0.1710 0.3790 0.6380 0.5560 0.6260 1.4169 0.2793 -1.1938 -2.3943 -1.0465 Test format: Bisclamar 0.3330 0.3000 0.5490 0.0680 0.2380 1.8530 0.4947 1.7927 0.0251 -0.1925 Does Homework 0.2030 0.2300 0.5030 0.9820 0.7930 -1.1940 -0.5907 2.8009 -0.2134 -0.6477 Reads At Home 0.4330 0.0660 0.1970 0.8560 0.4280 2.1907 0.6570 -0.1614 1.5061 1.4849 Attended Kindegarten 0.1640 0.0590 0.9480 0.2600 0.0410 -2.2836 -0.3851 -3.5478 -2.0995 -1.7518 Absent >1 Week 0.0270 0.0710 0.0160 0.0130 0.0030 -1.2535 -0.1739 -1.9799 -0.6356 -0.6435 Answered in Pictographs 0.0090 0.0890 0.0070 0.0720 0.0010 -0.4074 -0.0486 10.2436 -0.1529 -0.3305 Constant 0.9010 0.9470 0.0340 0.9480 0.8160 R^2 0.2471 0.2401 0.3793 0.3095 0.2064 N 1275 1275 1275 1275 1275 F statistic 16.8800 17.8400 33.2700 21.0600 12.3900 P-value 0.0000 0.0000 0.0000 0.0000 0.0000 Test for Normality of Residuals Shapiro-Wilkes 8.9250 8.6620 2.8750 7.3380 8.2860 P-value 0.0000 0.0000 0.0020 0.0000 0.0000 52 53 Table 24 Regression Results Part C Table 23 Regression Results Part C   Sub-test 6a Sub-test 6b Sub-test 7 Sub-test 8 LIST COMP DICT CWCPM RCOMP (%) (%) (TOTAL) 0.9024 0.0215 0.0321 -0.0608 Gender 0.5710 0.3160 0.3970 0.7450 7.4923 0.1049 0.1369 0.9157 Grade 2 0.0000 0.0000 0.0000 0.0000 22.1790 0.3395 0.3366 2.3532 Grade 3 0.0000 0.0000 0.0000 0.0000 4.4841 0.0516 0.1010 0.6630 Girl in Grade 2 0.0140 0.0490 0.0260 0.0040 8.8615 0.0755 -0.0228 0.5919 Girl in Grade 3 0.0010 0.0410 0.6200 0.0290 2.4190 0.0092 0.0511 0.1464 English at Home 0.0520 0.5760 0.0170 0.2410 -4.6955 -0.0266 0.0701 -0.2895 Bislama at Home 0.0030 0.1700 0.0080 0.0640 0.3595 -0.0128 -0.0181 0.1961 Owns Textbook 0.7570 0.4360 0.5090 0.1420 3.0238 0.0414 0.0843 -0.0196 Teacher Reads Aloud 0.1020 0.0920 0.0330 0.9430 Books available at 5.0234 0.0645 0.0800 0.3888 home 0.0000 0.0000 0.0000 0.0010 2.7597 0.0536 0.0726 0.5069 Mother Literate 0.0550 0.0090 0.0120 0.0020 1.8560 0.0156 0.0275 0.0402 Father Literate 0.2010 0.4430 0.3430 0.8020 2.4920 0.0495 0.1584 0.3228 Sibling Literate 0.0720 0.0120 0.0000 0.0360 53 54 Table 25 - Regression Results Part D Table 24 Regression Results Part D Sub-test 6a Sub-test 6b Sub-test 7 Sub-test 8 LIST COMP DICT CWCPM RCOMP (%) (%) (TOTAL) 0.1302 0.0307 -0.0669 0.2515 Other family member literate 0.9510 0.2400 0.1340 0.3020 -0.1747 -0.0136 -0.0422 -0.0122 Interaction: Mother literate, female 0.9340 0.6570 0.3030 0.9590 -0.9612 0.0178 0.0501 0.2826 Interaction: Father literate, female 0.6620 0.5670 0.2310 0.2350 -1.3530 -0.0496 -0.1521 -0.1741 Interaction: Sibling literate, female 0.5170 0.0800 0.0000 0.4310 Interaction: Other family member literate, -0.4881 -0.0474 0.0869 0.0898 female 0.8970 0.2750 0.1880 0.8320 -2.6862 -0.0090 0.0008 -0.4508 Test format: Bisclamar 0.0960 0.6830 0.9730 0.0050 -0.3022 -0.0142 -0.0026 -0.0147 Does Homework 0.8370 0.5170 0.9400 0.9370 -0.6259 0.0148 0.0072 0.2285 Reads At Home 0.6870 0.4490 0.8200 0.1460 2.6200 0.0235 0.0047 0.0137 Attended Kindegarten 0.0580 0.2610 0.8840 0.9400 -3.2516 -0.0291 -0.0416 -0.2891 Absent >1 Week 0.0020 0.0390 0.0240 0.0080 -0.8824 -0.0152 -0.0117 -0.1760 Answered in Pictographs 0.0360 0.0040 0.2510 0.0000 0.1986 -0.0666 -0.0045 0.6666 Constant 0.9440 0.0820 0.9470 0.0640 R^2 0.3444 0.3557 0.2591 0.3640 N 1275 1275 1275 1275 F statistic 24.8400 25.4800 20.0300 29.1600 P-value 0.0000 0.0000 0.0000 0.0000 Test for Normality of Residuals Shapiro-Wilkes 6.2890 8.8080 8.8060 8.7230 P-value 0.0000 0.0000 0.0000 0.0000 54 55 Grade progression was a significant factor in the acquisition of reading skills, with students in Grade 3 demonstrating more skill compared to those in Grade 1, with expected gains of 22.2 words per minute in oral reading fluency, and 34% in reading and listening comprehension on average. Students in Grade 2 compared to Grade 1, are expected to gain 7.5 words per minute, 10.5% in reading comprehension and 13.7% in listening comprehension on average. In addition, when grade was interacted with gender, girls demonstrated significantly improved scores in most subtests compared to boys in both Grades 3 and 4. Gender itself was not significant. Speaking English at home had a positive, significant association with most subtests, with students expected to gain 2.4 words per minute and 5.1% in listening comprehension on average compared to those who do not speak English at home. Speaking Bislama at home had a negative, significant effect on subtest scores with students expected to have 4.7 words per minute less than their counterparts with English-speaking households on average. Having books available at home had a positive, significant effect on subtests with students expected to gain around 5 words per minute and between 6-8% on comprehension scores. Family literacy was complex and interacted with gender. Maternal and sibling literacy had the most significant relationship across subtests and students with literate mothers or siblings are expected to gain 2.7-2.8 words per minute and between 5-15% in comprehension scores. Gender interactions were significant in some subtests. Female students benefitted from paternal and sibling literacy. This indicates that reading skill development is strongly associated with family circumstances and attitudes towards literacy. Doing homework was not significant and reading at home had a negative relationship on some subtests, but was not significant. Attending a kindergarten before Grade 1 had a positive and significant relationship with subtests 5 and 6a (unfamiliar words and oral fluency). Absenteeism had a negative, significant effect across all subtests. A student who answered part of the dictation component in pictographs is expected to score lower across all subtests, significantly so. This may indicate that students are struggling to bridge the language gap between their first language and English. In summary, as one would expect there is some evidence that students with literate family members, who have books at home, attended a kindergarten and speak English at home showed better and statistically significant reading outcomes. Students who were absent for more than one week in the school year performed worse at different grades and for various sub-tests in the survey. Attending kindergarten before Grade 1 showed a positive effect with statistical significance. Students who reported having family members who are literate, who read with them at home, on average, performed better in most sub-tests.   55 6 Chapter 6: Analysis of Teacher Factors Associated with Better Reading Outcomes We sought to explore the association between teacher characteristics –e.g., years of experience, professional qualification, in-service training, etc. and student reading outcomes using data collected in the teacher questionnaire. Table 26 summarizes the profile of teachers in the VANEGRA English sample. Most of the teachers in the sample are certified, experienced teachers with an average of 14 years teaching experience, and 13 years of experience as a certified teacher. 29% of teachers in the sample reported participating last year in some form of in-service training, with 20% of the teachers reporting they had participated in in-service training for reading in the last 2 years. The average number of days for the training workshops was 7 days. 74%, reported having a school library in their schools and 80% of those reported using it; a quarter of them, reported using it most of the time. 72% of teachers reported having a learning corner. 83% of the teachers surveyed reported having a PTA, with 45% having parent meetings, most of these (41%) at least once per term. 64% of teachers reported having the recommended reading texts and 31% reported using them most of the time. Association of Teacher Characteristics and Teaching Activities with Student Performance In order to explore which teacher characteristics and teaching activities impact on student performance, we performed individual z-tests on subtest results. These tests require several stringent assumptions, but can determine the statistical significance of characteristics. The commonly-used t-tests were also performed, but for reasons of brevity are not reported here. They are reported in Annex 1.14 The following analysis concentrates on comparisons between groups. Where a teaching activity, meeting or evaluation occurs on a regular basis, the two lowest groups are compared: generally “never� and “1-2 times per week� for teaching activities and “never� and “once per term� for meetings and evaluations exceptions to this rule will be noted. Table 26 gives a summary of these tests, full results are available in Annex 1. Overall, certification had a negative effect on all subtests. Teachers holding a certificate in Primary Teaching had students with significantly lower scores on all subtests compared to those who did not. Teachers whose highest qualification was a certificate in Education had students with significantly lower scores on all subtests compared to those teachers who had no qualification. Students taught by these teachers had 14.4 CWPM decrease in oral reading fluency and between 11-17% decrease in reading comprehension, listening comprehension and dictation scores. This suggests that teacher training may require substantial changes. Comparing the groups of teachers who had 0-4 years of experience and 5-10 years of experience, there was no significant difference between the groups in any of the subtests. This was a surprising result and may bear further investigation. Teachers who had between 0-4 years of experience had students with an average oral reading fluency of 24.6 CWPM, while teachers with between 5-10 years of experience had students 56 Table – Profile 26 25 Table of Anglphone – Profileof Teachers Anglphone Teachers in VANEGRA in VANEGRA Teacher factor Mean SD N Age (in years) 35 10 90 Percentage of teachers that hold a Certificate of Primary 72% 45% 96 Teaching (0=no; 1=yes) Teachers by type of professional qualification: None 16% 37% 98 Certificate of Primary Education 63% 48% 98 Certificate of Education 7% 26% 98 Other 0% 0% 98 Years of experience (in total) 14 10 98 Years of experience (as certified teacher) 13 10 81 Percentage that received in-service training last year (0=no; 29% 46% 96 1=yes) Percentage that received in-service reading training in last 2 years 20% 40% 96 (0=no; 1=yes) Average total number of days spent in training 7 6 19 Percentage of teachers that work in a school with library (0=no; 74% 44% 95 1=yes) How often does the teacher use it? Rarely 14% 35% 98 Half the time 17% 38% 98 Most of the time 24% 43% 98 Every lesson 0% 0% 98 Percentage of teachers that supervise students using the library 85% 36% 53 (0=no; 1=yes) Percentage of teachers that have a corner library (0=no; 1=yes) 72% 45% 90 Percentage of teachers that work in a school with a functioning 83% 38% 92 PTA (0=no; 1=yes) Percentage of teachers that meet with the parents of his/her 45% 50% 98 students (0=no; 1=yes) Once per term or less 41% 49% 98 Twice 1% 10% 98 Three times 1% 10% 98 Once a month 1% 10% 98 Percentage of teachers that have the recommended Reading Text 64% 48% 95 (0=no; 1=yes) How often do you use it? Rarely 7% 26% 98 Half the time 11% 32% 98 Most of the time 31% 46% 98 Every lesson 10% 30% 98 How useful do you find it? Not very useful 3% 17% 98 Moderately useful 15% 36% 98 Very useful 37% 48% 98 Percentage of teachers that have the teacher guide for reading 55% 50% 170 instruction (0=no; 1=yes) How useful do you find it? Not very useful 1% 10% 98 Moderately useful 6% 24% 98 Very useful 18% 39% 98 57 Most of the teachers in the sample are certified, experienced teachers with an average of 14 years teaching experience, and 13 years of experience as a certified teacher. 29% of teachers with an average oral reading fluency of 21.8 CWPM. Teachers with between 10-19 years of experience had students with an average oral reading fluency of around 16.6 CWPM. Further experience resulted in lower average oral reading fluency scores. More positively, inservice training had a significant, positive effect on student scores. Students whose teachers had attended an inservice training in the last two years had significantly higher levels of oral reading fluency (24.3 CWPM compared to 15.6 CWPM), reading comprehension and dictation skills. Reading- specific training had a significant and positive effect on all subtests. This suggests that these methods of teacher support offer a pertinent opportunity for improving the reading outcomes of Ni-Vanuatu students. The presence of a school library had a significant, positive outcome on all subtests. Students with access to a library had an average oral reading fluency of 20.1 CWPM. Those who did not had an average oral reading fluency of 13.6 CWPM. Teachers who supervised students in the library had a positive, significant outcome. Students whose teacher supervised their activities in the library had an average oral reading fluency of 22.1 CWPM. Those who did not had an average oral reading fluency of 15.9 CWPM. The presence of a reading corner in the classroom had a positive outcome. Those with a reading corner in the class had an average oral reading fluency of 20.1 CWPM and those who did not had an average oral reading fluency of 15.0 CWPM. These strong results suggest that literacy resources and supervision of students’ access to and use of them positively impacts the acquisition of literacy skills of Ni-Vanuatu students. Parental involvement with the school and specifically with their child’s teacher was also positive. Schools with a functioning Parents and Teachers Association (PTA) had students with significantly higher subtests scores across all subtests. Students at schools with a functioning PTA had average oral reading fluency of 18.5 CWPM. Those who did not had an average oral reading fluency of 13.1 CWPM. Students whose parents met with the teacher at least once per term also had significantly higher scores in all subtests. This suggests that parental involvement has a positive effect on the acquistion of reading skills. However, teachers who meet with parents twice per term rather than once per term only actually had a negative association with all subtests, significantly so. This should not assumed to be causative, however, as it may be evidence that teachers meet with parents of at-risk students more often. It is also possibly due to a small number of high-frequency meetings in the sample. Teachers’ ownership of the reading guide had a mixed association. It has a positive, significant association with dictation scores, but is not otherwise significant. This suggests that analysis of the guide and possibly updating the guide may be pertinent. Schools that had the appropriate and recommended reading for the grade had positive, significant increases in all subtest scores compared to those who do not, who had an average oral reading fluency of 13.9 CWPM. Those who attended schools with the materials available had an average oral reading fluency of 20.6 CWPM. This suggests that this resource is an important aspect to reading acquisition in Vanuatu. Students practicing identifying sounds had an insignificant association with all subtests except listening comprehension where it had a negative effect. Repeating words and sentences had a negative relationship 14 The commonly-used t-test requires fewer assumptions than the z-test and is less likely to return a result indicating that the variable is significant. They are a more conservative test in this case. 58 Table 27 Effects of Teacher and School Characteristics on Literacy Acquisition Table 26 Effects of Teacher and School Characteristics on Literacy Acquisition ORF RCOMP% LCOMP% DICT% z-statistic p-value z-statistic p-value z-statistic p-value z-statistic p-value Do you hold a Certificate in Primary -5.866 0.000 -5.136 0.000 -4.329 0.000 -6.123 0.000 Teaching? Highest Qualification -5.361 0.000 -5.422 0.000 -3.699 0.000 -6.181 0.000 Have you attended an inservice? 4.314 0.000 3.227 0.001 1.322 0.093 4.261 0.000 Have you attended an inservice? 4.154 0.000 2.881 0.002 3.081 0.001 4.809 0.000 Experience Level -1.023 0.153 -1.579 0.057 -0.320 0.374 -1.393 0.082 Library Present 4.356 0.000 3.618 0.000 5.691 0.000 2.967 0.002 Supervision in Library 2.240 0.013 1.747 0.040 1.515 0.065 1.916 0.028 Reading Corner Present 3.014 0.001 3.011 0.001 0.715 0.237 3.258 0.001 PTA Functioning 2.931 0.002 2.873 0.002 2.289 0.011 4.271 0.000 Meeting with Parents 7.201 0.000 7.417 0.000 5.970 0.000 6.021 0.000 How Often do Teachers Meet with Parents -18.089 0.000 -19.959 0.000 -10.209 0.000 -17.061 0.000 Teacher OwnsTeachers' Guide for Reading 0.966 0.167 1.202 0.115 -0.956 0.170 1.370 0.085 School has Recommended Reading 4.776 0.000 3.971 0.000 1.892 0.029 4.492 0.000 Students Practised Identifying the Sounds in 0.061 0.476 -0.365 0.358 -2.261 0.012 0.159 0.437 Letters The whole class repeated words or -6.041 0.000 -7.593 0.000 -5.625 0.000 -6.467 0.000 sentences that you said first Students copied down text from the 6.080 0.000 6.216 0.000 2.181 0.015 5.462 0.000 chalkboard Students copied downStudents retold a 3.839 0.000 3.226 0.001 2.783 0.003 4.373 0.000 story that they had read Students sounded out unfamiliar words 0.286 0.388 1.312 0.095 1.574 0.058 1.187 0.118 Students learned meanings of new words -0.727 0.234 -0.545 0.293 -1.826 0.034 0.171 0.432 Students Read Aloud 9.657 0.000 8.541 0.000 4.306 0.000 9.624 0.000 Students assigned reading on their own 7.993 0.000 5.368 0.000 4.843 0.000 8.218 0.000 Written Evaluations -2.129 0.017 -2.214 0.013 -2.598 0.005 -0.251 0.401 Oral Evaluations -0.342 0.366 -1.162 0.123 -4.676 0.000 -0.375 0.354 Review of Portfolios and other projects -3.012 0.001 -3.451 0.000 -5.202 0.000 -3.517 0.000 Student Reads aloud from chalkboard -3.506 0.000 -3.477 0.000 -4.559 0.000 -3.826 0.000 Review of homework -0.611 0.271 -1.407 0.080 -3.217 0.001 -1.026 0.152 Observation and/or competency checklists 4.564 0.000 4.525 0.000 3.125 0.001 4.082 0.000 59 60 when teachers who used this activity 1-2 days a week were compared with those who did it 3-4 times a week. This was the case with all subtests. Copying text from the chalkboard had a positive relationship with all subtests and was significant for all tests. Students who practiced this activity had an average oral reading fluency of 11.4 CWPM compared to those who did not with an average oral reading fluency of 2.6 CWPM. Retelling stories that a student had read was significant and positive for all subtests. Students sounding out unfamiliar words had a positive, significant effect on comprehension and dictation It was insignificant and positive for oral reading fluency. Learning the meanings of new words was insignificant for all subtests except listening comprehension where it had a negative effect. Students reading aloud had a positive, significant effect on all subtests as did students assigned reading on their own. Students who read aloud and were assigned reading 1-2 days per week had average oral reading fluency of 15.7 and 18.0 CWPM respectively. Those who did not had average oral reading fluencies of 1.2 CWPM and 4.1 CWPM respectively. The effect of student evauation was mixed. Written evaluations had a negative association on all subtests, and were significant for all. However, this is the case only for comparisons between evaluations occurring once per term and never. More regular evaluations appear to have a positive relationship. Oral evaluations had a negative association with all subtests but was only significant for listening comprehension. More regular evaluations appear to have a positive association with subtest results. Review of portfolios and other projects had a negative, significant association with all subtests. Observation and competency checklists had a generally positive, significant relationship with subtest scores. These results suggest that teacher support in the area of evaluation may be beneficial. Students reading aloud from the chalkboard had a generally negative association with all subtests except initial phoneme identification. It was significant for all subtests except correct phoneme identification. Review of homework had a negative, significant relationship with all subtests except correct letter identification and non-word reading. Effect of Teacher Expectations on Student Performance Finally, we explored the relationship between teacher’s expectations on reading outcomes and student performance in sub-test 6a (oral reading passage). Table 28 shows the distribution of teacher expectations about reading outcomes, including the median. For example, most teachers in Anglophone schools in Vanuatu expected students to read aloud a short passage with a few mistakes at the end of Grade 2 (34%), which is consistent with the median expectation for this outcome (highlighted in green in Table 28). On average, teachers expect students to recite the alphabet, recognize the sounds of letters and write their names at the end of Grade 1; and to sound out invented words, read aloud a short passage and understand stories they read by the end of Grade 2. In order to observe if teacher expectations held an effect on reading 60 distribution of teacher expectations about reading outcomes, including the median. For example, most teachers in Anglophone schools in Vanuatu expected students to read aloud a short passage with a few mistakes at the end of Grade 2 (34%), which is consistent with the median expectation for this outcome (highlighted in green in Table 27). Table 28 – Teachers median expectations about reading outcomes Table 27 – Teachers median expectations about reading outcomes Teacher responses Reading outcome Grade Mean SD N Kindergarten 11% 32% 108 Grade 1 21% 41% 108 Read aloud a short passage with Grade 2 34% 48% 108 few mistakes Grade 3 19% 39% 108 Not important 2% 14% 108 Kindergarten 19% 39% 108 Grade 1 52% 50% 108 Write name Grade 2 16% 37% 108 Grade 3 8% 28% 108 Not important 0% 0% 108 Kindergarten 5% 21% 108 Grade 1 36% 48% 108 Understand stories they read Grade 2 32% 47% 108 Grade 3 19% 39% 108 Not important 1% 10% 108 Kindergarten 5% 21% 108 Grade 1 39% 49% 108 Recognize the sounds of letters Grade 2 42% 50% 108 Grade 3 7% 26% 108 Not important 1% 10% 108 Kindergarten 3% 17% 108 Grade 1 31% 46% 108 Sound out unfamiliar words Grade 2 45% 50% 108 Grade 3 12% 33% 108 Not important 2% 14% 108 Kindergarten 12% 33% 108 Grade 1 44% 50% 108 Understand stories they hear Grade 2 19% 40% 108 Grade 3 18% 38% 108 Not important 0% 0% 108 Kindergarten 18% 38% 108 Grade 1 52% 50% 108 Recite alphabet Grade 2 14% 35% 108 Grade 3 8% 28% 108 Not important 3% 17% 108 62 61 fluency and comprehension, we measured teachers’ expectations and fluency by measuring differences in student performance when teachers’ expectations deviate from the median. To this end, we regressed student performance against teachers’ expectations, excluding the median expectation: result=α+β1 I_kindergarten+β2 I_1+β3 I_3 The coefficient gives us an average difference in performance of the baseline group of students (students whose teachers have the median expectation) and students whose teachers have different expectations. For example, we compare students whose teachers expect them to recite the alphabet before Grade 1 to students whose teachers expect them to do so in Grade 1 in how students perform in reading comprehension. Overall, students whose teachers expect them to perform a task too early or too late tend to perform worse than students whose teachers have more realistic expectations around the median expectation. Teachers’ expectations on when students should write their name, understand stories they read, sound out invented words, understand stories they hear and recite the alphabet are associated at statistically significantly levels with oral reading fluency and reading comprehension. Tables 29 and 30 provide the results of the regression analyses carried out on teacher expectations and student performance. Effects are reported in comparison to the average effect of students whose teacher’s expectations match the median expectation for a given skill. Student writes his/her own name (Median expectation = end of Grade 1): Most teachers expect students to write their name at the end of Grade 1 (54% of the sample). Students whose teachers expect them to do so in kindergarten (32% of the sample) read an average of 3.2 additional words and scored 3.2% higher on reading comprehension. Students whose teachers expect them to do so in Grade 3 (2% of the sample) read about 33 more words and scored 45% higher in reading comprehension. The effect of teacher expectation is only positive and statistically significant for Grade 3 students; however, the small size of the sample does not allow us to generalize the effect of late expectation for this task. Student understands stories s/he read (Median expectation = end of Grade 2): Most teachers expect students to understand stories they read in Grade 2. Compared to the median expectation( 40% of the sample), students whose teachers expect them to do so in kindergarten read 10.8 fewer words and scored 14% lower on comprehension. Although this is statistically significant, this group is made up of less than 5% of the sample. There are no statistically significant differences between the median expectation and other groups. Student sounds out invented words (Median expectation = end of Grade 2): Most students are expected to sound out invented words in Grade 2. This group makes up 35% of the sample. Students who are expected to do so in kindergarten, less than 2% of the sample, read an average of 16.5 fewer words, and scored 22% lower on comprehension. Although this is significant, the size of the group is too small to conclude the benefits of earlier expectation. Students who are expected to do so in grade 1, about 32% of the sample, read an average of 4 fewer words and 5.8% lower on comprehension. 62 Student understands stories they hear (Median expectation = end of Grade 1): Most students are expected to understand stories they hear in Grade 1. This group makes up 45% of the sample. Students who are expected to do so in kindergarten, 14% of the sample read an average of 11.5 more words and 13.5% higher reading comprehension. Students who are expected to do so in grade 2, 17% of the sample, read 5 more words and scored 6% higher. Those who are expected to do so in grade 3, 22% of the sample, read 13.5 more words and 17% higher comprehension. This difference is statistically significant and suggests that a more realistic expectation improves student performance. Students with teachers, who do not think this is important, 3% of the sample, read 10 fewer words and 13% lower comprehension. These results suggest that teachers’ expectations play a role in listening comprehension Student is able to recite the alphabet (Median expectation = end of Grade 1): Most students are expected to recite the alphabet in Grade 1. This group makes up 45% of the sample. Students who are expected to do so in Grade 2 read an average of 14 words more and 18% higher on comprehension. Although this is statistically significant, this group only makes up 9% of the sample. These results suggest that teachers’ more realistic expectations help student performance. 63 Table 29 - Regression analyses of average relationships of teachers’ expectations on fluency in reading Table 28 - Regression analyses of average relationships of teachers' expectations on fluency in reading Dependent variable Dropped category: median expectation per skill Correct words read per minute (CWCPM) Independent variables Coeff F R2 N Read aloud a short passage with few mistakes Kindergarten -6.72 3.69 0.02 1124 Grade 2 4.98 Grade 3 2.62 Write name Kindergarten 3.24 4.81 0.06 1161 Grade 2 -1.97 Grade 3 32.85 *** Understand stories they read Kindergarten -10.81 ** 7.21 0.01 1161 Grade 1 1.09 Grade 3 1.24 Recognize the sounds of letters Kindergarten 3.43 0.16 0.00 1175 Grade 2 -0.75 Grade 3 1.49 Sound out invented words Kindergarten -16.54 ** . 0.01 1161 Grade 1 -4.13 Grade 3 -1.68 Not important 0.23 Understand stories they hear Kindergarten 11.47 17.75 0.07 1175 Grade 2 5.04 Grade 3 13.58 ** Not important -10.26 *** Recite alphabet Kindergarten 0.61 1.53 0.03 1165 Grade 2 14.19 * Grade 3 -2.57 Significant levels: * p<0.05 ** p<0.01 *** p<0.001 64 Table 30 - Regression analyses of average relationships of teachers’ expectations on reading comprehension Table 29 - Regression analyses of average relationships of teachers' expectations on reading comprehension   Dependent variable Dropped category: median expectation per skill % Reading Comprehension Independent variables Coeff F R2 N Read aloud a short passage with few mistakes Kindergarten -7.43 2.30 0.02 1124 Grade 2 7.43 Grade 3 3.92 Write name Kindergarten 3.24 6.18 0.06 1161 Grade 2 -1.87 Grade 3 45.17 *** Understand stories they read Kindergarten -14.42 ** 6.70 0.01 1161 Grade 1 1.80 Grade 3 1.59 Recognize the sounds of letters Kindergarten 4.19 0.12 0.00 1175 Grade 2 0.05 Grade 3 1.64 Sound out invented words Kindergarten -22.39 *** . 0.01 1161 Grade 1 -5.85 Grade 3 -2.72 Not important 5.61 Understand stories they hear Kindergarten 13.48 17.79 0.07 1175 Grade 2 6.05 Grade 3 17.67 * Not important -13.22 *** Recite alphabet Kindergarten 0.24 1.57 0.03 1165 Grade 2 17.90 * Grade 3 -4.01 Significant levels: * p<0.05 ** p<0.01 *** p<0.001 65 In conclusion, results of the analysis of teacher factors identified a set of teacher characteristics, instructional and assessment methods, and teacher expectations that have an effect on the reading performance of Anglophone students in Vanuatu. In terms of teacher characteristics, teacher certification had an average negative effect. There was no difference between groups with 0-4 years experience and 5-10 years. Attendance to in-service training, holding meetings with parents, and use of the teacher guide, all showed statistically significant effects on student fluency in reading. These results stress not only the importance of parental oversight of the learning process of children but also the need to provide teachers with relevant and regular in-service training and to provide them with appropriate support materials for reading instruction. Regarding teacher instructional and assessment methods, the significant differences came from students whose teachers made the whole class repeat words or sentences daily (positive), copy down text from the board daily (positive), never have students read aloud to teachers (negative), and students who were never assigned independent reading during school time (negative). The negative statistical significance on student performance of teachers that never ask students to read aloud or independently in class highlights the importance of this activity to develop fluency in reading as students who carried out this activities daily showed better reading outcomes. The positive and statistically significant effect of copying from the chalkboard deserves further exploration. A possible explanation for it may be that teachers, confronted with a lack of instructional and reading materials, may use the blackboard to write down stories and activities for students to read – and re-read - at home and in the classroom. Matching symbol and sound through writing may also reduce interference from Bislama at home. In terms of methods of evaluation, oral evaluation, having students read aloud from the chalkboard and use of competency lists produce statistically significant differences. Consistently, the use of observation or competency lists produced negative and statistically significant effects on student fluency in reading, which calls into question if the skills and content listed in these checklists relate to proper assessment of reading abilities in the early grades. The positive and statistically significant effect associated with students who are never assessed by reading aloud from the chalkboard deserves further exploration. Finally, in terms of teacher expectations, students whose teachers expect them to perform a task too early or too late tend to perform worse than students whose teachers have more realistic expectations around the median expectation. However, many of these effects are brought to bear by a small percentage of students that complicates proper generalizations about the average effects for these expectations. 66 7 Chapter 7 - Next Steps VANEGRA English results call for immediate attention to the way reading development is taking place in the country. As a diagnostic study, its main purpose is to (a) generate data on the extent reading performance in the first cycle of primary education, (b) identify specific skills that could be hampering reading comprehension among Anglophone Ni-Vanuatu students, and (c) identify student and teacher characteristics and behaviors that contribute to better reading outcomes in Vanuatu. VANEGRA results indicate that while most students are able to develop some of the basic skills needed to become effective readers, poor knowledge of the alphabetic principle hinders the development of word-level reading, which will slow down fluency development and limit comprehension. Based on the data presented, specific recommendations to be considered are presented to improve the quality of reading instruction in Anglophone schools in Vanuatu: • Improve the focus and structure of reading instruction to promote greater fluency in reading by the end of Grade 3. Research has shown that developing fluency in reading is crucial to help students become effective readers in the first years of primary education. As students approach reading at a speed of about 45-60 words per minute, the reader becomes better able to focus on the meaning of the text rather than on individual letters and words. As shown by VANEGRA English results, students reading at least 45 correct words per minute were able to understand about 83% of the text they read. However, only 1 in 10 Anglophone students is able to reach this fluency level at the end of Grade 3. Two factors could contribute to explain these results. On the one hand, the low levels of letter recognition skills and poor knowledge of phonetic principles observed in the VANEGRA suggest current instruction falls short in developing a solid foundation for the development of reading fluency. On the other, poor language skills on students entering the Anglophone stream may contribute to delay the development of pre-reading skills in Grades 1 and 2 as students struggle to develop language and reading skills simultaneously. As the MoE sets forth to implement the new K-12 curricula, it is fundamental that instructional improvements in the early grades take into account the linguistic diversity of the country and provide adequate strategies to prepare students for reading development in a secondary language (L2). In addition, the new curriculum opens up the opportunity to improve instruction of letter and word-level reading skills to promote a better sequenced instruction of basic reading skills. • Ensure teachers working in the early grades have the knowledge to improve their practice to impact the reading outcomes of their students. In order to improve reading instruction in Vanuatu, teachers will have to improve their knowledge of reading instruction to improve classroom practice. Although most teachers in VANEGRA had fairly realistic expectations about the reading outcomes of their students, average scores per sub-tests in the assessment fell short of fulfilling these expectations. Moreover, the fact that some instructional methods typically associated with better reading outcomes showed no statistical significance in the Vanuatu context, while others showed counterintuitive 67 effects on student outcomes, suggests the need to review how these activities are carried out in the classroom to better understand the possible factors that are hindering their effectiveness. This is also true of the average effects of teacher methods observed on student outcomes. • Support the reading instruction skills of as many teachers working with beginning readers as possible. Data from the teacher questionnaire showed that only 29% of the teachers in the sample participated in general in-service training courses and only 20% had attended in-service training on reading in the last two years. If only 2 in 10 teachers in the country benefit from learning about specific ways in which they can improve their practice, Ni-Vanuatu teachers will continue to practice their profession in isolation. This would be particularly critical for the most experienced teachers –i.e., those in the >50th and 75th percentiles of experience- for whom experience tended to negatively affect the reading outcome of their students. The role of the newly created In-Service Unit (ISU) at the MoE will be critical to further develop teacher knowledge and practice for reading instruction. As such, it is recommended that VANEGRA findings inform the development of the lesson plans and materials and that ISU staff works in close collaboration with curriculum developers to ensure teachers understand the new curriculum goals and receive support on how to achieve them. • Establish reference reading standards to monitor reading development in the early grades. As the MoE moves on to establish an oral reading fluency standard under VERM, it is important to consider that these indicators should be considered reference standards and not high-stakes benchmarks that would jeopardize additional funding or the promotion of teachers. Since these reference standards are drawn from baseline data, additional measures will be needed in subsequent years to learn about the rate at which Ni-Vanuatu students develop reading abilities. In this sense, reading standards should not be seen as high-stakes but an essential piece to monitor reading progression in the classroom. In order to set up national reference standards to monitor system- level quality improvements, it would be best to use the percentage of zero-score students in selected sub-tests as a marker and track reductions in the shares at least biannually. Monitoring achievements over time will eventually provide more information on the rate and the way in which average fluency develops among Anglophone students. A modified version of the test could be used to screen students during the school year that may be in need of additional support. • Help teachers translate national reference standards into easy-to-assess, easy-to-monitor reading goals to monitor the reading progression of their students during the school year. In order for teachers and schools to be able to be held accountable for reading outcomes, teachers, school officials and parents need to understand what these standards mean and how each can support reading development in their own school. School development plans should contain reading improvement goals as part of their minimum service standards, as well as a description of activities aimed at encouraging reading. Parents and the community as a whole should be brought into this effort. 68 • Introduce policy actions that increase student exposure to literacy outside the school. VANEGRA English results showed how students who have reading books at home have better reading outcomes and are more likely to become fluent readers. The effect was positive for both boys and girls and for all basic reading skills. Thus, it is advisable that the MoE promtes increased student access to books at home. However, making more books available to students will not per se ensure better reading outcomes. Along with access to more reading materials, Ni-Vanuatu children will need support to develop a reading habit beyond the requirements of the school curriculum. One way of achieving this would be ensure the books being procured by the ongoing Book Flood program are not only grade- appropriate but that they are accessible to students in and outside of the classroom. Since 74% of the teachers reported having access to a school library, an adequate book-borrowing scheme carries the potential to expose students to print on a more regular basis. Another way of increasing exposure to literacy would be to develop community literacy programs where schools become a focal point of literacy in the community. Teachers and community leaders can start up reading clubs and reading competitions to further promote a reading culture among beginning and more experienced readers. • Promote strategies to assure greater parental and community involvement in the reading development process of children. Research shows that the earlier the parental involvement, the more powerful and long-lasting the effects will be both in terms of academic and behavioral outcomes of children. Moreover, research also shows that the most effective form of parental involvement includes those where parents participated in learning activities at home. As seen in VANEGRA results, holding PTA meetings at least once a term translated into one of the largest positive effects on reading outcomes next to access to books and guided reading by teachers. What is more, having parents read at home with their children also showed positive effects on student reading outcomes. If parents and schools communicate regularly on the academic progress of children, parents tend to monitor school and classroom activities, and coordinate efforts with teachers such as helping with homeworks and carrying out extracurricular activities. For this to happen, it is important that parents and schools commit jointly to the reading development process of children. In addition to providing information on the academic progress of children, schools can advise on –and even facilitate- different ways in which parents can promote reading at home. If reading outcomes are to improve in the country, reading development must be seen as a joint enterprise that extends beyond the teacher and the school classroom environment. • Finally, it is clear that more research is needed to better understand the factors that contribute to differences in reading performance between boys and girls. An analysis of the factors that contribute to these differences is beyond the scope of this survey. However, VANEGRA English data showed that boys and girls finish Grade 1 at similar levels of performance in reading, but in Grade 2 girls transition into word-level reading faster than boys and the difference increases at the end of Grade 3. A better understanding of this phenomenon is critical to inform sector policies and increase the success of future reading development programs. 69 Bibliographical References Abadzi, H. 2006. Efficient learning for the poor: Insights from the Frontier of Cognitive Neuroscience. Washington, DC: The World Bank. Chabbott, C. (2006). Accelerating early grades reading in high priority EFA Countries: A desk review. From http://www.equip123.net/docs/E1-EGRinEFACountriesDeskStudy.pdf (Accessed on October 29, 2010) Chiappe, P., L. Siegel, and L. Wade-Woolley. 2002. Linguistic diversity and the development of reading skills: A longitudinal study. Scientific Studies of Reading 6(4): 369–400. Ehri, L. C., Nunes, S. R., Stahl, S. A., & Willows, D. M. (2001). Systematic phonics instruction helps students learn to read: Evidence from the National Reading Panel’s meta- analysis. Review of Educational Research, 71(3), 393-447. Fuchs, L., D. Fuchs, M.K. Hosp, and J. Jenkins. 2001. Oral Reading Fluency as an Indicator of Reading Competence: A Theoretical, Empirical, and Historical Analysis. Scientific Studies of Reading 5(3), 239–256. Hasbrouck, J., & Tindal, G. A. (2006). Oral reading fluency norms: A valuable assessment tool for reading teachers. International Reading Association, 636–644. Jiménez, J. E. and I. O’Shanahan Juan. 2008. Enseñanza de la lectura: de la teoría y la investigación a la práctica educativa. Revista Iberoamericana de Educació, 45(5): 1–22. Available at http://www.rieoei. org/2362.htm (accessed October 28, 2010). Linan-Thompson, S., and S. Vaughn. 2007. Research based methods of reading instruction for English language learners: Grades K-4. Alexandria, VA:Association for Supervision and Curriculum Development. National Reading Panel. 2000. Teaching Children to Read: An Evidence-Based Assessment of the Scientific Research Literature on Reading and Its Implications for Reading Instruction. Washington, DC: National Institute of Child Health and Human Development. Available at http://www.nichd.nih.gov/publications/nrp/ report.cfm (accessed July 13, 2010). RTI. 2009. Early Grade Reading Assessment Toolkit. Prepared by RTI for The World Bank, Office of Human Development. Available at https://www.eddataglobal.org/documents/index.cfm?fuseaction=pubDetail& ID=149 (accessed September 03, 2010). RTI. 2010. Guyana Early Grade Reading Assessment: October 2008 Results. Prepared by RTI for The World Bank, Office of Human Development. Available at https://www.eddataglobal.org/ (accessed August 02, 2010). 70 Sprenger-Charolles, L. 2004. Linguistic Processes in Reading and Spelling: The Case of Alphabetic Writing Systems: English, French, German and Spanish. Pp. 43–66 in Handbook of Children’s Literacy. Edited by T. Nunes and P. Bryant. Dordrecht, the Netherlands: Kluwer Academic Publishers Share, D. L., Jorm, A., Maclearn, R., & Matthews, R. 1984. Sources of individual differences in reading acquisition. Journal of Education Psychology, 76, 1309-1324. Snow, C. E., Burns, M. S., & Griffin, P. (Eds.). 1998. Preventing reading difficulties in young children. Washington, DC: Committee on Preventing of Reading Difficulties in Young Children and National Academy Press. Stanovich, K. E. (2000). Progress in understanding reading: Scientific foundations and new frontiers. New York: Guilford Press. Stores, Christine. 2008. “Language, Literacy and Bilingualism. A Report Presented to MEWAC to support development of the National Language Policy� Vanuatu Education Support Program – Curriculum Reform Project. (Mimeo). Vanuatu Institute of Education (VITE). Diploma of Primary Education Teaching. Curriculum Document. (Mimeo)   71 72 ANNEX 1/ TABLES ANNEX 1/ TABLES Table 31 - VANEGRA English Reliability Matrix Table 30 -VANEGRA English Reliability Matrix Correct Letters Correct Non Correct Letters Phonemic Correct Words Oral Reading Reading Listening Sounds Per Words Per Dictation Per Minute Awareness Per Minute Fluency Comprehension Comprehension Minute Minute Correct Letters Per 1 .603** .737** .713** .680** .709** .683** .514** .711** Minute Phonemic .603** 1 .545** .466** .439** .468** .473** .454** .502** Awareness Correct Letters .737** .545** 1 .723** .637** .730** .699** .558** .710** Sounds Per Minute Correct Words Per Minute .713** .466** .723** 1 .873** .957** .846** .514** .803** Correct Non Words .680** .439** .637** .873** 1 .869** .744** .439** .730** Per Minute Oral Reading Fluency .709** .468** .730** .957** .869** 1 .867** .526** .807** Reading .683** .473** .699** .846** .744** .867** 1 .559** .785** Comprehension Listening .514** .454** .558** .514** .439** .526** .559** 1 .524** Comprehension Dictation .711** .502** .710** .803** .730** .807** .785** .524** 1 Statistics Mean 19.67 .47 43.26 12.31 6.63 16.56 .21 .44 .28 SD 20.17 .41 31.03 18.39 11.86 23.62 .30 .36 .29 Min .00 .00 .00 .00 .00 .00 .00 .00 .00 Max 97.00 1.00 146.34 136.36 75.00 160.91 1.00 1.00 1.00 73 Table 32 - Table 31Cronbach’s Alpha: Cronbach's Alpha: Grade Grade One One average item-test item-test Item Obs Sign inter item alpha correlation Table 31 Cronbach's Alpha: Grade Onecorrelation covariance Correct Letters Per Minute 427 + 0.84 0.79 0.49 0.89 average Phonemic Awareness 427 + item-test 0.57 item-test 0.45 0.56 0.91 Item Obs Sign inter item alpha correlation correlation Correct Letters Sounds Per Minute 427 + 0.71 0.62 covariance 0.53 0.90 Correct Words Per Correct Letters Minute Per Minute 427 427 + + 0.84 0.88 0.79 0.84 0.49 0.48 0.89 0.88 Phonemic Awareness Correct Non Words Per Minute 427 427 + + 0.57 0.79 0.45 0.72 0.56 0.51 0.91 0.89 Correct Letters Oral Reading Sounds Per Minute Fluency 427 427 + + 0.71 0.89 0.62 0.86 0.53 0.48 0.90 0.88 Correct Comprehension ReadingWords Per Minute 427 427 + + 0.88 0.79 0.84 0.72 0.48 0.51 0.88 0.89 Correct Non Listening Words Per Minute Comprehension 427 427 + + 0.79 0.56 0.72 0.45 0.51 0.56 0.89 0.91 Oral Reading Fluency Writing 427 427 + + 0.89 0.75 0.86 0.68 0.48 0.51 0.88 0.89 Reading Comprehension Test scale 427 + 0.79 0.72 0.51 0.51 0.89 0.91 Listening Comprehension 427 + 0.56 0.45 0.56 0.91 Writing Table + 427 32 Cronbach's 0.75 Alpha: Grade Two 0.68 0.51 0.89 Test scale 0.51 0.91 average inter item-test item-test Item Obs Sign item alpha Table 33 - Cronbach’s correlation Alpha: Grade correlation Two Table 32 Cronbach's Alpha: Grade Two covariance Correct Letters Per Minute 428 + 0.87 0.82 0.59 0.92 Phonemic Awareness 428 + 0.58 0.48 average 0.67inter 0.94 item-test item-test Item Obs Sign item alpha Correct Letters Sounds Per correlation correlation 428 + 0.80 0.74 0.61 covariance 0.93 Minute Correct Letters Per Minute 428 + 0.87 0.82 0.59 0.92 Correct Words Per Minute 426 + 0.93 0.90 0.58 0.92 Phonemic Awareness 428 + 0.58 0.48 0.67 0.94 Correct Non Words Per Minute 427 + 0.86 0.81 0.60 0.92 Correct Letters Sounds Per Oral Reading Fluency 427 428 + 0.92 0.80 0.90 0.74 0.58 0.61 0.92 0.93 Minute Reading Comprehension 428 + 0.87 0.82 0.59 0.92 Correct Words Per Minute 426 + 0.93 0.90 0.58 0.92 Listening Comprehension 428 + 0.62 0.53 0.66 0.94 Correct Non Words Per Minute 427 + 0.86 0.81 0.60 0.92 Writing 428 + 0.84 0.79 0.60 0.92 Oral Reading Fluency 427 + 0.92 0.90 0.58 0.92 Test scale 0.61 0.93 Reading Comprehension 428 + 0.87 0.82 0.59 0.92 Listening Comprehension 428 + 0.62 0.53 0.66 0.94 Writing 428 + 0.84 0.79 0.60 0.92 Test scale 0.61 0.93 74 73 74 Table 34 - Cronbach’s Alpha: Grade Three 74 Table 33 Cronbach's Alpha: Grade Three average item-test item-test Item Obs Sign inter item alpha correlation correlation covariance Correct Letters Per Minute 428 + 0.76 0.69 0.57 0.91 Phonemic Awareness 428 + 0.59 0.48 0.61 0.93 Correct Letters Sounds Per 428 + 0.79 0.72 0.56 0.91 Minute Correct Words Per Minute 428 + 0.90 0.87 0.53 0.90 Correct Non Words Per Minute 428 + 0.84 0.78 0.55 0.91 Oral Reading Fluency 428 + 0.90 0.87 0.53 0.90 Reading Comprehension 428 + 0.85 0.80 0.55 0.91 Listening Comprehension 428 + 0.59 0.49 0.61 0.93 Writing 428 + 0.83 0.77 0.55 0.91 Test scale 0.56 0.92 75 Table 35 - ANOVA Results: Differences in Means across Sub-tests Table 34 - ANOVA Results: Differences in Means across Sub-tests Sub-test Factor Sum of Squares df Mean Square F Grade Between 94,552.68 2 47,276.34 150.01 Within 403,076.42 1,279 315.15 Letter name Total 497,629.10 1,281 47,591.49 knowledge Gender Between 1,776.69 1 1,776.69 4.59 Within 495,852.41 1,280 387.38 Total 497,629.10 1,281 2,164.07 Grade Between 3,501.48 2 1,750.74 127.10 Within 17,617.73 1,279 13.77 Initial sound Total 21,119.21 1,281 1,764.51 recognition Gender Between 0.07 1 0.07 0.00 Within 21,119.14 1,280 16.50 Total 21,119.21 1,281 16.57 Grade Between 392,513.70 2 196,256.85 304.87 Within 823,345.24 1,279 643.74 Letter sound Total 1,215,858.95 1,281 196,900.59 knowledge Gender Between 7,657.61 1 7,657.61 8.11 Within 1,208,201.34 1,280 943.91 Total 1,215,858.95 1,281 8,601.52 Grade Between 101,947.59 2 50,973.79 225.61 Within 288,754.69 1,278 225.94 Familiar word reading Total 390,702.27 1,280 51,199.74 Gender Between 2,203.07 1 2,203.07 7.25 75 76 76 Sub-test Factor Sum of Squares df Mean Square F Within 388,499.20 1,279 303.75 Total 390,702.27 1,280 2,506.82 Grade Between 26,493.67 2 13,246.83 125.61 Within 134,882.06 1,279 105.46 Total 161,375.73 1,281 13,352.29 Invented word reading Gender Between 300.40 1 300.40 2.39 Within 161,075.32 1,280 125.84 Total 161,375.73 1,281 426.24 Grade Between 187,579.51 2 93,789.76 264.82 Within 452,968.53 1,279 354.16 Total 640,548.04 1,281 94,143.91 Oral reading passage Gender Between 4,021.90 1 4,021.90 8.09 Within 636,526.14 1,280 497.29 Total 640,548.04 1,281 4,519.19 Grade Between 875.03 2 437.51 291.36 Within 1,920.55 1,279 1.50 Reading Total 2,795.58 1,281 439.02 comprehension Gender Between 7.00 1 7.00 3.21 Within 2,788.58 1,280 2.18 Total 2,795.58 1,281 9.18 Grade Between 729.53 2 364.76 137.10 Listening Within 3,402.88 1,279 2.66 comprehension Total 4,132.40 1,281 367.42 Gender Between 1.55 1 1.55 0.48 77 Sub-test Factor Sum of Squares df Mean Square F Within 4,130.85 1,280 3.23 Total 4,132.40 1,281 4.78 Grade Between 2,040.64 2 1,020.32 280.37 Within 4,654.58 1,279 3.64 Total 6,695.22 1,281 1,023.96 Dictation Gender Between 33.48 1 33.48 6.43 Within 6,661.74 1,280 5.20 Total 6,695.22 1,281 38.69 77 78 Table Summary of student factors affecting reading students 78 36 – 35 Table – Summary of student factors affecting outcomes of readingoutcomes AnglophoneNi-Vanuatu of Anglophone Ni-Vanuatustudents Student factor Overall effect By Grade By Gender Positive and statistically significant for Grade 1 students only, for letter name Positive and statistically significant for Positive and statistically significant effect Table 5.1 - Student speaks knowledge, letter sound knowledge, boys only, in all sub-tests except initial for all sub-tests except for reading and English at home initial sound recognition, listening sound recognition and listening listening comprehension. comprehension and the weighted comprehension. dictation score. Table 5.2 - Student speaks Positive and statistically significant effect No statistical differences No statistical differences Bislama or vernacular at home only for listening comprehension. Positive and statistically significant in Table 5.3 - Student has the Positive and statistically significant effect Grade 1 only for the weighted dictation Positive and statistically significant for school textbook for all sub-tests. score; and in Grade 3 for the listening boys and girls, in all sub-tests. comprehension score. Positive and statistically significant in (a) Positive and statistically significant for Grades 2 for letter name and sound Table 5.4 - Student has a Positive and statistically significant effect boys for initial sound recognition, and knowledge, invented word reading, and teacher that reads aloud at for all sub-tests except for the weighted reading and listening comprehension; reading and listening comprehension; and school dictation score. positive and statistically significant for in Grade 3 only for familiar word reading girls for all sub-tests. and listening comprehension. Positive and statistically significant in Table 5.5 - Student has books Grade 2 for reading and listening Positive and statistically significant effect Positive and statistically significant for and/or reading materials at comprehension and in Grade 3 in all sub- for all sub-tests. boys and girls, in all sub-tests. home tests, except letter name knowledge and the weighted dictation score. Positive and statistically significant in Table 5.6 - Student has reading Positive and statistically significant effect Grade 2 for all sub-tests except familiar Positive and statistically significant for materials at home in English for all sub-tests. and invented word reading; and in Grade boys and girls in all sub-tests. language 3 for all sub-tests. 79 Student factor Overall effect By Grade By Gender Negative and statistically significant for Negative and statistically significant effect Negative and statistically significant in boys in initial sound recognition, and Table 5.7 - Student has reading for letter name knowledge, letter sound Grade 1 only for reading and listening reading and listening comprehension; for materials at home in Bislama knowledge, and reading comprehension. comprehension. girls, same effect in letter name knowledge only. Positive and statistically significant in Grade 1 for all sub-tests except invented Positive and statistically significant effect Positive and statistically significant for word reading, reading comprehension; in Table 5.8 - Student has a for letter name knowledge, letter sound boys only in the weighted dictation score; Grade 2, same effect for all except for literate mother knowledge, initial sound recognition, and and for girls, only for letter name listening comprehension; and in Grade 3 the weighted dictation score. knowledge and initial sound recognition. for listening comprehension and the weighted dictation score. Positive and statistically significant in Positive and statistically significant for Grade 1 for letter name and sound boys only in initial sound recognition, Table 5.9 - Student has a Positive and statistically significant effect knowledge, initial sound recognition and listening comprehension and the literate father for all sub-tests. listening comprehension; and in Grade 2 weighted dictation score; and for girls, for all sub-tests except for listening positive and significant for all sub-tests. comprehension. Positive and statistically significant for Positive and statistically significant effect Positive and statistically significant in boys only in initial sound recognition, for letter name knowledge, letter sound Grade 1 for letter name and sound listening comprehension and the Table 5.10 - Student parents knowledge, initial sound recognition, knowledge, initial sound recognition and weighted dictation score; and for girls, in are both literate invented word reading and listening listening comprehension; and in Grade 2 all sub-tests except familiar and invented comprehension. for all sub-tests. word reading, and the oral reading passage. Positive and statistically significant in Grade 1 for letter name and initial sound Positive and statistically significant for Positive and statistically significant effect recognition, the oral reading passage and Table 5.11 - Student has a boys in all sub-tests; positive and for all subtests except for invented word listening comprehension; in Grade 2 for literate sibling significant for girls in initial sound reading. letter name and sound knowledge and recognition only. initial sound recognition; and in Grade 3, in listening comprehension only. 79 80 80 Student factor Overall effect By Grade By Gender Positive and statistically significant for Positive and statistically significant in boys in all sub-tests except letter name Grade 1 for the weighted dictation score; knowledge and initial sound recognition; Table 5.12 - Student has a Positive and statistically significant effect in Grade 2 for letter sound knowledge, for girls, positive and significant for girls in family member that reads at for all sub-tests. reading comprehension and weighted letter name and sounds knowledge, initial home with him/her dictation; and in Grade 3 only for listening sound recognition, the oral reading comprehension. passage and reading and listening comprehension. Positive and statistically significant in Grade 1 for the listening comprehension; Positive and statistically significant for Positive and statistically significant effect in Grade 2 for all sub-tests except initial Table 5.13 - Student reads at boys for listening comprehension only; for all sub-test except initial sound sound recognition and the weighted home with his/her mother positive and significant for girls in all sub- identification. dictation; and in Grade 3 for all sub-tests tests except initial sound recognition. except letter name knowledge and initial sound recognition. Positive and statistically significant in Positive and statistically significant for Positive and statistically significant effect Grade 2 for reading comprehension and boys for all sub-tests except letter sound Table 5.14 - Student reads at for all sub-tests except initial sound the weighted dictation score; and in knowledge and listening comprehension; home with his/her father identification, letter sound knowledge Grade 3 for initial sound recognition, the positive and significant for girls only in and listening comprehension. oral reading passage and reading reading comprehension. comprehension. Negative and statistically significant in Grade 2 only in the familiar and invented word reading, the oral reading passage, Table 5.15 - Student reads at Negative and statistically significant only reading comprehension and the weighted No statistical differences. home with his/her sibling(s) for invented word reading. dictation score; in Grade 3, negative and significant for invented word reading and the oral reading passage. 81 Student factor Overall effect By Grade By Gender Positive and statistically significant for boys in all sub-tests except for letter Positive and statistically significant in Positive and statistically significant for all name and sound knowledge, the Grade 2 for letter name knowledge, initial Table 5.16 - Student attended sub-tests except for letter sound weighted dictation score; for girls, sound recognition, familiar and invented kindergarten before Grade 1 knowledge and the weighted dictation positive and significant for all sub-tests word reading, the oral reading passage score. except for letter sound knowledge, and the weighted dictation score. familiar word reading, and listening comprehension. Positive and statistically significant for Grade 1 students in all sub-tests except Positive and statistically significant for Table 5.17 - Student does Positive and statistically significant effect for listening comprehension; no effect in boys in all sub-tests except for the homework for all sub-tests. Grade 2; and in Grade 3, positive and weighted dictation score; and for girls, significant only for initial sound positive and significant in all sub-tests. recognition and reading comprehension. Positive and statistically significant in Grade 1 for all sub-tests except initial Positive and statistically significant for sound recognition, familiar and invented boys in all sub-tests except for initial Table 5.18 - Student receives word reading, the weighted dictation sound recognition, invented word Positive and statistically significant effect help from family member with score; in Grade 2, significant only in the reading, and listening comprehension; for for all sub-tests. his/her homework oral reading passage; and in Grade 3, girls, positive and significant only letter significant for all except for invented name knowledge, invented word reading word reading, and listening and reading comprehension. comprehension. Positive and statistically significant in Positive and significant for boys in all sub- Grade 1 for letter name and sound tests except for letter sound knowledge, knowledge, the oral reading passage and invented word reading, and listening Table 5.19 - Student receives Positive and statistically significant for all reading and listening comprehension; in comprehension; for girls, positive and help from his/her mother to do sub-tests except for letter sound Grade 2, only in the oral reading passage; significant only for letter name homework knowledge. and in Grade 3, for all except for invented knowledge, invented word reading, the word reading and listening oral reading passage and reading comprehension. comprehension. 81 82 82 Student factor Overall effect By Grade By Gender Positive and statistically significant for Positive and statistically significant for all boys only for letter name and sound Table 5.20 - Student receives Positive and statistically significant in sub-tests except for invented word knowledge; positive and significant for help from his/her father to do Grade 2 only, for letter name and sound reading, listening comprehension and the girls in all sub-tests except for invented homework knowledge. weighted dictation score. word reading and listening comprehension. Table 5.21 - Student receives Negative and statistically significant in Negative and statistically significant for help from his/her sibling(s) to No statistical differences Grade 2 for invented word reading and girls only, for letter name knowledge. do homework the weighted dictation score. Negative and statistically significant for Negative and statistically significant for boys for invented word reading, the oral Table 5.22 - Student reported Negative and statistically significant for all Grade 2 for reading comprehension; and reading passage and reading being absent for more than one subtests except for letter sound in Grade 3, for invented word reading, the comprehension; for girls, negative and week in the school year knowledge and initial sound recognition. oral reading passage and the weighted significant for familiar word reading, the dictation score. oral reading passage and the weighted dictation score. Table 5.23 - Student is overage No statistical differences No statistical differences No statistical differences (as a proxy for repetition) Negative and statistically significant in Negative and statistically significant for Grade 2 for all sub-tests except for initial boys in letter name and sounds Negative and statistically significant for all Table 5.24 - Grade retention sound identification, reading knowledge, initial sound recognition and sub-tests. comprehension, and the weighted reading comprehension; negative and dictation score. significant for girls in all sub-tests. 83 Table 37 - Results from differences in average scores determined by factor (tables 5.1 to 5.24) Table 36 - Results from differences in average scores determined by factor (tables 5.1 to 5.24) GRADE GENDER Table 5.1 - Student speaks English at With Without With Without home Grade 1 Grade 2 Grade 3 Grade 1 Grade 2 Grade 3 Boys Girls Boys Girls Letter name knowledge 13 21 31 7 19 29 20 22 16 19 Initial sound recognition 3 5 7 2 5 7 5 5 5 4 Letter sound knowledge 28 44 68 19 43 63 45 47 38 44 Familiar word reading 3 11 25 1 10 23 13 13 9 13 Invented word reading 3 6 13 1 5 12 8 7 5 7 Words in the oral reading passage 5 15 34 2 13 31 17 18 12 17 Reading comprehension 3 16 45 2 16 41 21 20 16 21 Listening comprehension 33 50 62 22 43 60 47 49 42 40 Weighted dictation 9 23 51 4 24 48 29 25 22 28 Note: bold numbers denote significance at p < 5%; bold and italicized denote significance at p < 1%. GRADE GENDER Table 5.2 - Student speaks Bislama or With Without With Without vernacular at home Grade 1 Grade 2 Grade 3 Grade 1 Grade 2 Grade 3 Boys Girls Boys Girls Letter name knowledge 8 19 29 11 21 32 17 20 18 21 Initial sound recognition 3 5 7 2 5 7 5 5 4 5 Letter sound knowledge 21 43 64 25 46 63 40 45 38 44 Familiar word reading 2 10 23 3 11 28 10 13 11 13 Invented word reading 1 5 12 1 6 14 6 7 6 6 Words in the oral reading passage 3 13 31 4 15 39 13 17 16 17 Reading comprehension 2 16 41 4 14 44 18 21 18 18 Listening comprehension 26 47 62 19 36 53 45 44 35 32 Weighted dictation 5 23 49 8 26 53 23 27 25 27 83 84 84 GRADE GENDER Table 5.3 - Student has the school With Without With Without textbook Grade 1 Grade 2 Grade 3 Grade 1 Grade 2 Grade 3 Boys Girls Boys Girls Letter name knowledge 9 20 30 7 17 25 19 21 9 15 Initial sound recognition 3 5 7 2 5 6 5 5 3 4 Letter sound knowledge 23 44 64 18 40 61 42 47 26 35 Familiar word reading 2 10 23 1 9 21 11 14 5 8 Invented word reading 1 6 12 1 4 11 6 7 2 4 Words in the oral reading passage 3 14 32 1 12 31 15 18 6 11 Reading comprehension 3 16 42 1 14 43 19 22 9 13 Listening comprehension 25 45 62 25 44 45 46 43 31 37 Weighted dictation 7 24 50 2 21 39 26 29 12 15 Note: bold numbers denote significance at p < 5%; bold and italicized denote significance at p < 1%. GRADE GENDER Table 5.4 - Student has a teacher that With Without With Without reads aloud at school Grade 1 Grade 2 Grade 3 Grade 1 Grade 2 Grade 3 Boys Girls Boys Girls Letter name knowledge 9 20 29 8 9 25 18 20 13 10 Initial sound recognition 3 5 7 2 2 6 5 5 2 3 Letter sound knowledge 22 44 64 21 32 50 40 46 30 29 Familiar word reading 2 10 24 2 6 12 10 13 6 3 Invented word reading 1 6 12 1 2 8 6 7 4 1 Words in the oral reading passage 3 13 32 2 7 19 14 18 8 5 Reading comprehension 2 16 42 2 6 23 18 22 8 5 Listening comprehension 26 46 62 17 30 39 45 43 24 25 Weighted dictation 5 24 50 8 20 35 24 28 18 15 Note: bold numbers denote significance at p < 5%; bold and italicized denote significance at p < 1%. 85 GRADE GENDER Table 5.5 - Student has books and/or With Without With Without reading materials at home Grade 1 Grade 2 Grade 3 Grade 1 Grade 2 Grade 3 Boys Girls Boys Girls Letter name knowledge 10 21 31 8 17 26 20 22 14 17 Initial sound recognition 3 5 7 2 5 6 5 5 4 4 Letter sound knowledge 24 46 67 19 40 56 44 49 32 38 Familiar word reading 2 12 26 1 8 17 12 16 6 8 Invented word reading 2 7 13 1 4 9 7 8 3 4 Words in the oral reading passage 3 15 34 2 10 25 17 21 9 12 Reading comprehension 2 20 45 2 10 33 22 26 11 13 Listening comprehension 27 51 66 23 36 49 48 49 36 32 Weighted dictation 7 26 52 4 20 42 28 31 16 20 Note: bold numbers denote significance at p < 5%; bold and italicized denote significance at p < 1%. Table 5.6 - Student has reading GRADE GENDER materials at home in English With Without With Without language Grade 1 Grade 2 Grade 3 Grade 1 Grade 2 Grade 3 Boys Girls Boys Girls Letter name knowledge 11 23 32 8 17 25 22 26 13 15 Initial sound recognition 3 6 7 2 4 6 5 6 4 4 Letter sound knowledge 26 48 68 20 39 57 48 54 31 37 Familiar word reading 3 13 27 1 8 17 14 19 6 8 Invented word reading 2 7 14 1 4 9 8 9 3 4 Words in the oral reading passage 4 17 36 2 10 25 19 24 8 12 Reading comprehension 3 22 47 2 10 33 26 31 10 13 Listening comprehension 30 54 67 22 38 50 53 54 34 33 Weighted dictation 8 28 54 4 20 41 31 37 16 19 Note: bold numbers denote significance at p < 5%; bold and italicized denote significance at p < 1%. 85 86 86 GRADE GENDER Table 5.7 - Student has reading With Without With Without materials at home in Bislama Grade 1 Grade 2 Grade 3 Grade 1 Grade 2 Grade 3 Boys Girls Boys Girls Letter name knowledge 6 18 28 9 20 29 14 15 18 21 Initial sound recognition 2 4 7 3 5 7 4 4 5 5 Letter sound knowledge 21 43 70 22 43 63 34 42 40 45 Familiar word reading 1 10 26 2 10 23 8 10 10 13 Invented word reading 1 7 13 1 5 12 4 6 6 7 Words in the oral reading passage 2 14 34 3 13 31 10 15 14 18 Reading comprehension 0 15 40 3 16 42 11 16 19 22 Listening comprehension 19 47 66 26 45 60 34 40 45 43 Weighted dictation 3 25 53 6 23 49 20 21 24 28 Note: bold numbers denote significance at p < 5%; bold and italicized denote significance at p < 1%. GRADE GENDER Table 5.8 - Student has a literate With Without With Without mother Grade 1 Grade 2 Grade 3 Grade 1 Grade 2 Grade 3 Boys Girls Boys Girls Letter name knowledge 11 24 31 6 16 28 19 23 17 17 Initial sound recognition 3 6 7 2 4 6 5 5 4 4 Letter sound knowledge 26 50 67 18 39 62 42 48 38 42 Familiar word reading 3 13 25 1 8 22 11 14 10 12 Invented word reading 2 8 13 1 4 12 6 7 5 6 Words in the oral reading passage 4 17 33 1 10 31 15 18 13 16 Reading comprehension 4 24 44 1 10 40 20 23 16 19 Listening comprehension 29 48 67 20 43 57 47 45 41 40 Weighted dictation 7 32 56 4 18 45 27 31 21 24 Note: bold numbers denote significance at p < 5%; bold and italicized denote significance at p < 1%. 87 GRADE GENDER Table 5.9 - Student has a literate With Without With Without father Grade 1 Grade 2 Grade 3 Grade 1 Grade 2 Grade 3 Boys Girls Boys Girls Letter name knowledge 11 24 30 7 17 29 20 23 16 18 Initial sound recognition 4 6 7 2 4 7 5 6 4 4 Letter sound knowledge 25 49 66 20 40 62 42 51 38 41 Familiar word reading 3 14 24 1 8 23 12 15 9 11 Invented word reading 2 8 12 1 4 12 7 8 5 6 Words in the oral reading passage 4 18 32 2 10 32 16 20 12 15 Reading comprehension 4 24 41 1 10 42 20 25 16 17 Listening comprehension 34 48 64 19 44 59 48 48 41 38 Weighted dictation 7 32 52 5 18 47 27 33 21 23 Note: bold numbers denote significance at p < 5%; bold and italicized denote significance at p < 1%. GRADE GENDER Table 5.10 - Student parents are both With Without With Without literate Grade 1 Grade 2 Grade 3 Grade 1 Grade 2 Grade 3 Boys Girls Boys Girls Letter name knowledge 12 28 30 7 17 29 21 24 16 18 Initial sound recognition 4 7 7 2 4 6 6 6 4 4 Letter sound knowledge 27 55 68 20 40 62 44 52 38 42 Familiar word reading 3 16 25 1 8 23 12 15 9 12 Invented word reading 2 10 12 1 4 12 7 8 5 6 Words in the oral reading passage 5 21 33 2 10 31 17 20 12 16 Reading comprehension 5 29 43 1 11 41 22 26 16 19 Listening comprehension 35 53 70 20 43 57 52 51 40 39 Weighted dictation 8 37 56 4 19 46 29 34 21 24 Note: bold numbers denote significance at p < 5%; bold and italicized denote significance at p < 1%. 87 88 88 GRADE GENDER Table 5.11 - Student has a literate With Without With Without sibling Grade 1 Grade 2 Grade 3 Grade 1 Grade 2 Grade 3 Boys Girls Boys Girls Letter name knowledge 10 22 30 8 17 29 21 21 15 19 Initial sound recognition 3 6 7 2 4 6 5 5 4 4 Letter sound knowledge 25 46 66 20 41 62 46 47 34 43 Familiar word reading 2 11 25 2 9 22 12 14 8 12 Invented word reading 1 6 12 1 5 12 7 7 5 6 Words in the oral reading passage 3 14 33 2 12 31 16 18 11 16 Reading comprehension 3 18 42 2 14 41 22 21 14 20 Listening comprehension 30 46 69 21 44 53 54 44 34 41 Weighted dictation 7 25 52 4 22 47 28 29 19 25 Note: bold numbers denote significance at p < 5%; bold and italicized denote significance at p < 1%. Table 5.12 - Student has a family GRADE GENDER member that reads at home with With Without With Without him/her Grade 1 Grade 2 Grade 3 Grade 1 Grade 2 Grade 3 Boys Girls Boys Girls Letter name knowledge 9 20 30 7 15 28 18 21 14 15 Initial sound recognition 3 5 7 2 4 7 5 5 4 4 Letter sound knowledge 23 45 65 17 34 58 41 46 30 35 Familiar word reading 2 10 24 1 7 20 11 13 7 9 Invented word reading 1 6 12 1 5 11 6 7 3 6 Words in the oral reading passage 3 14 32 2 10 29 14 18 10 13 Reading comprehension 3 17 43 1 9 33 19 22 10 14 Listening comprehension 26 45 63 21 43 44 45 44 34 32 Weighted dictation 6 25 50 3 13 41 25 28 14 17 Probability of fluency 0% 6% 23% 0% 2% 18% 8% 11% 3% 8% Note: bold numbers denote significance at p < 5%; bold and italicized denote significance at p < 1%. 89 GRADE GENDER Table 5.13 - Student reads at home With Without With Without with his/her mother Grade 1 Grade 2 Grade 3 Grade 1 Grade 2 Grade 3 Boys Girls Boys Girls Letter name knowledge 11 24 32 8 17 28 19 24 17 18 Initial sound recognition 3 5 7 3 5 7 5 5 5 5 Letter sound knowledge 24 51 69 21 40 62 43 51 38 42 Familiar word reading 3 13 28 2 9 21 12 16 9 11 Invented word reading 2 8 15 1 5 11 7 9 5 6 Words in the oral reading passage 3 17 38 2 11 29 15 22 13 15 Reading comprehension 3 22 47 2 13 39 20 25 17 19 Listening comprehension 30 53 66 22 42 59 49 47 41 40 Weighted dictation 7 30 56 5 21 46 26 32 22 25 Note: bold numbers denote significance at p < 5%; bold and italicized denote significance at p < 1%. GRADE GENDER Table 5.14 - Student reads at home With Without With Without with his/her father Grade 1 Grade 2 Grade 3 Grade 1 Grade 2 Grade 3 Boys Girls Boys Girls Letter name knowledge 9 22 32 9 19 29 21 21 17 20 Initial sound recognition 3 5 7 3 5 6 5 5 4 5 Letter sound knowledge 22 46 67 22 43 63 44 46 39 44 Familiar word reading 3 13 27 2 9 22 15 14 9 12 Invented word reading 2 7 14 1 5 11 8 7 5 6 Words in the oral reading passage 4 16 37 3 12 30 20 19 12 17 Reading comprehension 3 22 49 2 14 39 25 25 16 20 Listening comprehension 26 42 66 25 46 59 48 43 42 42 Weighted dictation 6 31 55 6 22 47 32 30 21 26 Note: bold numbers denote significance at p < 5%; bold and italicized denote significance at p < 1%. 89 90 90 GRADE GENDER Table 5.15 - Student reads at home With Without With Without with his/her sibling(s) Grade 1 Grade 2 Grade 3 Grade 1 Grade 2 Grade 3 Boys Girls Boys Girls Letter name knowledge 8 18 28 9 21 30 17 19 18 21 Initial sound recognition 3 5 6 3 5 7 5 5 5 5 Letter sound knowledge 23 42 62 21 44 65 42 45 39 45 Familiar word reading 2 8 21 2 11 25 9 12 11 13 Invented word reading 1 4 10 1 7 13 5 6 6 7 Words in the oral reading passage 3 11 27 3 15 34 12 16 14 18 Reading comprehension 2 12 38 2 18 44 16 20 19 21 Listening comprehension 23 42 63 26 47 59 44 43 43 42 Weighted dictation 7 19 47 5 26 50 23 27 24 27 Note: bold numbers denote significance at p < 5%; bold and italicized denote significance at p < 1%. GRADE GENDER Table 5.16 - Student attended With Without With Without kindergarten before Grade 1 Grade 1 Grade 2 Grade 3 Grade 1 Grade 2 Grade 3 Boys Girls Boys Girls Letter name knowledge 9 20 30 8 13 24 18 20 14 14 Initial sound recognition 3 5 7 2 4 6 5 5 4 3 Letter sound knowledge 22 44 64 21 40 62 40 45 37 38 Familiar word reading 2 10 24 2 6 18 10 13 7 9 Invented word reading 1 6 12 1 3 9 6 7 3 4 Words in the oral reading passage 3 14 32 2 9 24 14 18 9 11 Reading comprehension 2 16 42 1 10 35 18 21 12 14 Listening comprehension 25 46 61 21 39 57 45 43 34 38 Weighted dictation 5 24 49 8 17 44 24 28 23 18 Note: bold numbers denote significance at p < 5%; bold and italicized denote significance at p < 1%. 91 GRADE GENDER Table 5.17 - Student does homework With Without With Without Grade 1 Grade 2 Grade 3 Grade 1 Grade 2 Grade 3 Boys Girls Boys Girls Letter name knowledge 9 20 30 5 17 20 18 21 11 6 Initial sound recognition 3 5 7 2 5 3 5 5 3 2 Letter sound knowledge 23 43 64 14 46 48 41 47 28 20 Familiar word reading 2 10 24 1 9 15 11 14 5 2 Invented word reading 1 6 12 0 5 10 6 7 3 1 Words in the oral reading passage 3 13 32 1 11 24 14 18 7 4 Reading comprehension 3 16 42 1 16 22 19 22 9 3 Listening comprehension 26 46 61 19 37 49 45 44 29 22 Weighted dictation 6 24 49 2 23 42 24 29 14 7 Note: bold numbers denote significance at p < 5%; bold and italicized denote significance at p < 1%. Table 5.18 - Student receives help GRADE GENDER from family member with his/her With Without With Without homework Grade 1 Grade 2 Grade 3 Grade 1 Grade 2 Grade 3 Boys Girls Boys Girls Letter name knowledge 10 21 34 7 18 26 20 23 16 18 Initial sound recognition 3 5 7 2 5 6 5 5 4 4 Letter sound knowledge 25 45 70 20 42 59 45 47 36 43 Familiar word reading 3 11 27 1 9 21 12 14 8 11 Invented word reading 2 7 14 1 5 11 7 8 5 6 Words in the oral reading passage 4 15 36 2 12 28 16 19 11 15 Reading comprehension 4 18 48 1 14 36 22 24 15 18 Listening comprehension 29 47 64 21 44 58 47 45 41 40 Weighted dictation 7 26 57 4 22 43 28 30 20 24 Note: bold numbers denote significance at p < 5%; bold and italicized denote significance at p < 1%. 91 92 92 Table 5.19 - Student receives help GRADE GENDER from his/her mother to do With Without With Without homework Grade 1 Grade 2 Grade 3 Grade 1 Grade 2 Grade 3 Boys Girls Boys Girls Letter name knowledge 10 21 34 7 18 26 20 23 16 18 Initial sound recognition 3 5 7 2 5 6 5 5 4 4 Letter sound knowledge 24 45 69 20 42 59 44 47 36 43 Familiar word reading 3 11 26 1 9 21 12 14 8 11 Invented word reading 2 7 13 1 5 11 6 8 5 6 Words in the oral reading passage 4 15 36 2 12 28 16 19 11 15 Reading comprehension 4 18 48 1 14 36 22 24 15 18 Listening comprehension 29 47 64 21 44 58 47 45 41 40 Weighted dictation 7 26 57 4 22 43 28 30 20 24 Note: bold numbers denote significance at p < 5%; bold and italicized denote significance at p < 1%. GRADE GENDER Table 5.20 - Student receives help With Without With Without from his/her father to do homework Grade 1 Grade 2 Grade 3 Grade 1 Grade 2 Grade 3 Boys Girls Boys Girls Letter name knowledge 12 24 29 8 18 29 21 24 16 19 Initial sound recognition 3 6 7 3 5 6 5 5 4 4 Letter sound knowledge 24 47 63 21 42 64 44 51 38 43 Familiar word reading 3 12 26 2 9 22 13 17 9 11 Invented word reading 2 7 12 1 5 12 7 8 5 6 Words in the oral reading passage 4 16 33 2 12 31 17 22 12 16 Reading comprehension 4 20 42 2 14 42 21 26 17 19 Listening comprehension 25 47 60 25 45 61 47 45 42 41 Weighted dictation 7 29 46 5 22 50 26 33 23 25 Note: bold numbers denote significance at p < 5%; bold and italicized denote significance at p < 1%. 93 Table 5.21 - Student receives help GRADE GENDER from his/her sibling(s) to do With Without With Without homework Grade 1 Grade 2 Grade 3 Grade 1 Grade 2 Grade 3 Boys Girls Boys Girls Letter name knowledge 9 15 27 9 21 30 17 17 18 21 Initial sound recognition 3 5 7 3 5 7 5 5 5 5 Letter sound knowledge 23 40 63 21 45 64 40 45 40 44 Familiar word reading 2 8 21 2 11 24 9 11 10 13 Invented word reading 1 4 11 1 6 13 5 6 6 7 Words in the oral reading passage 3 10 28 3 14 33 12 15 14 18 Reading comprehension 2 11 37 2 18 43 15 18 19 22 Listening comprehension 25 45 60 25 45 61 47 39 42 43 Weighted dictation 5 17 46 6 26 50 22 24 24 28 Note: bold numbers denote significance at p < 5%; bold and italicized denote significance at p < 1%. 5.22 - Student reported being absent GRADE GENDER for more than one week in the With Without With Without school year Grade 1 Grade 2 Grade 3 Grade 1 Grade 2 Grade 3 Boys Girls Boys Girls Letter name knowledge 9 18 28 9 21 31 17 19 19 21 Initial sound recognition 3 5 6 3 5 7 5 4 5 5 Letter sound knowledge 21 43 62 22 44 66 38 43 41 46 Familiar word reading 2 10 21 2 10 25 9 11 11 14 Invented word reading 1 5 10 1 6 14 5 6 6 7 Words in the oral reading passage 2 12 29 3 14 34 12 15 15 19 Reading comprehension 2 14 40 2 17 43 16 19 19 22 Listening comprehension 23 46 58 27 45 63 42 39 45 45 Weighted dictation 5 21 45 6 25 52 22 23 25 30 Note: bold numbers denote significance at p < 5%; bold and italicized denote significance at p < 1%. 93 94 94 GRADE GENDER 5.23 - Student is overage (as a proxy With Without With Without for repetition) Grade 1 Grade 2 Grade 3 Grade 1 Grade 2 Grade 3 Boys Girls Boys Girls Letter name knowledge 8 17 28 9 22 31 17 19 18 21 Initial sound recognition 3 5 6 3 5 7 5 4 4 5 Letter sound knowledge 24 42 61 21 45 67 42 45 37 44 Familiar word reading 2 8 22 2 12 25 10 13 10 13 Invented word reading 1 4 12 1 7 12 6 7 6 6 Words in the oral reading passage 3 11 29 3 15 35 13 17 14 17 Reading comprehension 3 13 38 2 18 46 17 21 18 20 Listening comprehension 26 43 58 24 47 64 44 43 43 42 Weighted dictation 6 20 48 6 27 50 23 30 25 25 Note: bold numbers denote significance at p < 5%; bold and italicized denote significance at p < 1%. GRADE GENDER 5.24 - Grade retention With Without With Without Grade 1 Grade 2 Grade 3 Grade 1 Grade 2 Grade 3 Boys Girls Boys Girls Letter name knowledge 9 13 29 9 20 29 12 15 18 21 Initial sound recognition 3 5 5 3 5 7 4 4 5 5 Letter sound knowledge 21 33 66 22 45 64 32 33 41 46 Familiar word reading 2 6 26 2 10 23 7 7 10 14 Invented word reading 1 4 15 1 6 12 5 4 6 7 Words in the oral reading passage 2 7 37 3 14 31 10 10 14 18 Reading comprehension 2 7 44 2 17 41 10 11 19 22 Listening comprehension 30 37 55 24 46 61 40 33 44 43 Weighted dictation 5 15 53 6 24 49 18 15 24 29 Note: bold numbers denote significance at p < 5%; bold and italicized denote significance at p < 1%. 95 38 School Table Table and Teacher Characteristics: Mean Subtest Scores 37 School and Teacher Characteristics: Mean Subtest Scores Do you hold a Certificate in Primary Teaching? ORF RCOMP % LCOMP % DICT % Mean 26.157 0.308 0.538 0.401 No SD 30.211 0.346 0.350 0.325 N 288 288 288 288 Mean 14.830 0.190 0.433 0.268 Yes SD 20.941 0.291 0.365 0.283 N 784 785 785 785 Mean 17.873 0.222 0.461 0.304 Total SD 24.299 0.311 0.364 0.300 N 1072 1073 1073 1073 t-statistic -5.866 -5.136 -4.329 -6.123 d.f. 3.717 0.001 0.001 0.000 p-value t test 0.005 0.061 0.072 0.052 p-value z test 0.000 0.000 0.000 0.000 95 Table 39 - School and Teacher Characteristics: Mean Subtest Scores Table 38 School and Teacher Characteristics: Mean Subtest Scores Highest Qualification RCOMP LCOMP DICT ORF % % % Mean 29.295 0.355 0.539 0.440 None SD 33.092 0.371 0.356 0.327 N 166 166 166 166 Mean 14.902 0.189 0.425 0.270 Certificate in Primary SD 21.084 0.289 0.364 0.285 Teaching N 726 727 727 727 Mean 15.413 0.187 0.507 0.240 Certificate in Education SD 20.259 0.306 0.352 0.248 N 60 60 60 60 Mean 18.951 0.220 0.510 0.325 Other SD 24.640 0.285 0.339 0.304 N 118 118 118 118 Mean 17.610 0.218 0.456 0.301 Total SD 24.213 0.309 0.362 0.298 N 1070 1071 1071 1071 - t-statistic -5.361 -5.422 -3.699 6.181 d.f. 7.168 0.001 0.001 0.001 p-value t 0.001 0.058 0.084 0.051 test p-value z 0.000 0.000 0.000 0.000 test 96 97 Table 40 - School and Teacher Characteristics: Mean Subtest Scores Table 39 School and Teacher Characteristics: Mean Subtest Scores Have you attended an inservice? ORF RCOMP % LCOMP % DICT % Mean 15.593 0.202 0.454 0.277 No SD 20.601 0.292 0.362 0.279 N 802 802 802 802 Mean 24.329 0.277 0.488 0.372 Yes SD 31.563 0.351 0.366 0.334 N 279 280 280 280 Mean 17.848 0.221 0.463 0.302 Total SD 24.204 0.310 0.363 0.297 N 1081 1082 1082 1082 t-statistic 4.314 3.227 1.322 4.261 d.f. 4.086 0.001 0.001 0.000 p-value t test 0.006 0.096 0.206 0.073 p-value z test 0.000 0.001 0.093 0.000 Have You Attended a Reading Training? ORF RCOMP % LCOMP % DICT % Mean 16.119 0.208 0.448 0.280 No SD 21.958 0.300 0.361 0.286 N 899 899 899 899 Mean 26.390 0.287 0.539 0.406 Yes SD 31.863 0.349 0.366 0.328 N 182 183 183 183 Mean 17.848 0.221 0.463 0.302 Total SD 24.204 0.310 0.363 0.297 N 1081 1082 1082 1082 t-statistic 4.154 2.881 3.081 4.809 d.f. 6.083 0.001 0.001 0.001 p-value t test 0.003 0.106 0.100 0.065 p-value z test 0.000 0.002 0.001 0.000 97 Table 41 - School and Teacher Characteristics: Mean Subtest Scores Table 40 School and Teacher Characteristics: Mean Subtest Scores Experience Level ORF RCOMP % LCOMP % DICT % Mean 24.574 0.305 0.512 0.390 0-4 years SD 29.306 0.342 0.354 0.314 N 222 222 222 222 Mean 21.752 0.254 0.501 0.348 5-9 years SD 27.236 0.323 0.363 0.307 N 198 199 199 199 Mean 16.594 0.216 0.456 0.294 10-19 years SD 23.211 0.315 0.371 0.308 N 296 296 296 296 Mean 15.722 0.201 0.442 0.273 20-29 years SD 19.357 0.287 0.367 0.274 N 268 268 268 268 Mean 4.923 0.044 0.313 0.143 30-39 years SD 8.283 0.123 0.295 0.196 N 90 90 90 90 Mean 9.116 0.112 0.307 0.171 40+ years SD 18.397 0.245 0.328 0.231 N 208 208 208 208 Mean 16.558 0.205 0.435 0.284 Total SD 23.618 0.304 0.361 0.294 N 1282 1283 1283 1283 t-statistic -1.023 -1.579 -0.320 -1.393 d.f. 7.579 0.001 0.001 0.001 p-value t test 0.170 0.180 0.401 0.198 p-value z test 0.153 0.057 0.374 0.082 98 Table 42 - School and Teacher Characteristics: Mean Subtest Scores Table 41 School and Teacher Characteristics: Mean Subtest Scores Library Present ORF RCOMP % LCOMP % DICT % Mean 13.565 0.175 0.366 0.264 No SD 20.035 0.287 0.350 0.294 N 286 286 286 286 Mean 20.112 0.249 0.506 0.325 Yes SD 25.669 0.321 0.361 0.303 N 770 771 771 771 Mean 8.237 0.084 0.295 0.237 99 missing code SD 10.323 0.172 0.289 0.228 N 38 38 38 38 Mean 17.988 0.224 0.462 0.306 Total SD 24.155 0.311 0.362 0.300 N 1094 1095 1095 1095 t-statistic 4.356 3.618 5.691 2.967 d.f. 2.253 0.000 0.001 0.000 p-value t test 0.024 0.086 0.055 0.103 p-value z test 0.000 0.000 0.000 0.002 Supervision in Library ORF RCOMP % LCOMP % DICT % Mean 15.861 0.197 0.430 0.279 No SD 19.512 0.274 0.375 0.276 N 60 60 60 60 Mean 22.145 0.264 0.507 0.352 Yes SD 27.860 0.332 0.361 0.316 N 508 509 509 509 Mean 21.481 0.257 0.499 0.345 Total SD 27.155 0.326 0.363 0.312 N 568 569 569 569 t-statistic 2.240 1.747 1.515 1.916 d.f. 7.764 0.001 0.003 0.001 p-value t test 0.030 0.165 0.186 0.153 p-value z test 0.013 0.040 0.065 0.028 Table 42 School and Teacher Characteristics: Mean Subtest Scores 99 Reading Corner Present Table 43 - School and Teacher Characteristics: Mean Subtest Scores Table 42 School and Teacher Characteristics: Mean Subtest Scores Reading Corner Present DICT ORF RCOMP % LCOMP % % Mean 15.005 0.184 0.462 0.268 No SD 22.264 0.285 0.346 0.272 N 253 253 253 253 Mean 20.091 0.249 0.480 0.335 Yes SD 25.436 0.324 0.366 0.310 N 729 730 730 730 Mean 7.269 0.098 0.329 0.131 99 missing code SD 13.801 0.194 0.357 0.213 N 45 45 45 45 Mean 18.276 0.226 0.469 0.309 Total SD 24.480 0.312 0.362 0.301 N 1027 1028 1028 1028 t-statistic 3.014 3.011 0.715 3.258 d.f. 2.838 0.000 0.001 0.000 p-value t test 0.047 0.102 0.302 0.095 p-value z test 0.001 0.001 0.237 0.001 100 Table 44 - School and Teacher Characteristics: Mean Subtest Scores Table 43 School and Teacher Characteristics: Mean Subtest Scores PTA Functioning DICT ORF RCOMP % LCOMP % % Mean 13.089 0.164 0.402 0.217 No SD 21.754 0.265 0.359 0.274 N 172 173 173 173 Mean 18.539 0.230 0.471 0.317 Yes SD 24.728 0.318 0.364 0.301 N 866 866 866 866 Mean 24.325 0.277 0.531 0.442 99 missing code SD 19.184 0.247 0.320 0.311 N 26 26 26 26 Mean 17.799 0.220 0.461 0.304 Total SD 24.237 0.309 0.363 0.300 N 1064 1065 1065 1065 t-statistic 2.931 2.873 2.289 4.271 d.f. 3.441 0.001 0.001 0.001 p-value t test 0.030 0.107 0.131 0.073 p-value z test 0.002 0.002 0.011 0.000 101 Table 45 - School and Teacher Characteristics: Mean Subtest Scores Table 44 School and Teacher Characteristics: Mean Subtest Scores Meeting with Parents ORF RCOMP % LCOMP % DICT % Mean 12.995 0.158 0.401 0.255 No SD 17.372 0.250 0.347 0.271 N 576 576 576 576 Mean 23.539 0.297 0.530 0.363 Yes SD 28.969 0.353 0.367 0.319 N 518 519 519 519 Mean 17.988 0.224 0.462 0.306 Total SD 24.155 0.311 0.362 0.300 N 1094 1095 1095 1095 t-statistic 7.201 7.417 5.970 6.021 d.f. 2.140 0.000 0.000 0.000 p-value t test 0.009 0.043 0.053 0.052 p-value z test 0.000 0.000 0.000 0.000 102 Table 46 - School and Teacher Characteristics: Mean Subtest Scores Table 45 School and Teacher Characteristics: Mean Subtest Scores How Often do Teachers Meet with Parents ORF RCOMP % LCOMP % DICT % Mean 26.188 0.331 0.568 0.398 About Once per Term SD 29.476 0.357 0.358 0.318 N 464 465 465 465 Mean 0.400 0.000 0.128 0.035 About Twice per Term SD 2.000 0.000 0.199 0.077 N 25 25 25 25 Mean 1.214 0.000 0.200 0.089 About Thrice per Term SD 3.725 0.000 0.260 0.076 N 14 14 14 14 Mean 0.778 0.000 0.333 0.069 About Once per Month SD 2.333 0.000 0.412 0.110 N 9 9 9 9 Mean 23.800 0.300 0.533 0.366 Total SD 29.036 0.354 0.367 0.320 N 512 513 513 513 t-statistic -18.089 -19.959 -10.209 -17.061 d.f. 2.022 0.000 0.002 0.000 p-value t test 0.002 0.016 0.031 0.019 p-value z test 0.000 0.000 0.000 0.000 103 Table 47 - School and Teacher Characteristics: Mean Subtest Scores Table 46 School and Teacher Characteristics: Mean Subtest Scores School has Recommended Reading ORF RCOMP % LCOMP % DICT % Mean 13.899 0.179 0.434 0.254 No SD 18.501 0.277 0.357 0.263 N 402 402 402 402 Mean 20.584 0.254 0.477 0.335 Yes SD 27.102 0.330 0.366 0.317 N 663 664 664 664 Mean 18.061 0.226 0.461 0.304 Total SD 24.424 0.313 0.363 0.300 N 1065 1066 1066 1066 t-statistic 4.776 3.971 1.892 4.492 d.f. 1.956 0.000 0.001 0.000 p-value t test 0.066 0.079 0.155 0.070 p-value z test 0.000 0.000 0.029 0.000 Teacher Owns Curriculum Guide ORF RCOMP % LCOMP % DICT % Mean 18.359 0.227 0.482 0.310 No SD 23.641 0.305 0.357 0.293 N 808 808 808 808 Mean 20.215 0.257 0.456 0.342 Yes SD 26.767 0.341 0.375 0.327 N 239 240 240 240 Mean 18.783 0.234 0.476 0.317 Total SD 24.389 0.314 0.361 0.301 N 1047 1048 1048 1048 t-statistic 0.966 1.202 -0.956 1.370 d.f. 3.676 0.001 0.001 0.001 p-value t test 0.203 0.221 0.257 0.201 p-value z test 0.167 0.115 0.170 0.085 104 Table 48 - School and Teacher Characteristics: Mean Subtest Scores Table 47 School and Teacher Characteristics: Mean Subtest Scores Students Practised Identifying the Sounds in Letters ORF RCOMP % LCOMP % DICT % Mean 21.817 0.310 0.640 0.338 Never SD 24.335 0.375 0.280 0.278 N 20 20 20 20 Mean 22.159 0.279 0.492 0.348 1 or 2 days SD 27.709 0.343 0.372 0.327 N 376 376 376 376 Mean 18.360 0.227 0.438 0.313 3 or 4 days SD 26.163 0.310 0.362 0.299 N 263 264 264 264 Mean 13.981 0.171 0.442 0.264 Daily SD 18.299 0.267 0.354 0.271 N 435 435 435 435 Mean 17.988 0.224 0.462 0.306 Total SD 24.155 0.311 0.362 0.300 N 1094 1095 1095 1095 t-statistic 0.061 -0.365 -2.261 0.159 d.f. 30.163 0.007 0.004 0.004 p-value t test 0.476 0.389 0.133 0.450 p-value z test 0.476 0.358 0.012 0.437 105 Table 49 - School and Teacher Characteristics: Mean Subtest Scores Table 48 School and Teacher Characteristics: Mean Subtest Scores The whole class repeated words or sentences that you said first ORF RCOMP % LCOMP % DICT % Mean 28.621 0.375 0.587 0.435 1 or 2 days SD 29.369 0.362 0.357 0.334 N 277 277 277 277 Mean 15.291 0.173 0.423 0.272 3 or 4 days SD 24.061 0.275 0.359 0.276 N 330 331 331 331 Mean 13.767 0.172 0.417 0.256 Daily SD 18.525 0.272 0.351 0.273 N 487 487 487 487 Mean 17.988 0.224 0.462 0.306 Total SD 24.155 0.311 0.362 0.300 N 1094 1095 1095 1095 t-statistic -6.041 -7.593 -5.625 -6.467 d.f. 4.852 0.001 0.001 0.001 p-value t test 0.002 0.042 0.056 0.049 p-value z test 0.000 0.000 0.000 0.000 106 Table 50 - School and Teacher Characteristics: Mean Subtest Scores Table 49 School and Teacher Characteristics: Mean Subtest Scores Students copied down text from the chalkboard ORF RCOMP % LCOMP % DICT % Mean 2.581 0.019 0.260 0.084 Never SD 5.063 0.059 0.313 0.133 N 43 43 43 43 Mean 11.428 0.139 0.380 0.232 1 or 2 days SD 16.409 0.228 0.353 0.237 N 177 177 177 177 Mean 23.901 0.312 0.531 0.397 3 or 4 days SD 28.009 0.362 0.357 0.334 N 257 258 258 258 Mean 18.730 0.230 0.475 0.306 Daily SD 24.411 0.307 0.362 0.297 N 603 603 603 603 Mean 18.121 0.226 0.464 0.307 Missing SD 24.262 0.312 0.362 0.301 N 1080 1081 1081 1081 Mean 14.616 0.170 0.330 0.264 Total SD 22.481 0.277 0.339 0.265 N 2574 2579 2579 2579 t-statistic 6.080 6.216 2.181 5.462 d.f. 2.095 0.000 0.003 0.001 p-value t test 0.013 0.051 0.137 0.058 p-value z test 0.000 0.000 0.015 0.000 107 Table 51 - School and Teacher Characteristics: Mean Subtest Scores Table 50 School and Teacher Characteristics: Mean Subtest Scores Students retold a story that they had read ORF RCOMP % LCOMP % DICT % Mean 10.314 0.139 0.364 0.200 Never SD 18.490 0.266 0.364 0.257 N 112 112 112 112 Mean 18.166 0.232 0.470 0.322 1 or 2 days SD 23.955 0.321 0.366 0.309 N 507 507 507 507 Mean 23.288 0.283 0.556 0.361 3 or 4 days SD 27.568 0.322 0.339 0.309 N 311 312 312 312 Mean 19.245 0.237 0.357 0.302 Daily SD 21.285 0.299 0.365 0.274 N 93 93 93 93 Mean 18.962 0.238 0.474 0.318 Missing SD 24.633 0.316 0.363 0.304 N 1023 1024 1024 1024 Mean 14.616 0.170 0.330 0.264 Total SD 22.481 0.277 0.339 0.265 N 2574 2579 2579 2579 t-statistic 3.839 3.226 2.783 4.373 d.f. 4.155 0.001 0.001 0.001 p-value t test 0.009 0.096 0.110 0.072 p-value z test 0.000 0.001 0.003 0.000 108 Table 52 - School and Teacher Characteristics: Mean Subtest Scores Table 51 School and Teacher Characteristics: Mean Subtest Scores Students sounded out unfamiliar words ORF RCOMP % LCOMP % DICT % Mean 14.283 0.162 0.391 0.235 Never SD 25.319 0.303 0.358 0.298 N 89 89 89 89 Mean 15.098 0.208 0.457 0.275 1 or 2 days SD 20.939 0.304 0.367 0.275 N 459 460 460 460 Mean 21.923 0.258 0.460 0.339 3 or 4 days SD 28.792 0.333 0.361 0.330 N 312 312 312 312 Mean 20.147 0.236 0.501 0.355 Daily SD 22.320 0.296 0.355 0.297 N 222 222 222 222 Mean 18.035 0.224 0.461 0.306 Total SD 24.257 0.312 0.363 0.301 N 1082 1083 1083 1083 t-statistic 0.286 1.312 1.574 1.187 d.f. 8.075 0.001 0.002 0.001 p-value t test 0.391 0.207 0.180 0.223 p-value z test 0.388 0.095 0.058 0.118 109 Table 53 - School and Teacher Characteristics: Mean Subtest Scores Table 52 School and Teacher Characteristics: Mean Subtest Scores Students learned meanings of new words ORF RCOMP % LCOMP % DICT % Mean 18.198 0.226 0.499 0.281 Never SD 27.040 0.339 0.390 0.313 N 85 85 85 85 Mean 15.915 0.204 0.415 0.287 1 or 2 days SD 22.346 0.317 0.365 0.298 N 396 396 396 396 Mean 19.621 0.240 0.497 0.332 3 or 4 days SD 24.159 0.299 0.350 0.300 N 400 401 401 401 Mean 19.464 0.242 0.481 0.305 Daily SD 26.632 0.314 0.362 0.300 N 199 199 199 199 Mean 18.121 0.226 0.464 0.307 Total SD 24.262 0.312 0.362 0.301 N 1080 1081 1081 1081 t-statistic -0.727 -0.545 -1.826 0.171 d.f. 9.758 0.002 0.002 0.001 p-value t test 0.243 0.341 0.159 0.446 p-value z test 0.234 0.293 0.034 0.432 110 Table 54 - School and Teacher Characteristics: Mean Subtest Scores Table 53 School and Teacher Characteristics: Mean Subtest Scores Students Read Aloud ORF RCOMP % LCOMP % DICT % Mean 1.161 0.013 0.206 0.056 Never SD 3.267 0.050 0.294 0.090 N 31 31 31 31 Mean 15.728 0.192 0.454 0.291 1 or 2 days SD 21.884 0.298 0.356 0.286 N 248 248 248 248 Mean 14.907 0.181 0.427 0.277 3 or 4 days SD 22.836 0.291 0.353 0.292 N 459 460 460 460 Mean 25.000 0.320 0.535 0.376 Daily SD 26.449 0.331 0.367 0.313 N 356 356 356 356 Mean 17.988 0.224 0.462 0.306 Missing SD 24.155 0.311 0.362 0.300 N 1094 1095 1095 1095 Mean 14.616 0.170 0.330 0.264 Total SD 22.481 0.277 0.339 0.265 N 2574 2579 2579 2579 t-statistic 9.657 8.541 4.306 9.624 d.f. 2.256 0.000 0.003 0.001 p-value t test 0.005 0.037 0.073 0.033 p-value z test 0.000 0.000 0.000 0.000 111 Table 55 - School Table and 54 School Teacher and Teacher Characteristics: Mean Characteristics: Mean Subtest ScoresScores Subtest Students assigned reading on their own ORF RCOMP % LCOMP % DICT % Mean 4.102 0.068 0.264 0.108 Never SD 8.430 0.173 0.312 0.130 N 59 59 59 59 Mean 18.049 0.220 0.483 0.303 1 or 2 days SD 24.605 0.314 0.361 0.302 N 329 329 329 329 Mean 18.351 0.230 0.491 0.315 3 or 4 days SD 25.027 0.324 0.353 0.294 N 324 325 325 325 Mean 19.772 0.247 0.449 0.331 Daily SD 24.033 0.306 0.370 0.311 N 382 382 382 382 Mean 17.988 0.224 0.462 0.306 Total SD 24.155 0.311 0.362 0.300 N 1094 1095 1095 1095 t-statistic 7.993 5.368 4.843 8.218 d.f. 3.018 0.001 0.002 0.001 p-value t test 0.002 0.059 0.065 0.039 p-value z test 0.000 0.000 0.000 0.000 112 113 Table 56 - School Table and 55 School Teacher and TeacherCharacteristics: Mean Characteristics: Mean Subtest Subtest Scores Scores Written Evaluations RCOMP LCOMP DICT ORF % % % Mean 17.551 0.206 0.433 0.271 Never SD 25.608 0.318 0.368 0.293 N 164 164 164 164 Mean 12.553 0.140 0.340 0.264 Once per term SD 18.806 0.251 0.325 0.289 N 234 234 234 234 Mean 17.913 0.231 0.475 0.297 Once or twice per term SD 25.138 0.323 0.365 0.310 N 279 279 279 279 Mean 23.368 0.298 0.566 0.369 Weekly or Monthly SD 26.389 0.329 0.353 0.307 N 332 333 333 333 Mean 18.406 0.228 0.467 0.309 Total SD 24.637 0.314 0.362 0.304 N 1009 1010 1010 1010 - t-statistic -2.129 -2.214 -2.598 0.251 d.f. 5.479 0.001 0.001 0.001 p-value t 0.043 0.135 0.117 0.422 test p-value z 0.017 0.013 0.005 0.401 test 113 114 Table 57 - School and Teacher Characteristics: Mean Subtest Scores Table 56 School and Teacher Characteristics: Mean Subtest Scores Oral Evaluations RCOMP LCOMP DICT ORF % % % Mean 17.983 0.280 0.710 0.306 Never SD 20.789 0.321 0.271 0.294 N 20 20 20 20 Mean 16.300 0.193 0.404 0.281 Once per term SD 23.302 0.301 0.357 0.286 N 209 209 209 209 Mean 23.257 0.281 0.494 0.371 Once or twice per term SD 27.836 0.336 0.355 0.318 N 263 263 263 263 Mean 18.334 0.236 0.500 0.312 Weekly or Monthly SD 23.482 0.310 0.363 0.295 N 506 507 507 507 Mean 19.198 0.240 0.483 0.321 Total SD 24.717 0.316 0.361 0.301 N 998 999 999 999 - t-statistic -0.342 -1.162 -4.676 0.375 d.f. 23.109 0.005 0.004 0.004 p-value t 0.368 0.226 0.067 0.386 test p-value z 0.366 0.123 0.000 0.354 test 114 Table 58 - School and Teacher Characteristics: Mean Subtest Scores Table 57 School and Teacher Characteristics: Mean Subtest Scores Review of Portfolios and other projects RCOMP LCOMP DICT ORF % % % Mean 22.025 0.272 0.546 0.352 Never SD 25.526 0.330 0.347 0.307 N 298 298 298 298 Mean 15.430 0.178 0.383 0.259 Once per term SD 24.271 0.291 0.362 0.297 N 226 226 226 226 Mean 15.636 0.213 0.459 0.276 Once or twice per term SD 19.757 0.288 0.366 0.267 N 190 191 191 191 Mean 20.783 0.256 0.486 0.344 Weekly or Monthly SD 27.914 0.337 0.362 0.316 N 243 243 243 243 Mean 18.884 0.234 0.475 0.313 Total SD 24.988 0.317 0.363 0.302 N 957 958 958 958 - t-statistic -3.012 -3.451 -5.202 3.517 d.f. 4.774 0.001 0.001 0.001 p-value t 0.020 0.090 0.060 0.088 test p-value z 0.001 0.000 0.000 0.000 test 115 Table 59 - School and Teacher Characteristics: Mean Subtest Scores Table 58 School and Teacher Characteristics: Mean Subtest Scores Student Reads aloud from chalkboard ORF RCOMP % LCOMP % DICT % Mean 52.067 0.576 0.718 0.632 Never SD 36.042 0.418 0.300 0.317 N 17 17 17 17 Mean 19.211 0.203 0.338 0.307 1 or 2 days SD 29.044 0.310 0.347 0.310 N 74 74 74 74 Mean 22.309 0.277 0.501 0.354 3 or 4 days SD 30.976 0.364 0.371 0.328 N 107 107 107 107 Mean 16.595 0.209 0.464 0.299 Daily SD 21.478 0.291 0.355 0.290 N 820 821 821 821 Mean 17.978 0.222 0.463 0.311 Total SD 23.990 0.307 0.358 0.299 N 1018 1019 1019 1019 t-statistic -3.506 -3.477 -4.559 -3.826 d.f. 83.144 0.011 0.007 0.007 p-value t test 0.000 0.089 0.069 0.081 p-value z test 0.000 0.000 0.000 0.000 116 Table 60 - School and Teacher Characteristics: Mean Subtest Scores Table 59 School and Teacher Characteristics: Mean Subtest Scores Review of homework ORF RCOMP % LCOMP % DICT % Mean 17.291 0.241 0.505 0.290 Never SD 21.149 0.319 0.363 0.284 N 59 59 59 59 Mean 14.957 0.167 0.309 0.240 1 or 2 days SD 25.117 0.306 0.366 0.301 N 90 90 90 90 Mean 25.970 0.345 0.590 0.409 3 or 4 days SD 28.745 0.373 0.357 0.342 N 173 173 173 173 Mean 17.908 0.220 0.467 0.308 Daily SD 23.440 0.296 0.353 0.290 N 700 701 701 701 Mean 18.977 0.238 0.476 0.318 Total SD 24.625 0.316 0.361 0.303 N 1022 1023 1023 1023 t-statistic -0.611 -1.407 -3.217 -1.026 d.f. 14.385 0.003 0.004 0.002 p-value t test 0.275 0.197 0.096 0.246 p-value z test 0.271 0.080 0.001 0.152 117 Table 61 - School and Teacher Characteristics: Mean Subtest Scores Table 60 School and Teacher Characteristics: Mean Subtest Scores Observation and/or competency checklists ORF RCOMP % LCOMP % DICT % Mean 16.237 0.205 0.454 0.276 Never SD 21.958 0.292 0.352 0.289 N 191 191 191 191 Mean 28.312 0.354 0.565 0.404 1 or 2 days SD 31.522 0.376 0.365 0.351 N 222 222 222 222 Mean 12.970 0.163 0.486 0.248 3 or 4 days SD 15.974 0.236 0.351 0.236 N 262 263 263 263 Mean 17.787 0.219 0.415 0.307 Daily SD 26.209 0.320 0.369 0.306 N 268 268 268 268 Mean 18.612 0.232 0.478 0.307 Total SD 25.062 0.316 0.363 0.302 N 943 944 944 944 t-statistic 4.564 4.525 3.125 4.082 d.f. 6.967 0.001 0.001 0.001 p-value t test 0.002 0.069 0.099 0.076 p-value z test 0.000 0.000 0.001 0.000 118