72346 v2 ead ing ra d e R a r ly G uatu E en t Va n s s essm urv ey A e line S ) B a s G RA (VanE Francophone Stream | R e s u lt s R e p o r t ©2012 International Bank for Reconstruction and Development / International Development Association or The World Bank 1818 H Street NW Washington, DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this work is subject to copyright. Because The World Bank encourages dissemination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. For permission to reproduce any part of this work for commercial purposes, please send a request with complete information to the Copyright Clearance Center Inc., 222 Rosewood Drive, Danvers, MA 01923, USA; telephone: 978-750-8400; fax: 978-750-4470; internet: www.copyright.com. All other queries on rights and licenses, including subsidiary rights, should be addressed to the Office of the Publisher, The World Bank, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2422; email: pubrights@worldbank.org Table of Contents ..................................................................................................................................... 1 Table of Contents. .............................................................................................................................................. 7 ABSTRACT. ACKNOWLEDGEMENTS.......................................................................................................................... 8 ...................................................................................................................... 9 SUMMARY AND FINDINGS. Survey results and main findings........................................................................................................ 10 Factors contributing to greater reading fluency and comprehension for Francophone students in Vanuatu................................................................................................. 11 At the student level....................................................................................................................... 11 At the teacher level....................................................................................................................... 12 From assessment to intervention: next steps...................................................................................... 13 Chapter 1- Introduction........................................................................................................................ 16 Structure of the Report...................................................................................................................... 17 Chapter 2: Survey Implementation. ..................................................................................................... 18 .................................................................................................................................. 18 Sample Design. ......................................................................................... 19 Development of the VANEGRA Instrument. Fieldwork and Data Entry................................................................................................................... 19 ................................................................................................................ 19 Reliability of the Instrument. Chapter 3: VANEGRA French Results.................................................................................................. 24 Structure of the Assessment.............................................................................................................. 24 ............................................................................ 26 Administration of the VANEGRA French Instrument. VANEGRA French Results per Sub-test.............................................................................................. 30 Sub-test 1 - Initial Sound Recognition........................................................................................... 30 Sub-test 2 - Grapheme Sound Identification.................................................................................. 31 Sub-test 3 - Familiar Word Reading............................................................................................... 32 Sub-test 4 - Invented Word Reading............................................................................................. 33 1 Sub-test 5a - Oral Passage Reading............................................................................................. 34 Sub-test 5b - Reading Comprehension......................................................................................... 35 Sub-test 6 - Listening Comprehension.......................................................................................... 37 Sub-test 7 - Dictation.................................................................................................................... 38 Differences in Performance by Grade and Gender......................................................................... 43 Summary of Assessment Results....................................................................................................... 45 Chapter 4: Performance in Oral Reading Fluency and Reading Comprehension. ............................ 46 Chapter 5: Analysis of Student Factors Associated with Better Reading Outcomes......................... 49 Chapter 6: Analysis of Teacher Factors Associated with Better Reading Outcomes......................... 55 Effect of Teacher Characteristics on Student Performance............................................................. 56 Effect of Teacher Expectations on Student Performance................................................................ 60 Chapter 7 - Next Steps......................................................................................................................... 66 Bibliographical References...................................................................................................................... 69 ................................................................................................................................ 71 ANNEX 1 / TABLES. List of Tables ................................... 11 Table 1 - Summary of Student-Specific Factors Significant to Literacy Acquisition. Table 2 - Teacher and School-Specifc Characteristics Effecting Literacy Acquisition................................. 12 Table 3 - VANEGRA French sample by region, grade level and gender..................................................... 18 Table 4 - Reliability of the VANEGRA French assessment......................................................................... 20 Table 5 - Reliability of the French EGRA Assessment: Zero Scores Removed........................................... 20 Table 6 - Reliability of the French EGRA Assessment: Grades 2 and 3 Only............................................. 21 Table 7 - Reliability of the French EGRA Assessment: Grades 2 and 3 Only, Zero Scores Removed.............................................................................................................. 21 Table 8 - VANEGRA French Instrument Structure and Early Skills Tested.................................................. 25 Table 9 - Logit Results: Zero-Score Cases vs Some Correct Answers...................................................... 29 ........................................ 31 Table 10 - Sub-test 1 Initial Sound Recognition: Results by Grade and gender. Table 11 - Sub-test 2 Grapheme Identification: Results by Grade and gender.......................................... 32 2 Table 12 - Sub-test 3 Familiar Word Reading: Results by Grade and Gender........................................... 33 Table 13 - Sub-test 4 Invented Word Reading: Results by Grade and Gender.......................................... 34 Table 14 - Sub-test 5a Oral Passage Reading: Results by Grade and Gender.......................................... 35 Table 15 - Sub-test 5b Reading Comprehension: Results by Grade and Gender...................................... 36 Table 16 - Percentage of Correct Answers in Sub-test 5b........................................................................ 37 Table 17 - Sub-test 6 Listening Comprehension: Results by Grade and Gender....................................... 38 Table 18 - Sub-test 7 Dictation: Results by Grade and Gender................................................................ 39 Table 19 - Sub-test 7 Dictation: Number of letters and full words (total and correct) written by grade...................................................................................................................... 40 Table 20 - Sub-test 7 Dictation: Number of letters and full words (total and correct) written by gender.................................................................................................................... 40 Table 21 - Distribution of Students by Fluency......................................................................................... 48 Table 22 - Average fluency, accuracy and reading comprehension levels, ........................................................................................................... 48 by condition of fluency.. .................. 49 Table 23 - Characteristics of students in the sample along several student and family factors. Table 24 - Multiple Regression Results: Part A......................................................................................... 51 Table 25 - OLS Multiple Regression Results: Part B................................................................................. 52 ........................................................................................ 53 Table 26 - Multiple Regression Results: Part C. ......................................................................................... 54 Table 27 - Multiple Regression Results Part D. ........................................................................ 55 Table 28 - Profile of Francophone Teachers in VANEGRA. Table 29 - Teacher and School-Specific Factors Associated with Reading Acquisition.............................. 68 ......................................................... 61 Table 30 - Teachers median expectations about reading outcomes. Table 31 - Regression analyses of average effects of teachers’ expectations on fluency in reading............................................................................................................... 63 Table 32 - Regression analyses of average effects of teachers’ expectations on reading comprehension...................................................................................................... 64 ............................................................... 71 Table 33 - VANEGRA French Reliability Matrix, Grades 2 and 3. .................................................................................... 72 Table 34 - Descriptive Statistics: Grades 2 and 3. .................................................................................. 73 Table 35 - Cronbach’s Alpha, Grade 1 Subsample. 3 .................................................................................. 74 Table 36 - Cronbach’s Alpha, Grade 2 Subsample. .................................................................................. 75 Table 37 - Cronbach’s Alpha, Grade 3 Subsample. ............................ 76 Table 38 - Cronbach’s Alpha, Grade 1 Subsample, Zero Scores Converted to Missing. ............................ 77 Table 39 - Cronbach’s Alpha, Grade 2 Subsample, Zero Scores Converted to Missing. ............................ 78 Table 40 - Cronbach’s Alpha, Grade 3 Subsample, Zero Scores Converted to Missing. Table 41 - ANOVA Results: Differences in Means across Sub-tests.......................................................... 79 ................... 80 Table 42 - Teacher and School Specific Characteristics: Subtest Averages Between Groups. ................... 81 Table 43 - Teacher and School Specific Characteristics: Subtest Averages Between Groups. ................... 82 Table 44 - Teacher and School Specific Characteristics: Subtest Averages Between Groups. ................... 83 Table 45 - Teacher and School Specific Characteristics: Subtest Averages Between Groups. ................... 84 Table 46 - Teacher and School Specific Characteristics: Subtest Averages Between Groups. ................... 85 Table 47 - Teacher and School Specific Characteristics: Subtest Averages Between Groups. ................... 86 Table 48 - Teacher and School Specific Characteristics: Subtest Averages Between Groups. ................... 87 Table 49 - Teacher and School Specific Characteristics: Subtest Averages Between Groups. ................... 88 Table 50 - Teacher and School Specific Characteristics: Subtest Averages Between Groups. ................... 89 Table 51 - Teacher and School Specific Characteristics: Subtest Averages Between Groups. ................... 90 Table 52 - Teacher and School Specific Characteristics: Subtest Averages Between Groups. ................... 91 Table 53 - Teacher and School Specific Characteristics: Subtest Averages Between Groups. ................... 92 Table 54 - Teacher and School Specific Characteristics: Subtest Averages Between Groups. ................... 93 Table 55 - Teacher and School Specific Characteristics: Subtest Averages Between Groups. ................... 94 Table 56 - Teacher and School Specific Characteristics: Subtest Averages Between Groups. ................... 95 Table 57 - Teacher and School Specific Characteristics: Subtest Averages Between Groups. ................... 96 Table 58 - Teacher and School Specific Characteristics: Subtest Averages Between Groups. ................... 97 Table 59 - Teacher and School Specific Characteristics: Subtest Averages Between Groups. ................... 98 Table 60 - Teacher and School Specific Characteristics: Subtest Averages Between Groups. ................... 99 Table 61 - Teacher and School Specific Characteristics: Subtest Averages Between Groups. ................. 100 Table 62 - Teacher and School Specific Characteristics: Subtest Averages Between Groups. 4 ................. 101 Table 63 - Teacher and School Specific Characteristics: Subtest Averages Between Groups. ................. 102 Table 64 - Teacher and School Specific Characteristics: Subtest Averages Between Groups. ................. 103 Table 65 - Teacher and School Specific Characteristics: Subtest Averages Between Groups. List of Figures Figure 1 - Letter Level, Word Level, Oral Reading Fluency and Reading Comprehension Performance..... 22 Figure 2 - Oral Reading Fluency, Reading and Listening Comprehension................................................. 22 Figure 3 - Oral Reading Fluency, Reading Comprehension and Dictation Scores...................................... 23 ............................................................................................. 24 Figure 4 - Stages of Reading Development. Figure 5 - Early Grader Reading Assessment Components...................................................................... 25 Figure 6 - VANEGRA French: Zero-score students as a percentage in the sample as a whole.................. 27 Figure 7 - VANEGRA French: Zero-score students as a percentage in the sample per grade.................... 28 Figure 8 - Distribution of student responses to the writing items (total sample)......................................... 41 Figure 9 - Distribution of student responses to the writing items (per grade)............................................. 41 ........................................... 42 Figure 10 - Distribution of student responses to spelling items (total sample). Figure 11 - Distribution of student responses to the writing items (per grade)........................................... 42 Figure 12 - Differences in performance between boys and girls by grade, measured by the .................................................. 43 number of correct familiar words read per minute (CFWPM). Figure 13 - Differences in performance between boys and girls by grade, measured by the number of correct words read in a connected text per minute (CWCPM)................................................ 44 Figure 14 - Differences in performance between boys and girls in dictation by grade, ............................................................................................................. 44 as a weighted score. Figure 15 - Average Scores in Oral Reading Fluency (sub-test 5a) ........................................................................... 46 and Reading Comprehension (sub-test 5b). Figure 16 - Average Reading Comprehension Levels in Fluent Students (N=110)..................................... 47 5 6 ABSTRACT In August 2010, the Government of Vanuatu carried out early grade reading baseline assessments in English and Francophone schools with financial support from local education partners and technical assistance from the World Bank. The results of the Vanuatu Early Grade Reading Assessment (VANEGRA) are cause for concern. French language findings show that while most students develop some fundamental skills in grade levels 1, 2 and 3, by the end of Grade 3, less than 1 in 4 students are able to develop fluency in reading to understand most of the text they read. The VANEGRA survey also collected data on the attributes of students, teachers and schools. Factors that were shown to be predictors of better reading performance in the early grades include: speaking French at home, owning the school textbook, having literate parents, having books at home, reading at school and at home, attending kindergarten, doing homework, and receiving help from a family member to do homework. Neither teacher experience nor in-service training showed statistical effects on student reading outcomes and two types of certification showed negative and statistically significant effects on student outcomes both in reading fluency and comprehension. VANEGRA also asked about use of seven reading instructional activities. Students who were never asked to learn the meaning of new words or practice grapheme sound correspondences showed negative and statistically significant effects. Conversely, students who were assigned reading daily in their own school time showed positive and statistically significant results. Lastly, VANEGRA asked about teacher expectations for students’ reading performance. Interestingly, the fact that some teachers allowed students to consolidate some reading skills later than the median expectation was associated with better and statistically significant results. Based on the analysis presented, recommendations for improved reading instruction and greater parental involvement are presented at the end of the report. 7 ACKNOWLEDGEMENTS The Vanuatu Early Grade Reading Assessment (VANEGRA) baseline surveys in French and English are the result of the Government of Vanuatu’s commitment to improve reading levels in the country through a mid- term process that incorporates assessment data to improve reading instruction and promote greater parental and community involvement in students learning outcomes. VANEGRA diagnoses comprise the first step in the process by providing Ni-Vanuatu education officials with a system-level diagnosis of how well – and at what pace -- Ni-Vanuatu children in Francophone and Francophone schools develop foundational skills needed to become literate. The reports were prepared by Myrna Machuca-Sierra (Education Specialist) and James A. Stevens (Senior Operations Officer) of the World Bank’s East Asia and the Pacific Education Unit (EASHE). Eleanor Wang (Junior Professional Associate, EASHE) supported data entry activities. Jose Ramon Laguna, Margaret Triyana and Steph de Silva provided support during data cleaning and validation, and the analysis of results. The report benefitted greatly from the comments made by Eduardo Velez-Bustillo (Education Sector Manager, EASHE), Stephen D. Close (Human Development Specialist, EASHE), Warwick Elley (University of Canterbury, New Zealand) Cedric Croft (Consultant, Ministry of Education Vanuatu) and Barbara Thorton (International Development Consultant). In a technical workshop following the presentation of preliminary results, staff from the Ministry of Education’s Curriculum Development Unit and the In-Service Unit provided excellent insights and recommendations on how best to advance a reading improvement agenda that benefits from VanEGRA results. Errors or omissions are the authors’ sole responsibility. The authors would like to express their gratitude to the Ministry of Education of Vanuatu, in particular to the Hon. Min. Charlot Salwai, and Mr. Roy Obed (Acting Director General of Education), for their leadership in this project. The VANEGRA Survey received great support from senior officials including Mr. Charley Robert (Acting PEO Curriculum and Assessment), and Mr. Donald Wotu (Acting Senior Education Advisor, Examinations and Assessments Unit). In particular, the authors would like to acknowledge the work of the VANEGRA Francophone and Francophone core team members, enumerators and supervisors whose hard work and dedication made the French and French VANEGRA surveys possible: Alvin Tari, Ansen Veremaeto, Asanat Tasale, Antoinette Bihu, Bill Bule, Bris Mermer, Buddy Bule, Carmel Melsul, Clemontine Etul, Collin Jacob, Daniel Kohea, Daniel Norlan, Dolores Ngelgen, Dolores Virelala, Don Joseph, Dorneth Kalo, Edmon Hillary, Edward Ben, Enoch Leon, Fred Ottiman, George Josiah, Georgeline John, Gladys Esecher, Gossip Miken, Hapina Kapotua, Harkuk Vocor, Imbert Tevi, Jeffry Ruben, Jenny Sanga, Jerome Ludvaune, Jesica Gambetta, Joseph Buleru, Joshian Molvurai, Kalmaire Morrison, Katchiri Tanga, Kathrine Naliupis, Leah Viro, Lenah Tambe, Lidcha Nanuman, Lucian Bires, Marie Assumpsion, Marie Manu, Marie Tavussi Moli, Marie-Pierre Malere, Mele Socopoe, Michelle Atuary, Particia Mabontare, Patrick Esecher, Paul Michael, Paul Tabi, Paul Thompson, Peter Jacob, Peter Patison, Prescilla Olul, Presley Gaiala, Rachel Henry, Redina Api, Rossie Rihu, Samuel Kaltoutak, Seth Niavie, Silas Boas, Simon Bulekap, Simon Namol, Stangley Lanson, Steven Yawiko, Suthy Lunabek, Tania Melenamou, Thomas Butu, Timothy Lokai, Yamei Johnson. Last but not least, the authors would like to thank the 1,282 Francophone students and the 1,293 Francophone students who enthusiastically participated in the survey. To all, tenkiu tumas. 8 SUMMARY AND FINDINGS This report summarizes the results of the French-language early grade reading assessment conducted in August 2010 by staff from the Vanuatu Ministry of Education. Overall, 1,293 students were assessed from 33 randomly-selected Francophone primary schools in Vanuatu. A separate survey was carried out in English- language schools at the same time. Financial support for the survey was provided by local education partners through the Vanuatu Education Road Map (VERM). Technical assistance and management support was provided by the World Bank. The assessment is part of a global initiative aimed at helping countries measure how well children are learning to read in the early grades of primary education. It aims to help educators develop local knowledge about the specific skills students are struggling with and the factors that appear to contribute to reading development in their schools. Equipped with such evidence, education stakeholders can come together to devise response strategies to improve reading instruction, monitor student’s reading progression, and promote greater parental and community involvement to ensure all children develop the skills needed to become effective readers. The VANEGRA French assessment consisted of seven modules or sub-tests covering basic reading skills such as phonemic awareness, recognition of grapheme sounds, automatic word reading, decoding, oral reading fluency and comprehension – measured in terms of both reading and listening to short narrative passages. A short dictation exercise was included to test early writing skills such as spelling, orientation to text, spacing, capitalization, and punctuation. The VANEGRA French student test was complemented by a student contextual interview which collected information about socioeconomic characteristics, such as availability of reading books at home and literacy prevalence among family members. The survey also included a teacher questionnaire that gathered data on the qualifications of Francophone teachers, their expectations about reading outcomes and the frequency with which they use methods of reading instruction and assessment. Although the VANEGRA French instrument followed a standardized process of adaptation to the local context, results are meant to be used to diagnose gaps in reading instruction and not for cross-country comparison. The survey seeks to provide a baseline standard of reading fluency in French. A version with assessment instructions in Bislama was also produced to accommodate students whose limited proficiency in French could have limited their understanding of the instructions in each sub-test. Grade 1 test results were disappointing due to floor effects and the grade was dropped from a large portion of the analysis. The reliability of the instrument to capture reading abilities in Grades 2 and 3 was estimated using Cronbach’s alpha, a common measure of reliability of survey instruments; at a coefficient of 0.90 . In a scale from 0 to 1, the minimum Cronbach coefficient acceptable in research is 0.7. The analysis of VANEGRA French student data included descriptive statistics (means and standard deviations) to measure average levels in basic reading skills. Also, regression analyses were carried out to estimate average oral reading fluency and associated reading comprehension levels for the sample as a whole. Lastly, statistical analyses were carried out to estimate the relationship between teacher characteristic and student reading outcomes. The most relevant results are presented below, followed by a discussion of policy implications for reading instruction and teacher professional development in the country. 9 Survey results and main findings As students progress across grades, they develop better competency in all basic reading skills; however, these gains do not lead to demonstrated reading fluency (words read correctly per minute) and improved comprehension for most Grade 3 students. As Francophone students progress from Grade 1 to 3, average competency in all sub-tests improves with the largest gains observed at the end of Grade 2. Yet, poor knowledge of the alphabetic principle appears to be one of the main reasons why students struggle to read words both in isolation and in the oral reading passage. Poor word-level reading abilities may be hindering the development of fluency in reading which could explain why at the end of Grade 3, only about 1 in 4 students (24% of Grade 3 students) is able to read at the fluency level needed to understand 60% or more of the text they read. In the Francophone sample, no Grade 1 student achieved the fluency standards. 2% of Grade 2 and 24% of Grade 3 students can be considered fluent readers. Among fluent students (N=110), 8% are in Grade 2 and 92% are in Grade 3, indicating the critical role Grade 3 plays in the development of reading fluency among beginning readers. Low scores are partly due to the number of students for whom the test had to be discontinued because they lacked the minimum knowledge tested. In VANEGRA French, a particular subtest could be discontinued if the student was unable to read the minimum number of letters or words needed. Early-stop cases are allowed in all sub-tests except listening comprehension and dictation. Students assessed as early-stop cases i.e., zero-score serve as a measure of the number of students with the lowest score possible. In VANEGRA French, for the sample as a whole, the proportion of zero-score students is above 30% in sub- tests requiring phonemic awareness (initial sound identification) and word-level reading skills (familiar word reading, invented word reading, and oral passage reading). In sub-test 4 alone, 55% of the students in the sample were unable to successfully decode the first 5 invented words in the exercise, which suggests most students struggle to match letters (or groups of letters) to their sounds to create words. The inability to answer any questions correctly in a subtest was particularly notable in Grade 1, the floor effects being so severe that the grade was dropped from the majority of the analysis. The characteristics of students who were unable to answer any questions correctly are explored in detail. Achieving oral fluency in reading is crucial to improve reading comprehension. As students achieve automaticity on the “mechanics� of reading –i.e., matching letters and graphemes to sounds to make up words and sentences- they develop fluency in reading, allowing them to read longer texts and focus on the meaning of the text. An analysis of oral reading fluency and reading comprehension among Francophone Ni-Vanuatu students showed that students achieve greater levels of reading comprehension when they read at an average rate of at least 45 correct words per minute (CWCPM). At this standard, 7% of the sample, or 107 out of 1,293 Francophone students tested, could be considered fluent in reading. The differences in oral reading fluency and comprehension between fluent and less-than-fluent are striking: while fluent students read almost all the narrative passage (56 out of 58 words) at an average fluency of 68 correct words per minute and comprehend about 64% of the text they read; less-than-fluent students read on average about 14 out of 58 words in the passage at an average fluency of 7 correct words per minute, which allows an average comprehension of only 9% of the text. 10 Finally, VANEGRA French results showed that in many sub-tests, girls performed better than boys and appeared to progress faster in some –though not all- of the skills tested. Girls’ scores were statistically better than boys in 5 out of 7 sub-tests (grapheme recognition, familiar and unfamiliar word reading, oral passage reading, and the weighted dictation score). Non-statistically significant results in the remaining two sub-tests suggest that, on average, boys and girls struggle equally to isolate sounds of letters in the context of words (phonemic awareness skills) and understand stories they hear (listening comprehension). In addition, average differences in performance by gender resulting from the interaction of gender and grade showed that after controlling for the grade effect, gender differences are statistically significant starting in Grade 2 but the difference widens at the end of Grade 3. These differences suggest girls move into word- level fluency and reading fluency faster than boys, which may contribute to a larger number of girls achieving fluency - 9% or 67 of the 645 girls in the sample versus 5% or 38 of the 646 boys in the sample. Factors contributing to greater reading fluency and comprehension for Francophone students in Vanuatu At the student level VANEGRA included a series of questions about student characteristics and behaviors which could be associated with reading abilities. For example, as one may expect, better performance on most VANEGRA French sub-tests was associated with students who reported having literate parents and having books at home. These associations, however, vary by grade and gender. Following is a summary of characteristics and behaviors that were found to have the most impact on student reading outcomes:   Table 1 Student Table Factors  1 Student Associated  Factors to Better  Associated to Reading Better Reading Outcomes  Outcomes   Factor  Impact  Estimated Difference  On average, students who reported having books at home scored almost 6 words per  Books  minute more in the reading fluency measure, about 6% better in reading  available at  Positive  comprehension and about 9% better in listening comprehension than those who  home  reported not owning books at home.  Familial  Mostly  Various effects and interactions, often depending on student gender.   Literacy  Positive  Absent more  On average, students who reported being absent more than one week during the  than one  Negative  school year scored 5 words per minute less in the reading fluency measure, 4% and 7%  week  less in the reading and listening comprehension measures, respectively.  Attending a  On average, students who reported having attended kindergarten scored 5% better in  Positive  Kindegarten  listening comprehension and almost of a quarter of a percentage score in dictation.    The  study  also  examined  in�depth  the  characteristics  of  students  who  were  unable  to  answer  any of the questions on a subtest correctly. The same factors were significant as those impacting  on  literacy  acquisition  generally,  however  owning  the  text  book  had  a  positive  relationship 11 as  students owning the text were more likely to be able to answer some questions.  The study also examined in-depth the characteristics of students who were unable to answer any of the questions on a subtest correctly. The same factors were significant as those impacting on literacy acquisition generally, however owning the text book had a positive relationship as students owning the text were more likely to be able to answer some questions. At the teacher level VANEGRA also explored the association between teacher characteristics and student performance using data collected through the teacher questionnaire on experience, certification, methods of instruction and assessment, and learning expectations. Interestingly, teacher experience had a small but positively significant effect on reading but teacher certification (Certificate of Primary Education) had a negative and statistically significant effect on reading fluency and comprehension. The effect of having the list of recommended reading texts is positive and statistically significant. The following teacher characteristics and behaviors were associated with better student reading outcomes: Table 2 Teacher and School Characteristics Associated to Better Reading Outcomes Table 2 Teacher and School Characteristics Associated to Better Reading Outcomes Factor Impact Estimated Difference Average gain of 5 CWPM in reading fluency, about 6% better Students copied down text reading comprehension scores but 7% worse scores in listening Positive from the chalkboard comprehension. Also, on average, students showed 2% improvement in dictation scores Average decline of almost 11 CWPM in reading fluency, 16% Students retold a story that Negative poorer reading comprehension and 4% poorer listening they had read comprehension scores, as well as 9% decline in dictation scores Average decline of almost 12 CWPM in reading fluency, 11% Students sounded out Negative poorer reading and listening comprehension scores, and a unfamiliar words decline of about 6% in dictation scores Average gain of 25 CWPM in reading fluency, 6% better reading Mixed but Students Read Aloud comprehension and 7% better listening comprehension scores, mostly positive yet a 2% decline in dictation scores Average decline of almost 10 CWPM in reading fluency, 6% Students assigned reading on Mixed better reading comprehension and almost 7% better listening their own comprehension scores, but a 2% decline in dictation scores Average gain of 8 CWPM in reading fluency, almost 7% better reading comprehension and almost 4% better listening Library Present Positive comprehension scores, as well as 7% improvement in dictation scores Average gain of 20 CWPM in reading fluency, about 15% better Supervision in Library Positive reading comprehension and 11% better listening comprehension scores, as well as 14% improvement in dictation scores Average gain of almost 5 CWPM in reading fluency, 9% better reading comprehension and almost 11% better listening Reading Corner Present Positive comprehension scores, as well as 2% improvement in dictation scores Average decline of almost 2 CWPM in reading fluency, almost 3% PTA Functioning Mixed decrease in reading comprehension but 7% better listening comprehension scores 12 14 Teacher Table 2 Table and School 2 Teacher Characteristics and School Associated Characteristics to Better Associated Reading to Better Outcomes Reading Outcomes (contd) Factor Impact Estimated Difference Average Average gain gain ofof 5 CWPM almost inareading half CWPM fluency, in readingabout 6% better fluency, almost Students copied Meeting down text with Parents Mixed reading comprehension 1% decrease scores but 7% worse in reading comprehension but 3%scores better listening in listening Positive from the chalkboard Also, on average, comprehension. comprehension students showed 2% scores Average gain ofimprovement almost 10 CWPM in dictation in readingscores fluency, 7% better School has Recommended Positive reading comprehension scores and almost 5% improvement in Reading Texts Average decline of almost 11 CWPM in reading fluency, 16% Students retold a story that dictation scores Negative poorer reading comprehension and 4% poorer listening they had read Mixed comprehension Average decline scores, as well of almost as 9% in 2 CWPM decline in fluency reading dictation butscores 3% (Certificate in better reading comprehension and almost 10% better listening Certification Primary of almost Average declinescores. comprehension 12 CWPM An average ofin8%reading decline fluency, 11% in dictation Students sounded out Education vs Negative poorer reading and listening comprehension scores, and a scores unfamiliar words None) decline of about 6% in dictation scores Average decline of about 5 CWPM in reading fluency, about 3% Inservice Attendance Negative poorer Average reading gain comprehension of 25 CWPM in reading and 5% poorer fluency, listening 6% better reading Mixed but Students Read Aloud and 7%as comprehension scores, as 8% decline well listening better scores in dictationscores, comprehension mostly positive yet a 2% decline in dictation scores Teacher experience (recent Average gain of 11 CWPM in reading fluency, 13% better reading graduates vs next group by Positive Average decline comprehension andof almost 11% 10 listening better CWPM incomprehension reading fluency, 6% scores, Students assigned reading on experience) Mixed better reading as well ascomprehension 16% improvement and almost 7% better in dictation scoreslistening their own comprehension scores, but a 2% decline in dictation scores Note: CWCPM Note: CWCPM stands “Correct forfor stands Words “Correct Wordsin Connected Text in Connected Per Text Minute� Per Average gain Minute� of 8 CWPM in reading fluency, almost 7% better reading comprehension and almost 4% better listening Library Present Positive comprehension scores, as well as 7% improvement in dictation scores From assessment to intervention: next steps Average gain of 20 CWPM in reading fluency, about 15% better Supervision in Library Positive reading comprehension and 11% better listening comprehension VANEGRA French survey results call for an immediate response scores, to14% as well as improve reading improvement instruction in dictation to ensure scores From assessment Francophone students are to intervention: equipped next Average with the knowledge steps gain of almost 5 CWPM in reading fluency, required to become skilled readers. Specific reading comprehension and almost 11% better listening 9% better Reading Corner recommendations Present include: Positive VANEGRA French survey results call for an immediate comprehension scores, as response to improve well as 2% improvement reading in dictation scores instruction to ensure Francophone students are equipped with the knowledge required to • Improve the focus and structure of reading instruction to promote greater fluency in reading by the become skilled end readers. of Grade Specific 3. Research recommendations has shown that developing include: fluency in reading is crucial to help students 14 become effective readers in the first years of primary education. As students approach reading at • Improve the focus and structure of reading instruction to promote greater fluency in a speed of reading about by the 45-60 end of words Grade per3.minute, Research the reader becomes has shown that better able to focus developing fluencyon the meaning in reading is the of text rather crucial to help on individual thanstudents letterseffective become and words. As shown readers in theby VANEGRA first years French results, of primary students education. readingAs at least 45 correct students approach words per minute reading at a were speedable of to understand about 45-60 about words 83% perof the text minute, reader the read. they becomes However, lessbetter than 1 inable to focus onstudents 10 Francophone the meaning is able to the text ofreach ratherlevel this fluency than on at the of Grade letters individual end and words. 3. Two factors As shown could contribute to by VANEGRA explain French these results. results, On the students one hand, reading poor decoding at least 45 correct words per minute were able to understand about 83% skills suggest instruction falls short of developing a solid foundational knowledge of the alphabetic of the text they read. However, principle. lesspoor On the other, than 1 in 10 French Francophone skills students of students entering theis able to reach Francophone this may stream fluency level contribute the end at delay to the of Grade 3. Two development factors could of pre-reading skills contribute in Grades 1 to explain and these results. 2 as students struggleOnto the one develop hand, poor language decoding and reading skills suggest skills simultaneously. As instruction fallstoshort the MoE sets forth of developing implement a solid the new K-12 curricula, foundational knowledge of the alphabetic principle. On the other, poor it is fundamental that instructional improvements in the early grades take into account the linguistic French skills of students entering the Francophone stream may contribute to delay the development of diversity of the country and provide adequate strategies to prepare students for reading development pre-reading skills in Grades 1 and 2 as students struggle to develop language and in a secondary language (L2). In addition, the new curriculum opens up the opportunity to improve reading skills simultaneously. As the MoE sets forth to implement the new K-12 instruction of letter and word-level reading skills to promote a better sequenced instruction of basic curricula, it is fundamental that instructional improvements in the early grades take into reading account skills. the linguistic diversity of the country and provide adequate strategies to prepare students for reading development in a secondary language (L2). In addition, the new curriculum opens up the opportunity to improve instruction of letter and word- 13 level reading skills to promote a better sequenced instruction of basic reading skills. • Ensure teachers working in the early grades have the knowledge to improve their practice to impact the reading outcomes of their students. In order to improve reading instruction in Vanuatu, teachers will have to improve their knowledge of reading instruction to improve classroom practice. Though most Francophone teachers in VANEGRA use their professional judgment to adjust expectations about reading outcomes, a number of them still consider some of these skills to be unimportant in reading development. Also, while some instructional activities rendered expected outcomes, the fact that other activities typically associated with better reading outcomes showed no statistical significance in Francophone schools suggests the need to review how these activities are carried out in the classroom to better understand the possible factors that are hindering their effectiveness. This is also true of the average effects of teacher methods observed on student outcomes. • Support the reading instruction skills of as many teachers working with beginning readers as possible. Data from the teacher questionnaire showed that only 26% of the teachers in the sample participated in general in-service training courses and only 25% had attended in-service training on reading in the last two years. If only 1 in 4 teachers in the country benefit from learning about specific ways in which they can improve their practice, Ni-Vanuatu teachers will continue to practice their profession in isolation. The role of the newly created In-Service Unit (ISU) at the MoE will be critical to further develop teacher knowledge and practice for reading instruction. As such, it is recommended that VANEGRA findings inform the development of the lesson plans and materials and that ISU staff works in close collaboration with curriculum developers to ensure teachers understand the new curriculum goals and receive support on how to achieve them. • Establish reference reading standards to monitor reading development in the early grades. As the MoE moves on to establish an oral reading fluency standard under VERM, it is important to consider that these indicators should be considered reference standards and not high-stakes benchmarks that would jeopardize additional funding or the promotion of teachers. Since these reference standards are drawn from baseline data, additional measures will be needed in subsequent years to learn about the rate at which Francophone students develop reading abilities. In this sense, reading standards should not be seen as high-stakes but an essential piece to monitor reading progressionin the classroom. In order to set up national reference standards to monitor system-level quality improvements, it would be best to use the percentage of zero-score students in selected sub-tests as a marker and track reductions in the proportion at least biannually. Monitoring achievements over time will eventually provide more information on the rate and the way in which average fluency develops among Francophone students. A modified version of the test could be used to screen students during the school year that may be in need of additional support. • Help teachers translate national reference standards into easy-to-assess, easy-to-monitor reading goals to monitor the reading progression of their students during the school year. In order for teachers and schools to be able to be held accountable for reading outcomes, teachers, school officials and parents need to understand what these standards mean and how each can support reading development in their own school. School development plans should contain reading improvement goals as part of their minimum service standards, as well as a description of activities aimed at encouraging reading. Parents and the community as a whole should be brought into this effort. 14 • Introduce policy actions that increase student exposure to literacy outside the school. VANEGRA French results showed how students who have reading books at home have better reading outcomes and are more likely to become fluent readers. The effect was positive for both boys and girls and for all basic reading skills. Thus, it is advisable that the MoE promotes increased student access to books at home. However, making more books available to students will not per se ensure better reading outcomes. Along with access to more reading materials, Ni-Vanuatu children will need support to develop a reading habit beyond the requirements of the school curriculum. One way of achieving this would be ensure the books being procured by the ongoing Book Flood program are not only grade- appropriate but that they are accessible to students in and outside of the classroom. Since 52% of the teachers reported having access to a school library, an adequate book-borrowing scheme carries the potential to expose students to print on a more regular basis. Another way of increasing exposure to literacy would be to develop community literacy programs where schools become a focal point of literacy in the community. Teachers and community leaders can start up reading clubs and reading competitions to further promote a reading culture among beginning and more experienced readers. • Promote strategies to assure greater parental and community involvement in the reading development process of children. Research shows that the earlier the parental involvement, the more powerful and long-lasting the effects will be both in terms of academic and behavioral outcomes of children. Moreover, research also shows that the most effective form of parental involvement includes those where parents participated in learning activities at home. However, in order for parental involvement to be more effective, parents need not only to be informed about the academic progress of their children but also about ways in which teachers and the school planned to improve outcomes. If parents and schools communicate regularly on the academic progress of children, parents tend to monitor school and classroom activities, and coordinate efforts with teachers such as helping with homework and carrying out extracurricular activities. For this to happen, it is important that parents and schools commit jointly to the reading development process of children. In addition to providing information on the academic progress of children, schools can advise on –and even facilitate- different ways in which parents can promote reading at home. If reading outcomes are to improve in the country, reading development must be seen as a joint enterprise that extends beyond the teacher and the school classroom environment. • Finally, it is clear that more research is needed to better understand the factors that contribute to differences in reading performance between boys and girls. An analysis of the factors that contribute to these differences is beyond the scope of this survey. However, VANEGRA French data showed that boys and girls finish Grade 1 at similar levels of performance in reading, but in Grade 2 girls transition into word-level reading faster than boys and the difference increases at the end of Grade 3. A better understanding of this phenomenon is critical to inform sector policies and increase the success of future reading development programs. 15 1 Chapter 1- Introduction Since 2007, the Vanuatu Ministry of Education (MoE) has administered the Vanuatu Standardized Test of Achievement (VANSTA), a national assessment to monitor literacy and numeracy skills of students in Grades 4 and 6 of primary education. VANSTA’s 2007 and 2009 results revealed that a large share of students is failing to achieve reading comprehension and writing outcomes expected at their grade level (SPBEA, 2009). While VANSTA provides an indication that many students are not reading at levels deemed appropriate for Grades 4 and 6, it does not provide detailed findings about problems in the primary system that lead to poor reading performance in Grades 1 through 3. To complement VANSTA results, the Vanuatu MoE sought to learn if students in Grades 1 to 3 are developing the basic reading skills needed to read fluently and understand what they read. If VANSTA scores are the result of low competence in basic reading skills, it is necessary to identify gaps in instruction where additional instruction and inputs may be needed. In response, the Vanuatu MoE, local education stakeholders including donors,1 and the World Bank joined efforts to conduct a national assessment of basic reading skills in Anglophone and Francophone schools, using adapted versions of the Early Grade Reading Assessment (EGRA) tool appropriate to the Vanuatu context. The assessments are part of a global initiative aimed at helping countries measure how well children are learning to read in the early grades of primary education. From June 2nd to August 26th, 2010 a team of Anglophone and Francophone Ni-Vanuatu reading and language specialists, consultants, and staff from the MoE worked together with the World Bank to develop, trial, and administer the Vanuatu Early Grade Reading Assessment (VANEGRA) Surveys before the end of the 2009-2010 school year. The purpose of the VANEGRA surveys was three-fold: 1. To develop a baseline survey of basic reading skills and temporary reference standards to monitor reading performance in schools and system wide; 2. To build local capacity to replicate early grade reading assessments in the future; and 3. To work with local education stakeholders to interpret VANEGRA findings and analyze their policy and sector investment implications. In particular, the Vanuatu assessments aimed to answer the following questions: • What are the basic reading skills acquired by Anglophone and Francophone Ni-Vanuatu students in Grades 1, 2 and 3? • What are the reading fluency levels at which Ni-Vanuatu students reach high enough levels of comprehension to understand what they read? • What are the factors that influence the acquisition of reading skills among Ni-Vanuatu students? 1 VANEGRA surveys received financial support from pooling partners (AusAID, NZAID, the European Union, UNICEF) in the Vanuatu Education Partners Group. 16 To answer these questions, the latest English and French versions of the EGRA tool were adapted to the Vanuatu context. Because the EGRA tool is an orally-administered test – i.e. carried out as an interview - it is suitable to be administered to young children whose reading and writing skills have not fully developed. Given Vanuatu’s linguistic diversity and the difficulties reading assessment in a secondary language (L2) impose to students with poor language competence, two additional versions of the VANEGRA instruments (Bislama-English and Bislama-French) with instructions in Bislama to accommodate students whose limited proficiency in French or English could have limited their understanding of the instructions for each activity.2 The VANEGRA tools comprised three instruments: (1) a diagnostic instrument assessing basic reading, listening and writing skills among Grade 1, Grade 2 and Grade 3 students; (2) a student contextual interview gathering information on the student’s background, administered to all participating students; and (3) a teacher questionnaire regarding teacher characteristics, expectations and assessment and instruction methods, answered by all Grade 1, Grade 2 and Grade 3 teachers in the sample schools. Each set of instruments was developed in English and French for their administration in Anglophone and Francophone schools, respectively. This report summarizes the main findings from the VANEGRA French survey and provides policy recommendations to inform sector discussions and literacy improvements in Vanuatu. Equipped with information about the specific skills students are struggling with and the factors that appear to contribute to reading development in their country, education stakeholders in Vanuatu can come together to develop response strategies to improve reading instruction and monitor student progression, in order to ensure all children develop the skills needed to become effective readers. Structure of the Report Chapter 1 briefly presents the purpose of the survey and how VANEGRA results are expected to improve learning outcomes in Vanuatu. Chapter 2 summarizes the overall implementation of the survey, in particular, the process followed to develop the VANEGRA French instrument. Chapter 3 presents the main results from each of the sub-tests administered. Chapter 4 presents the analysis of oral fluency and reading comprehension levels and a discussion about the establishment of a reference standard for oral reading fluency in the country. Chapter 5 and Chapter 6 present the results of the analysis of student and teacher factors associated with reading acquisition among Francophone students. Finally, Chapter 7 draws conclusions about the key results to present possible recommendations for improving the quality of reading instruction in Francophone schools. 2 Literacy acquisition in a secondary language (L2) is a complex process even if one does not consider the effect of oral competence in the subject’s primary language (L1). Research on reading and writing suggest literacy acquisition in a sec- ondary language (L2) may produce varying outcomes depending on the nature of literacy in the primary language (L1) and/ or the extent to which it has been mastered (e.g. Alderson, 1984; Carrell, 1991; Carson, 1991). Many students will confront literacy acquisition in L2 with good foundational skills in L1 whereas others will do without sufficient oral competence to support literacy acquisition in L2. Since literacy acquisition in Vanuatu is instructed in L2, VANEGRA results should be inter- preted as a measure of reading outcomes in L2. 17 2 Chapter 2: Survey Implementation3 Chapter 2: Survey Implementation3 In order to build local capacity to replicate early grade assessments in the future, the Ministry of  In order to build Education local capacity   of  Vanuatu to replicate   requested early grade   technical assessments   assistance   from in  the thefuture,   World the Ministry   Bank of Education   during   survey of  Vanuatu requested preparation technical assistance   and  administration. from the    Between World   June   2 Bank nd during  survey   to  August 26   and th preparation   November and   3administration. rd   to  the  17 ,  th 2011, Between  the   World June 2nd   Bank   provided to August   in� 26th and country  support November 3rd to the   to   selected 17th, 2011, the World  staff   Ministry Bank  to   undertake provided   the  in-country sample   design,   develop   the   VANEGRA   instruments   in   English support to selected Ministry staff to undertake the sample design, develop the VANEGRA instruments in  and   French   language,   facilitate   the   training English   of  enumerators and French and  supervisors, language,   facilitate the training   coordinate of enumerators   survey and  logistics   during supervisors,   the  pilot coordinate   and  survey fieldwork,  and logistics during  carry the  out pilot and  test  marking fieldwork, and  and  data carry out entry.    test marking and data entry.    Sample Design Sample Design On June 2nd – 31st, 2011, a series of preparatory meetings took place to discuss the scope and purpose of On  June  2nd  –  31st,  2011,  a  series  of  preparatory  meetings  took  place  to  discuss  the  scope  and  the survey. With advice from the World Bank, the Vanuatu MoE chose a national representative sample with purpose   of  the  survey.   With  advice  from  the  World  Bank,  the  Vanuatu  MoE  chose  a  national  contrast groups representative  accordingsample  with to Grade level   contrast (Grades   groups 1, 2 and 3).   according   toThe   Gradefinal sample   level design  did   (Grades 1,  2not   and incorporate   3).   The  contrast groups by school type and regions, thus survey final sample design did not incorporate contrast groups by school type and regions, thus results can only suggest estimates by Grade  survey   results  can only and gender. 4 The   suggest  estimates target population  by  was Grade as defined and  gender. students enrolled 4   The targetin Grades population 1 to 3 in  was   defined primary  as  schools students implementing   enrolled   in  Grades the official   1  to  3 curriculum.   in  primary Using enrollment   schools data  fromimplementing the Vanuatu   the   official  curriculum. Education Management    Using   enrollment Information System  data   from (EMIS),  the Vanuatu a sample of 33   Education schools was Management selected using  Information a stratified random Systemdesign  (EMIS),   a  with sample   of   33   schools   was   selected   using   a   stratified   random   design proportional allocation based on school type –government or government-assisted-, region, and school size   with   proportional5   allocation   based   onall to ensure   school school   type types   –government and regions would   or  government have a probability �assisted �,  region, of selection 5 equal  and school to  their   size actual   to  ensure distribution   in all the school  The   country. types   and final   regions sample   would consisted   have of 1,293   a  probability students, 646 girls   of and   selection 647 boys   equal   to  1). their  actual  (see Table distribution  in  the  country.  The  final  sample  consisted  of  1,293  students,  646  girls  and  647  boys  (see Table 1).     Table 3 - VANEGRA French sample by region, grade level and gender Table 3 � VANEGRA French sample by region, grade level and gender  1st Grade  2nd Grade  3rd Grade  Province   Male  Female  Total  Male  Female  Total  Male  Female  Total  Total  Malampa  41  39  80  39  41  80  40  40  80  240  Penama  23  26  49  29  21  50  24  26  50  149  Sanma  43  43  86  42  44  86  45  42  87  259  Shefa  44  42  86  45  45  90  45  43  88  264  Tafea  57  59  116  58  57  115  59  57  116  347  Torba  4  7  11  4  7  11  5  7  12  34  Total  212  216  428  217  215  432  218  215  433  1293    refers to the set of VANEGRA documents administered to students and teach-                                                       3 The term “survey instruments� is used 3 ers.The The term“survey term instruments� “assessment� refers tois used the refers to VANEGRA the set of diagnostic VANEGRA instrument documents consisting administered of 8 sub-tests to students or sections. and The term teachers. The term “assessment� refers to the VANEGRA diagnostic instrument consisting of 8 sub-tests “EGRA tool� is used as reference to the latest English, French and Spanish versions which have been adapted in EGRA- or sections. The term “EGRA participating tool� countries isthe to fit used as context. reference to the latest English, French and Spanish versions which have been local adapted 4 in EGRA-participating countries to fit the local context. 4 In EGRA minimum sample sizes, some 400 children are needed for any combination of contrast groups of interest. For In EGRA minimum sample sizes, some 400 children are needed for any combination of contrast groups of example, a sample comparing male-urban would require some 400, as would female-urban, male-rural, and female-rural; interest. For example, a sample comparing male-urban would require some 400, as would female-urban, male-rural, thus, distinguishing by gender and locality would require a sample size of some 1600, whereas a simple baseline per grade and female-rural; thus, distinguishing by gender and locality would require a sample size of some 1600, whereas a would require only 400 per grade. In order to meet VANEGRA logistical costs and timeline, a decision was made to include simple baseline per grade would require only 400 per grade. In order to meet VANEGRA logistical costs and only gender and grade (≥1,200 students) as contrast groups. timeline, a decision was made to include only gender and grade (≥1,200 students) as contrast groups. 5 5 Due to geographical and cost limitations, only the main island in each of the provinces was surveyed. Due to geographical and cost limitations, only the main island in each of the provinces was surveyed. 18 20    Development of the VANEGRA Instrument Due to differences in language, culture and expectations about learning outcomes, the EGRA tool is adapted and piloted to fit the context of each country where applied. From June 20th to 30th, 2011, teams of four Anglophone and four Francophone reading, language and assessment specialists from the MoE developed draft versions of the instruments which were piloted in Port Vila from July 12th – 14th, 2011. Minor changes were incorporated to the pilot instruments to improve readability and clarity of questions in the narrative passages. From August 2nd to 11th, four training workshops - 2 for Anglophone and 2 for Francophone enumerators - were held simultaneously in Port Vila and Luganville. These workshops were attended by temporary and retired teachers as well as zone curriculum advisors from the six provinces in the country. The decision to hold four parallel training sessions was made to reduce transport costs and develop capacity among VITE/CDU and EAU staff to conduct similar trainings in the future. On August 10th, each training session held an enumerator practicum in selected schools in Luganville and Port Vila. After the practicums, each team had feedback meetings where great emphasis was placed on the importance of ensuring that forms were completed fully, clearly, and correctly by every enumerator. Inter- rater reliability – i.e. the ability of enumerators to administer the assessment correctly and consistently - was calculated with results of 83% and above for all sub-tests except Sub-test 2 –Grapheme Recognition – for which a 70% rate of reliability was achieved. Fieldwork and Data Entry Data collection took place between August 11th and 26th, 2011. Data collection was carried out by 32 enumerators, 3 VANEGRA Francophone trainers and 1 fieldwork coordinator in 33 sample schools in the islands of Gaua (Torba Province), Pentecost (Penama), Malekula (Malampa), Éfaté (Shefa), Santo (Sanma), and Tanna (Tafea). Complete survey documents were brought back to Port Vila by the VANEGRA trainers and fieldwork coordinators for marking. Data entry took place between November 3rd and 17th, 2010. There were several instances of ‘out of range responses’ from the teacher and student questionnaires that were coded as missing data in the analysis. For the 105 teachers surveyed in French-speaking schools, item response rates were above 80%. For the students, item response rates were also above 80%. Teacher responses were matched to their students based on school, section and grade where section and grade –e.g. grade 1 section A - indicate the particular student-teacher pair within a school. Reliability of the Instrument Cronbach’s alpha was calculated for the VANEGRA instrument to measure the reliability of the test. Results showed a strong internal consistency with a coefficient of 0.93. As a rule of thumb, an alpha coefficient of 0.80 is considered good and 0.7 is typically the minimum acceptable. The alphas for each sub-test were close to 1, suggesting high reliability across VANEGRA sub-tests (see Annex 1 for the VANEGRA correlation matrix). 6 The authors would like to thank an anonymous referee for pointing this out. 19 Once children learn to apply the foundational reading skills with a certain level of reflex or automaticity, they can move beyond the task of decoding a text (Stage 1) to begin deriving its meaning (Stage 2). As children learn sounds that link to form words, they can begin connecting those sounds to printed words and the idea behind those words. Then they can link words to form sentences, paragraphs, and stories. In other words, children transition from learning to read     Table Table4 4- Reliability  �  Reliabilityof the  of theVANEGRA  VANEGRAFrench  French assessment assessment  Table 4 � Reliability of the VANEGRA French assessment  average  item�test  item�retest   item    average inter Item  Obs  Sign  item�test correlation item�retest     correlation     covariance inter item    alpha  Item  Obs  Sign  correlation  correlation  covariance  alpha  Phonemic Awareness   1293  +  0.70  0.61  0.66  0.93  Phonemic Awareness   1293  +  0.70  0.61  0.66  0.93  Correct Graphemes Per Minute  1293  +  0.89  0.85  0.60  0.91  Correct Graphemes Per Minute  1293  +  0.89  0.85  0.60  0.91  Correct Words Per Minute  1293  +  0.93  0.90  0.59  0.91  Correct Words Per Minute  1293  +  0.93  0.90  0.59  0.91  Correct Non Words Per Minute  1293  +  0.90  0.87  0.60  0.91  Correct Non Words Per Minute  1293  +  0.90  0.87  0.60  0.91  Oral Reading Fluency  1292  +  0.89  0.85  0.60  0.91  Oral Reading Fluency  1292  +  0.89  0.85  0.60  0.91  Reading Comprehension  1293  +  0.83  0.77  0.62  0.92  Reading Comprehension  1293  +  0.83  0.77  0.62  0.92  Listening Comprehension   1293  +  0.59  0.48  0.70  0.94  Listening Comprehension   1293  +  0.59  0.48  0.70  0.94  Writing   1293  +  0.82  0.76  0.63  0.92  Writing   1293  +  0.82  0.76  0.63  0.92  Test scale              0.63  0.93  Test scale              0.63  0.93      due  to However,  due However, thehigh to the   highnumber   number of  zero-scores of  zero�scores   indata, in the it was   the  data, deemed   it possible   was  deemed that the  that statistics   possible   the  in However,   Table 4  could due   to   the be  biased   high upwards  number due   of   zero � to a spurious scores   in   the   data,   it   was   deemed   possible   that   the     due correlation. The statistics were   statistics  with recalculated the 6 statistics in  Table 4  could   be  biased   upwards to  a  spurious   correlation. 6  The were     Table  4  could statistics  inremoved zero-scores   be   biased   upwards   due  to  a  spurious  correlation.   The  statistics  were  6 recalculated  with the and zero are given �scores in Table  removed 5 below.  and  are given in Table 5 below.   recalculated with the zero�scores removed and are given in Table 5 below.   Table 5� Reliability of the French EGRA Assessment: Zero Scores Removed  Table 5- Table 5Reliability of � Reliability the  of French  the  EGRAAssessment:  FrenchEGRA Zero  Assessment: Zero Scores  Scores Removed  Removed   average  item�test  item�test  average Item  Obs  Sign  item�test inter item    alpha  Item  Obs  Sign  correlation item�test     correlation    inter item   alpha  correlation  correlation  covariance   Phonemic Awareness   covariance  Phonemic  Awareness   1296  +  0.71  0.59  0.60  0.91  (totals)   1296  +  0.71  0.59  0.60  0.91  (totals)   Correct Graphemes Per  Correct 1037  +  0.86  0.81  0.54  0.89  Minute    Graphemes Per  1037  +  0.86  0.81  0.54  0.89  Minute  Correct Words Per Minute  826  +  0.91  0.88  0.53  0.89  Correct Words Per Minute  826  +  0.91  0.88  0.53  0.89  Correct Non Words Per  Correct 586  +  0.78  0.82  0.55  0.90  Minute    Non Words Per  586  +  0.78  0.82  0.55  0.90  Minute Oral Reading    Fluency  808  +  0.79  0.83  0.54  0.89  Oral Reading Reading  Fluency     Comprehension 808  +  0.79  0.83  0.54  0.89  Reading   Comprehension   1296  +  0.82  0.75  0.55  0.89  (total)  1296  +  0.82  0.75  0.55  0.89  (total)   Listening Comprehension  Listening  Comprehension  1296  +  0.63  0.49  0.63  0.92  (total)   1296  +  0.63  0.49  0.63  0.92  (total) Writing     1296  +  0.81  0.73  0.52  0.90   scale   Writing Test   1296        +  0.81     0.73     0.52 0.56     0.90     0.91 Test scale              0.56  0.91      20     Table 6  - Table Reliability 6�  of Reliability the  of French  the  EGRA Assessment:  FrenchEGRA Grades Assessment: Grades 2 and  2 and 3 Only  3 Only  Table 6� Reliability of the French EGRA Assessment: Grades 2 and 3 Only  average  average   item� item test  �test   item� item test  �test   Item  Item   Obs  Obs   Sign  Sign   inter  item inter item    alpha  alpha  correlation correlation    correlation correlation    covariance covariance    Phonemic  Phonemic  Awareness Awareness   %    %   866  866   +  +   0.63  0.63   0.52  0.52   0.61 0.61    0.92  0.92   Correct  Correct Graphemes   Graphemes Per   Per   866  866   +  +   0.86  0.86   0.81  0.81   0.54  0.54   0.89  0.89   Minute Minute   Correct  Correct  Words Words  Per   Per Minute   Minute   865  865   +  +   0.92  0.92   0.88  0.88   0.53  0.53   0.89  0.89   Correct  Correct  Non Non  Words   Words Per   Per Minute   Minute   865  865   +  +   0.89  0.89   0.85  0.85   0.53  0.53   0.89  0.89   Oral  Oral  Reading Reading  Fluency   Fluency   865  865   +  +  0.88  0.88   0.83  0.83   0.54  0.54   0.89  0.89   Reading  Reading  Comprehension Comprehension  %   %  866  866   +  +  0.81  0.81   0.75  0.75   0.56  0.56   0.90  0.90   Listening  Listening Comprehension   Comprehension %%    866 866   + +   0.54 0.54   0.41 0.41   0.64  0.64   0.93  0.93   Writing   Writing    866  866   +  +   0.78  0.78   0.70  0.70   0.57  0.57   0.90  0.90   Test Test  scale scale                           0.57 0.57   0.91 0.91     However,  when  the  three  grades  were  treated  as  three  separate  subsamples  in  this  manner,  it  However, when the became  apparent three   that   ingrades were   Grade  1  thetreated as three   very  high separate   ofsubsamples   proportion in   zero�scores this   in   themanner, it became   data  resulted   in  However,  when  the  three  grades  were  treated  as  three  separate  subsamples  in  this  manner,  it  apparent that  in an  unreliable Grade 1 the instrument.   Asvery high  proportion   a  result, the  Grade of zero-scores 1  results   have in the  excluded been data resulted   fromin an  majority   the unreliable  became  apparent  that  in  Grade  1  the  very  high  proportion  of  zero�scores  in  the  data  resulted  in  instrument. As a result, of  the  analysis the   in  this Grade 1   report. results   The have been excluded   characteristics from the   underlying majority   the of   floor the analysis   effects in this  report.   in  Grade 1  are  an  unreliable  instrument.  As  a  result,  the  Grade  1  results  have  been  excluded  from  the  majority  The characteristics underlying the floor effects in Grade 1 are explored in Chapter 3. Tables 6 and 7 give the explored  in  Chapter  3.  Tables  6  and  7  give  the  reliability  of  the  subsample  of  Grades  2  and  3  of  the  analysis reliability   in  this  report. of the subsample   The of Grades 2  and characteristics 3 only; Table 6 with zero  scores   underlying the  floor   effects included and  in   Grade Table   1  are 7 with the  only; Table 6 with zero scores included and Table 7 with the zero scores excluded. In both cases,  explored zero scores   Chapter In   inexcluded. 3.  Tables both   6  and cases,   7  give  the Cronbach’s   reliability alpha is at or  above   subsample of  the0.90. Complete Grades  tables   of reliability 2  and by 3  Cronbach’s  alpha  is  at  or  above  0.90.  Complete  reliability  tables  by  Grade  (including  Grade  1)  only; Table Grade  6 with (including  zero Grade 1) scores  included are given  and Table 7 with the zero scores excluded. In both cases,  in the annex. are given in the annex.  Cronbach’s  alpha  is  at  or  above  0.90.  Complete  reliability  tables  by  Grade  (including  Grade  1)  are given Table in  7 Table -  the  annex. Reliability 7 of   the � Reliability French of the EGRA  French Assessment:  EGRA Grades 22  Assessment: Grades and  and  3 3 Only, Only, Zero  Zero Scores  Scores Removed  Removed   average Table 7� Reliability of the French EGRA Assessment: Grades 2 and 3 Only, Zero Scores    Removed   item�test  item�test  Item  Obs  Sign  inter item  alpha  correlation  correlation  average    covariance item�test  item�test  Item  Obs  Sign  inter item alpha  Phonemic Awareness  (totals)  866  +  0.64    correlation 0.51    correlation 0.58     0.91  covariance  Correct Graphemes Per  Phonemic Awareness  (totals)  825  866 +  0.85  0.64 0.79  0.51 0.51  0.58 0.88  0.91 Minute  Correct  Correct Graphemes  Words  Per     Per Minute 758  0.91  0.88  0.49  0.87  825 +  0.85 0.79 0.51 0.88 Minute  Correct Non Words Per  Correct 572  0.87  0.82  0.52  0.88  Minute  Words Per Minute  758 +  0.91 0.88 0.49 0.87 Correct  Non Words Oral Reading  Per  Fluency     724   +   0.88   0.83   0.50   0.88   572 + 0.87 0.82 0.52 0.88 Minute   Reading Comprehension  Oral  Reading  Fluency  866  724   +  +  0.80  0.88   0.73  0.83   0.52  0.50   0.88 0.88   (total)   Reading Comprehension Listening  Comprehension    866  866   +  +   0.80  0.56   0.73  0.42   0.52  0.60   0.88  0.91   (total)  (total)   Listening Writing     Comprehension  866 866   +  +   0.56  0.77   0.42  0.69   0.60  0.53   0.89 0.91   (total)   Test scale              0.53  0.90  Writing   866  +  0.77  0.69  0.53  0.89    21 23 23       Test scale              0.53  0.90    The  acquisition  of  reading  skills  is  individual  in  the  manner  in  which  they  are  acquired.  Some  The acquisition of reading skills is individual in the manner in which they are acquired. Some skills are attained skills  are  attained  simultaneously  in  some  children,  while  others  proceed  in  a  more  linear  simultaneously in some children, while others proceed in a more linear fashion. However, there are some fashion.  However,  there  are  some  average  progressions  that  can  be  observed  across  the  sample.  average progressions that can be observed across the sample. Figure 1 indicates letter level, word level, Figure  1  indicates  letter  level,  word  level,  reading  fluency  and  comprehension  attainment  by  reading fluency and comprehension attainment by percentile in the VANEGRA French sample. The graph percentile  in  the  VANEGRA  English  sample.  The  graph  indicates  that  the  various  skills  are  indicates that the various skills are acquired together, though in this sample, certain skills such as decoding acquired  together,  though  in  this  sample,  certain  skills  such  as  decoding  unfamiliar  words  are  unfamiliar words are acquired at a later stage compared to familiar words and letter recognition. acquired at a later stage compared to familiar words and letter recognition.  Figure 1 Letter Figure Level,  1 Letter Word  Level, Level,  Word Oral  Level, Reading  Oral  Fluency and  ReadingFluency Reading and Reading Comprehension  Comprehension Performance  Performance    Figure  2  indicates  the  progression  of  oral  reading  fluency  with  reading  and  listening  Figure 2 indicates the progression of oral reading fluency with reading and listening comprehension. In this comprehension.  In  this  sample,  on  average,  listening  comprehension  is  attained  ahead  of  sample, on average, listening comprehension is attained ahead of reading comprehension and oral reading reading comprehension and oral reading fluency.  fluency. Figure 2 Oral Reading Fluency, Reading and Listening Comprehension  Figure 2 Oral Reading Fluency, Reading and Listening Comprehension 24      22 Figure 3 indicates the progression of reading comprehension, oral reading fluency and dictation  skills. It shows that dictation skills are acquired at a very similar rate to reading comprehension  in this sample and shows the relationship between the stabilization of memory through writing    skills. It shows Figure 3 indicates the progression of reading comprehension, oral reading fluency and dictation that dictation skills are acquired at a very similar rate to reading comprehension in this sample and shows Figure 3 indicates the progression of reading comprehension, oral reading fluency and dictation  the relationship between the stabilization of memory through writing and the skills required to comprehend skills. It shows that dictation skills are acquired at a very similar rate to reading comprehension  text that is read. in this sample and shows the relationship between the stabilization of memory through writing  and the skills required to comprehend text that is read.  Figure 3 Oral Reading Fluency, Reading Comprehension and Dictation Scores Figure 3 Oral Reading Fluency, Reading Comprehension and Dictation Scores    In  summary,  the  VANEGRA  English  instrument  has  good  reliability,  even  when  zero�score  cases In   (discussed summary,   in  Chapter the VANEGRA   3)instrument English   are  removed,   butreliability, has good   this  required   removing even when   Grade zero-score   1 (discussed from  the  cases majority  of  the  analysis.  The  results  of  the  survey  indicate  that  the  basic  literacy  skills  are  in Chapter 3) are removed, but this required removing Grade 1 from the majority of the analysis. The results acquired  together  and,  although  they  are  acquired  at  different  rates  and  in  different  of the survey indicate that the basic literacy skills are acquired together and, although they are acquired at progressions amongst students, some average relationships can be observed across the sample.  different rates and in different progressions amongst students, some average relationships can be observed across the sample. 25    23 3 Chapter 3: VANEGRA French Results Chapter 3: VANEGRA French Results Structureof Structure ofthe theAssessment Assessment As As   has has   been been   confirmed confirmed   by  scholars working  to by scholars working   to  understand understand   reading reading acquisition   acquisition in multiple   in  multiple (Jimenez   languages, languages,  (Jimenez  and  O’Shanahan  Juan,  2008;  Linan�Thompson  and  Vaughn,  2007;  Abadzi,  and O’Shanahan Juan, 2008; Linan-Thompson and Vaughn, 2007; Abadzi, 2006; Sprenger-Charolles, 2004; 2006;  Sprenger�Charolles,  2004;  Chiappe  et  al.,  2002),  being  able  to  read  well  requires  a  grasp  of  Chiappe et al., 2002), being able to read well requires a grasp of five basic skills in almost any alphabetic five  basic  skills  in  almost  any  alphabetic  language  in  which  print  can  be  decoded  into  sounds  language in which print can be decoded into sounds (National Reading Panel, 2000): (National Reading Panel, 2000):    • phonemic phonemic awareness–focusing   awareness –focusing on,  on,   manipulating, manipulating,   breaking breaking   apart, apart,   and  putting together  together  and putting sounds sounds orally;     orally; phonics–linking    • phonics –linking written written  letters letters to to  their their  sounds sounds and and  forming forming  spelling spelling  patterns;  patterns;  • fluency fluency –achieving  speed,accuracy, –achieving speed,  accuracy,  and and  expression expression  in reading;  in reading;  • vocabulary vocabulary –knowing  words –knowing words  (both (both  oral oral and  and  written) written) and   and meaning; their their meaning; and  and   • comprehension –understanding the concepts read or heard.  comprehension–understanding the concepts read or heard.   Though Though not   notall children   all develop   children their   reading   develop abilities their  reading in the same   abilities way   in  the or pace,   same   way the literature or  pace,   theshows that     literature shows all  that readers  all readers progress  progress through  through a series  a series of phases of phases or  stages  or stagessimultaneously- –sometimes  –sometimes simultaneously in their reading �  in their reading development process development (RTI, 2010).  process (RTI, 2010).     Figure 4  - Figure Stages of  Reading Stages 4 �  Development Reading Development   Source: RTI, 2010.          Source: RTI, 2010.    Once children learn to apply the foundational reading skills with a certain level of reflex or automaticity, they Once  children  learn  to  apply  the  foundational  reading  skills  with  a  certain  level  of  reflex  or  can move beyond the task of decoding a text (Stage 1) to begin deriving its meaning (Stage 2). As children automaticity,  they  can  move  beyond  the  task  of  decoding  a  text  (Stage  1)  to  begin  deriving  its  learn sounds that link to form words, they can begin connecting those sounds to printed words and the idea meaning  (Stage  2).  As  children  learn  sounds  that  link  to  form  words,  they  can  begin  connecting  behind those words. Then they can link words to form sentences, paragraphs, and stories. In other words, those  sounds  to  printed  words  and  the  idea  behind  those  words.  Then  they  can  link  words  to  form  sentences,  paragraphs,  and  stories.    In  other  words,  children  transition  from  learning  to  24 read  to  reading  to  learn  (Stage  3  and  beyond).  Comprehension  is  the  ultimate  prize—the  children transition from learning to read to reading to learn (Stage 3 and beyond). Comprehension is the difference  between “reading it� and “getting it� (RTI), 2010).  The structure of the EGRA tool in  difference ultimate different between    international prize—the  “reading difference  it� and between   applications  “getting “reading   used  it� it� and this  (RTI),    2010). “getting knowledge    as The it� (RTI),  a   structure 2010). The reference    point  of the EGRA structure  tool of the   during  in   EGRA   the different tool   international in different adaptation    of international applications applications the tool to local    contexts used used   this   this knowledge  (Figure  2). knowledge   as   a  reference   point   during as a reference point during the adaptation of   the   adaptation the   tool to local of the  tool to contexts  local 2). (Figure contexts (Figure 2).    Figure 5 � Early Grader Reading Assessment Components  Figure Figure 5 5- Early Grade  � Early GraderReading Assessment Components  Reading Assessment   Components   Source: RTI, 2010.      Source: Source: RTI, RTI, 2010.  2010.  In  particular,  the  VANEGRA  French  assessment  focused  on  measuring  these  skills  plus  basic  In  particular, In particular, the VANEGRA    the   VANEGRA French assessment focused on measuring these skills skills listening plus  basic and listening   and writing French   skills   through   assessment seven  modules  focused   or   sub �tests:  (1)   identification on  measuring these plus   of   initial basic    writing listening skills   andthrough   writingseven modules   skills  throughor sub-tests:    seven (1) identification   modules of initial sounds in words; (2) grapheme sound sounds  in   words;   (2)  grapheme   sound knowledge; or  sub�    (3)   familiar tests: word    (1)   identification reading;   of  initial   (4)  invented   word   knowledge; sounds   in (3)   words;familiar word reading; (4) invented word reading; (5) oral reading fluency with comprehension; reading;   (5)   oral   (2)   grapheme reading   fluency  sound  with   knowledge;   (3)  familiar   comprehension;   word  reading;   (6)  listening   (4)  invented   comprehension;   word   (7)   and   reading; (6) listening   (5)   oral   reading comprehension;   fluency (7)   with and dictation   comprehension; (see Annex 2 for a   (6) copy dictation  (see  Annex  2  for  a  copy  of  the  VANEGRA  French  instrument).  Table  3  below  shows   listening   of the VANEGRAcomprehension;   (7) French instrument).   and   dictation Table 3 below how these   (see   shows  skills Annex  relate   2  to how for   a  copy these  each  skills of the  of   the to relate VANEGRA VANEGRA each of the  French   French VANEGRA   instrument).  components, French   Table components,  measures   3  below  and  measures  shows indicators.    and how  these skills relate to each of the VANEGRA  French components, measures and indicators.  indicators.   Table Table8 8- VANEGRAFrench VANEGRA  �   French Instrument Structure Instrument Structure and  and Early  Early Skills  Skills Tested  Tested   Table 8 � VANEGRA French Instrument Structure and Early Skills Tested  Skill  demonstrated  by  students’  ability   Measure and Indicator  Sub�test  Early reading skill  to:    demonstrated  by  students’  ability   Measure and Indicator  Skill Sub�test  Early reading skill  to:  Phoneme  segmentation  as  the  Segment words into 2 to 5 phonemes  1. Identification of  Phonemic  Identify  words Segment words  withinto  2  to 5 phonemes different     Phoneme   beginning number of    sounds segmentation  correctly    identified as  the    1.  sounds    of   Identification initial Phonemic awareness    number   of   sounds   correctly   identified    Identify or ending   words   with  different  beginning   phoneme initial sounds   awareness  or ending phoneme  Provides  the  sound  of  upper�  and  Grapheme  sound  fluency  in  terms  of  2. Grapheme  Phonics  Provides lowercase   the   sound  of   graphemes     upper�  and distributed   in    Grapheme   sound  fluency correct  grapheme   sounds   in     identified terms  of   2.  Grapheme knowledge      Phonics  lowercase random order   graphemes      distributed  in  correct   grapheme per minute  (CGPM)    sounds  identified  knowledge   random order   per  minute Familiar   word (CGPM)     fluency   in  terms  of  3. Familiar word  Read  simple  and  common  one�  and  Word reading  correct   word Familiar familiar   fluency   words   in    read terms of       per 3.  Familiar reading    word  two� syllable Read and   common  one�  and  simple  words Word reading  correct minute familiar    words  read  per    (CFWPM) reading  two�syllable words  Make  grapheme�phoneme  minute   word   fluency  in  terms  of   (CFWPM) Invented 4. Invented word  Alphabetic  Make   correspondences grapheme   (GPCs) �phoneme   through   the correct    invented    Invented word  fluency   words   in     read terms     per of   4.  Invented reading    word  principle    Alphabetic correspondences reading  of  simple   (GPCs)   through   invented     –i.e., the   correct minute    (CUWPM) invented  words  read  per  reading  principle  reading invented   � of   simple words   invented  to test  decoding –i.e.,    skills     minute (CUWPM)  invented� words to test decoding skills  Oral  reading  fluency  in  terms  of  5. Oral reading fluency  Oral reading  Read  a  text  with  little  effort  and  at  a  Oral   reading correct   words    fluency read  per   in   terms   minute    inof  a    5.   Oral  reading with comprehension   fluency   Oral  reading fluency     Read   a  text sufficient   with  rate     little  effort  and  at  a  correct   words narrative  passage   read     per  minute (CWCPM)     in  a  with comprehension  fluency  sufficient rate  narrative passage  (CWCPM)  25 27  Skill  demonstrated  by  students’  ability   Measure and Indicator  Sub�test  Early reading skill  to:  Respond correctly to different types of  Response  to  questions  after  reading  a  Reading  questions,  including  literal  and  story  as  a  percentage  of  correct  comprehension  inferential  questions  about  the  text  answers   they have read  Respond  correctly  to  different  types  of  Response  to  questions  after  hearing  a  6.  Listening  Listening  questions  including  literal  and  story  as  a  percentage  of  correct  comprehension  comprehension  inferential  questions  about  the  text  the  answers  enumerator reads to them  Write,  spell,  and  use  grammar  properly  through  a  dictation  exercise,  Alphabetic  Write,  spell,  and  use  grammar  properly  determined  by  the  percentage  of  7.  Dictation   principle  through a dictation exercise  overall  early  writing  skills  (spelling  and  basic  conventions),  weighted  score.  Note: Adapted by the authors, based on RTI, 2009 and Linan�Thompson, 2010.        Administration of the VANEGRA French Instrument Administration of the VANEGRA French Instrument The VANEGRA French assessment was administered via face-to-face interviews between an enumerator and The  VANEGRA  French  assessment  was  administered  via  face�to�face  interviews  between  an  a student.7 Each interview lasted 20 to 25 minutes from the onset of the test to completion of the student enumerator and a student. 7  Each interview lasted 20 to 25 minutes from  the onset of the test to  background questionnaire. In five of the seven sub-tests in the VANEGRA instrument, students had 60 seconds completion  of  the  student  background  questionnaire.    In  five  of  the  seven  sub�tests  in  the  to complete the sub-test in order to assess automaticity in a given skill. To be successful readers, basic VANEGRA  instrument,  students  had  60  seconds  to  complete  the  sub�test  in  order  to  assess  reading competencies automaticity   in  a  givenhave to be     skill. automatic. Fluency  measures To  be  successful readers,  assess not only whether basic  reading or not a competencies child  knows   have to  be  something, but whether they have internalized the knowledge and automatic.    Fluency  measures  assess  not  only  whether  or  not  a  child  knows  something,  but  can process the information automatically (Linan-Thompson, whether  they  have 2007). Time-limitation   internalized allows proper   the  knowledge comparison   and   can  process of fluency across slow     the  information readers automaticallyand fast  (Linan�who readers Thompson, may register   2007). Timescores the  same �limitation   allows at different   proper periods   comparison of time (RTI, 2009).   of  fluency  across  slow  readers  and  fast  readers  who  may  register  the  same  scores  at  different  periods  of  time  (RTI,  In VANEGRA, student scores in time-limited sub-tests were calculated as the number of correct items – 2009).   i.e. letter names, letter sounds, or words - read per minute. If a student completes all of the words before   the time expires, student In VANEGRA, the time scores  in timeis of completion recorded �limited  suband the were �tests number of items  correctly  calculated as the numberread per  ofminute  correct is  estimated items – i.e. on that time  letter  names, period. 8 Selected  letter  sounds, sub-tests  or wordsapplied an “early-stop  � read  per  minute. rule�   to If discontinue the administration a student completes  all of  the of a  words   before sub-test   the  time if students were unable  the   expires, to  correctly time  of  completion respond to  any is  recorded of the items   and in the first of  of   number the   items ten lines  correctly  read (Sub-tests,   per or 2,3,4,5a), minute if their  responses is  estimated for   onfirst the   that   time five items   period. 8     Selected were incorrect   sub�1). (Sub-test tests 9 applied In  this   an  situation, “early the �stop  rule� enumerator to  discontinue was asked to mark the box  that   the administration read “Check   a  sub of  box this �test if the   if  students exercise   were  unable was discontinued because   to  correctly   respond the child has no correct answers in the first line� and to proceed to the next sub-test in the test (RTI, 2010).10     to   any   of   the   items   in   the   first   of   ten   lines   (Sub � tests,   2,3,4,5a),   or  if   their responses  for  the  first  five  items  were  incorrect  (Sub�test  1).  9  In  this  situation,  the  enumerator  was  asked  to  mark  the  box  that  read  “Check  this  box  if  the  exercise  was  discontinued  because  7 The term enumerator is used in reference to purposely trained interviewers administer early grade reading assessments. 8 Correct Items Per Minute = (Total items read – Total items incorrect) / [(60 – Time remaining on stopwatch) / 60] (RTI,                                                        2009) 7 The term enumerator is used in reference to purposely trained interviewers administer early grade reading 9 The rule was established to avoid frustrating students who did not have the skill or did not understand the task of the assessments. Correct(RTI, sub-test 8 Items2010). Per Minute = (Total items read – Total items incorrect) / [(60 – Time remaining on stopwatch) / 60] (RTI, 10 2009)   See Annex 2 for a copy of the VANEGRA French instrument. 9 The rule was established to avoid frustrating students who did not have the skill or did not understand the task of 26 the sub-test (RTI, 2010). 28  the  child  has  no  correct  answers  in  the  first  line�  and  to  proceed  to  the  next  sub�test  in  the  test  (RTI, 2010).10       The  justification  to  discontinue  a  sub�test  has  two  reasons.  First,  the  early  stop  rule  helps  avoid  frustration among students whose dispirit may affect their performance in subsequent sub�tests.  Second,  the  early  stop  rule  is  also  an  approximation  of  zero�scores  throughout  the  test:  as  in  The justification to discontinue a sub-test has two reasons. First, the early stop rule helps avoid frustration most   psychometric  tests,  it  is  assumed  that  students  who  fail  the  first  initial  items  will  fail  the  among remainder students   of  the whose dispirit may   test,  especially affect   if  test their items performance in subsequent   become  progressively sub-tests.   more Second,   difficult,   as  inthe early   the stop   case   rule is also an approximation of zero-scores throughout the of VANEGRA.  Sub�tests 6 and 7 did not apply the “early stop rule� so results in these sub�tests  test: as in most psychometric tests, it is assumed that students relate   to  the  totalwho  fail the first sample   of initial studentsitems will  fail   and the remainder report   percentage of  theof  test, correct especially   answers if test   over items   thebecome   total  number  of items progressively more  the sub  indifficult, as�test.    case of VANEGRA. Sub-tests 6 and 7 did not apply the “early stop rule� in the   so results in these sub-tests relate to the total sample of students and report percentage of correct answers   over the total number of items in the sub-test. For  each  sub�test  in  VANEGRA,  results  are  presented  first  in  terms  of  the  percentage  of  zero� For each score   students,sub-test   asin VANEGRA,   well results are   as  the  average presented   score   tested.first in terms   The   reason of  the forpercentage   this  is  that of zero-score   the   percentage students,   of  as well as the average score tested. The reason for zero�score students represents students who showed no evidence of the skill tested whereas the this is that the percentage of zero-score students average represents   score   represents students   the  average who showed   rate  of no evidence   acquisition of the skill tested   of  a   given  skill. whereas    In  VANEGRA the average score represents   French,   the across average   sub rate�tests   and  grades, of acquisition of a  the   largest given skill.   proportions In VANEGRA   French, of  early �stop  cases across   appeared sub-tests and grades,   in  sub the�test   4  largest (invented proportions   word   reading), of early-stop cases  sub �test  5a  in appeared (oral   passage sub-test   reading), 4 (invented word   and   sub�test reading),   3  (familiar sub-test 5a (oral  passageword  reading).     The   sub � test   that   showed   the   lowest   percentage reading), and sub-test 3 (familiar word reading). The sub-test that showed the lowest percentage of early-   of   early � stop   cases   was   sub � test   1  stop cases  was (grapheme sound   knowledge) sub-test 1 (grapheme   where sound  the   test  hadwhere knowledge)   to  be   discontinued the test had to be   discontinued only  for  20% only   of   the for 20%   students  in the sample. of the students in the sample.      Figure 6  – Figure 6–VANEGRA  VANEGRA  French: Zero-score French: students Zero�score students  asas a percentage  a percentage in sample  in the the sample as a whole  as a whole      As expected, the assessment had to be discontinued for a larger percentage of Grade 1 students (see Figure                                                        104 below). However, students make great progress in reading development in Grades 2 and 3 as the number See Annex 2 for a copy of the VANEGRA French instrument. of zero-score students decreases notably. In these grades, less than 20% of the students in Grade 3 were unable to decode a single word, and less than one in ten was unable to read the first 9 words in the29   oral   reading passage or read the first five familiar words in the activity. 27   As  expected,  the  assessment  had  to  be  discontinued  for  a  larger  percentage  of  Grade  1  students  (see  Figure  4  below).    However,  students  make  great  progress  in  reading  development  in  Grades  2  and  3  as  the  number  of  zero�score  students  decreases  notably.  In  these  grades,  less  than  20%  of  the  students  in  Grade  3  were  unable  to  decode  a  single  word,  and  less  than  one  in  ten was unable to read the first 9 words in the oral reading passage or read the first five familiar  words in the activity.   Figure 7– VANEGRA Figure French:  7– VANEGRA Zero-score  French: students  Zero�score  as a percentage  studentsas percentage  inin the  the sample  sample per  per grade  grade      The  characteristics  of  students  who  were  unable  to  answer  any  questions  on  a  subtest  correctly  The characteristics of students who were unable to answer any questions on a subtest correctly were were  explored  in  a  series  of  logit  models.  The  logit  model  is  similar  to  an  Ordinary  Least  Squares in explored a series model, of   but logit models.   instead The logit model   of  a  continuous is similar   dependent to an Ordinary   variable, Least   it  explores Squares   the model,   relationship but   of   a  instead of a continuous dependent variable, it explores the relationship of a binary dependent binary  dependent  variable.  In  our  case,  the  binary  variable  was  constructed  as  students  able  to variable. In   our case, answer the binary   some variable   questions was constructed   correctly   on  the as students relevant able to   subtest answer   and   those some questions   who   were  ablecorrectly on the     to  answer none.   subtest and those who were able to answer none. relevant Table Table 9  gives the  results 9  gives the  results   of  of these these logit   logiton models   models   on each of the each  of  the   subtests. As  expected, subtests. grade   As  expected,   grade progression has a   progression substantial   has  a  substantial relationship   relationship with the ability   with to answer some   the  ability correct   tostudents answers:   answerin   some   correct Grades 2 and  3answers: are more   students likely   in to fall   Grades into   2  and the group   3  are who more are able   likely to answer   fall  into   tosome   the  group questions   who correctly.   are  able Gender was  to   answer not   some  a significant questions predictor in  any correctly. except     Gender subtest, the  identification inwas not  a  significant   predictor of phonemes,   in  any dictation   subtest, where   except girls were more  likely in  the to   identification answer   of  phonemes, some questions   dictation correctly.   where  girls When interacted were  more with  Grade in the  likely   tocomprehension listening questions   answer  some test. In this   correctly. case,   When girls in   interacted Grade   with 2 were more   Grade likely   in  the to answer   listening some   comprehension questions than boys.   test.  In  this  case,  girls  in  Grade 2 were more likely to answer some questions than boys.   The languages spoken at home did not have a significant relationship with the binary dependent variable and neither did doing homework except on the dictation subtest. However, reading at home had a significant, positive association with the ability to answer some questions correctly on the listening comprehension subtest and attending a kindergarten had a significant, positive association with being able to read some familiar words correctly and answer some dictation elements correctly. Reading materials at home had a 30  positive relationship with some subtests including unfamiliar word-reading skills and listening comprehension.   Overall, students in the early-stop or zero-score category were less likely to have reading materials or literate family members at home and less likely to use them. They were less likely to be girls and more likely to be students in younger grades. 28 Table 9 – Logit Results: Zero Score Cases v Some Correct Answers Table 9� Logit Results: Zero�Score Cases vs Some Correct Answers  Sub�test  Sub�test     Sub�test 1  Sub�test 2  Sub�test 3  Sub�test 4  Sub�test 6  Sub�test 7  5a  5b  Initial  Binary Variables:  CSPM  CFWPM  CIWPM  CWCPM  RCOMP  LIST COMP  DICT  Sounds  �0.2151  0.3846  0.2943  0.3229  0.1253  �0.0487  �0.3377  0.4110  Gender  0.3170  0.0560  0.2710  0.5320  0.6090  0.9220  0.1410  0.0410  1.2775  2.5168  2.9241  3.2944  2.2261  1.8017  0.8977  1.2126  Grade 2  0.0000  0.0000  0.0000  0.0000  0.0000  0.0000  0.0000  0.0000  2.4927  3.8422  4.3555  4.7298  3.7864  3.6395  1.8747  2.3295  Grade 3  0.0000  0.0000  0.0000  0.0000  0.0000  0.0000  0.0000  0.0000  0.4116  �0.1815  0.3112  �0.0597  0.4006  0.4555  0.5900  0.2336  Girl in Grade 2  0.1680  0.6610  0.3990  0.9140  0.2310  0.4060  0.0520  0.4730  0.4915  0.5679  0.3304  �0.0954  0.1802  0.2975  0.3305  �0.0187  Girl in Grade 3  0.1800  0.5120  0.5290  0.8680  0.6900  0.5810  0.3030  0.9650  English (French) at  �0.0860  �0.2485  �0.0854  0.0213  0.2665  �0.1143  0.0616  0.0813  Home  0.5980  0.2460  0.6780  0.9040  0.1540  0.5100  0.6840  0.6560  0.0350  �0.3241  �0.0554  0.0334  0.1974  0.6226  0.3359  �0.0211  Bislama at Home  0.8750  0.2470  0.8430  0.8960  0.4330  0.0200  0.1180  0.9290  0.1319  0.0323  0.3918  0.4722  0.3168  0.3323  �0.1139  0.2845  Owns Textbook  0.3680  0.8600  0.0300  0.0050  0.0560  0.0520  0.4190  0.0680  0.0876  0.4277  0.1941  �0.2541  0.4305  �0.2246  0.2537  0.2344  Teacher Reads Aloud  0.7030  0.1120  0.5020  0.3650  0.0980  0.4260  0.2630  0.3180  Reading Material at  0.2090  0.0146  0.2100  0.4283  �0.2268  0.3815  0.4864  �0.0517  Home  0.1630  0.9420  0.2710  0.0080  0.1870  0.0140  0.0000  0.7570  Family Member  0.1248  0.1489  �0.0149  0.4874  0.0900  0.4722  0.5929  0.1921  Literate  0.5630  0.5670  0.9560  0.0490  0.7180  0.0660  0.0050  0.3880  0.0650  0.2206  �0.0661  0.0299  0.0684  �0.1572  0.1235  0.3922  Does Homework  0.7200  0.2870  0.7710  0.8980  0.7390  0.5120  0.4910  0.0320  0.2722  0.4682  0.2377  �0.0538  �0.0556  0.1874  0.2746  0.2238  Reads At Home  0.0980  0.0230  0.2500  0.7750  0.7690  0.3190  0.0770  0.2010  0.2675  0.0960  0.5080  0.3327  0.3752  0.2174  0.1851  0.3826  Attended Kindegarten  0.1900  0.7180  0.0440  0.1400  0.1060  0.3540  0.3470  0.0740  �0.2256  �0.1220  �0.2740  �0.3369  0.0279  �0.2334  �0.0845  �0.0070  Absent >1 Week  0.0920  0.4770  0.1060  0.0250  0.8550  0.1130  0.5000  0.9610  �1.4117  �0.8460  �2.6056  �4.2193  �2.5711  �4.3473  �2.4315  �1.3987  Constant  0.0000  0.0160  0.0000  0.0000  0.0000  0.0000  0.0000  0.0000                             pseudo R^2  0.2042  0.3213  0.4351  0.3744  0.3460  0.2872  0.1572  0.1833  N  1296.0000  1296.0000  1296.0000  1296.0000  1296.0000  1296.0000  1296.0000  1296.0000  Wald chi^2  352.5500  416.5300  737.0600  668.5900  592.5400  464.1900  282.2300  269.5300  P�value  0.0000  0.0000  0.0000  0.0000  0.0000  0.0000  0.0000  0.0000  Test for Goodness of Fit  Pearson Chi^2  526.6600  656.3300  589.9700  541.8600  564.9700  514.8400  503.6800  520.7200  goodness of fit  pvalue  0.4226  0.0000  0.0193  0.2552  0.0891  0.5679  0.6991  0.4953  % correct classification  75  83.72  86.27  78.7  82.72  79.32  69.21  76.85    Despite pilot testing, which did not indicate any such problems, the floor effects in Grade 1 were so pronounced that in most of the following analysis (unless otherwise specified) the grade has been removed. 31    29 VANEGRA French Results per Sub-test VANEGRA French results show reading gains across the three grades tested. Across grades, Francophone students appear to have a basic knowledge of letter and grapheme sounds, which progressively consolidates for almost all of them at the end of Grade 3 as seen by the low proportion of Grade 3 students scoring zero in sub-test 1. However, the average rate of progress is slow which may be hindering the development of word-level reading skills, oral reading fluency and comprehension. Looking at differences in performance between boys and girls, VANEGRA French results showed evidence of gender differences in some sub-tests in favor of girls, though the size and significance of the effect varies per skill tested. Gender differences are generally more pronounced in later grades, and these differences are statistically significant for familiar word reading, narrative passage reading and writing. For each of the sub-tests below, average results are presented for the sample as a whole and per grade, as well as without the proportion of zero-score students. There are two reasons behind this decision. On the one hand, some researchers argue that in cases where there is a large presence of zero-score students, overall means tend to underestimate the true average score of the population. On the other hand, the use of average means without zero-score students tends to overestimate the true average score. Since one of the purposes of this survey is to inform policy decisions over the establishment of temporary reference standards of oral reading fluency in the country, we believe it is important to present both results in order to inform policy discussions and future decisions over where and how to establish adequate reference reading standards for Vanuatu. Sub-test 1 – Initial Sound Recognition11 In order to read, each of us must turn the letters we see into sounds, sounds into words, and words into meaning. Successfully managing this process requires the ability to work in reverse; that is, students should also grasp that words are composed of individual sounds and understand the process of separating (and manipulating) words into sounds (Snow et al., 1998). The ability to identify sounds in words, to separate words into sounds, and to manipulate those sounds is termed phonemic awareness, found to play an important role in reading acquisition and the number one predictor of success in reading, better than socioeconomic status, preschool attendance, or reading time in the home (Share, Jorm, Maclearn, & Matthews, 1984). Testing for and remediating this skill is thus important for later reading development. Thus far, the EGRA tool has piloted an assessment of phonemic awareness in two different ways: using phoneme segmentation and identification of onset and rime sounds (first and last sounds). Phoneme segmentation – i.e. the division of words into phonemes - is one of the most complex skills of phonological awareness and should be emphasized in the early grades (Linan-Thompson & Vaughn, 2007). It is also one of the most predictive of later learning skills. Thus far, phoneme segmentation has proved difficult to administer. The VANEGRA instrument selected a simpler task – i.e. initial sound identification - to assess student’s ability 11 Throughout this section, sub-test description is based on the Early Grade Reading Assessment Toolkit (RTI, 2009). 30 in phoneme segmentation. A set of 10 French familiar words was selected from a list of words commonly used in children books, community life and school texts books available in Vanuatu. Students were asked to identify the initial sound in each of the words. The enumerator read each word aloud twice before asking the student to identify the sound and recording the answer as correct, incorrect, no answer in each item in the sub-test. Scores are defined as the average number of initial sounds identified from a list of 10 one- and two-syllable words included in the exercise. On average, students identified 5.4 initial sounds. Student performance in this sub-test showed one of the best improvements across grades (21% reduction in zero-score students from Grade 2 to Grade 3) with students scoring an average of 4.1 initial sounds at the end of Grade 2 to an average of 6.6 initial sounds at the end of Grade 3. Boys and girls performed similarly on the identification of initial sounds. Both boys read 5.1 and girls 5.6 initial sounds out of ten. The differences between boys and   girls in each grade are small and they are statistically significant only for Grade 3. Table 10 – Sub�test 1 Initial Sound Recognition: Results by Grade and gender    Table 10 – Sub-test 1 Initial Sound Recognition: Results by Grade and gender 95% Confidence  Subtest 1 �Number of correct initial  interval  N  Mean  SD  Min  Max  sounds identified   Lower  Upper  bound  Bound   Overall  866  5.38  3.66  0  10  5.14  5.63     Minus zero score students  669  6.97  2.50  1  10  6.78  7.16     Grade   Overall  433  4.14  3.66  0  10  3.79  4.48  Grade 2  Minus zero score students  289  6.20  2.70  1  10  5.89  6.51   Overall  433  6.63  3.20  0  10  6.33  6.93  Grade 3  Minus zero score students  380  7.56  2.15  1  10  7.34  7.77     Gender   Overall  435  5.12  3.68  0  10  4.77  5.46  Boys  Minus zero score students  327  6.81  2.56  1  10  6.53  7.08   Overall  431  5.65  3.61  0  10  5.31  5.99  Girls  Minus zero score students  342  7.12  2.44  1  10  6.86  7.38  Sub-test 2 – Grapheme Sound Identification Knowledge  of  how  letters  or  graphemes  (i.e.,  groups  of  letters  smallest  semantically  distinguishing unit in a written language) correspond  to sounds is another critical skill children  must  master  to  become  successful  readers.   Letter�sound  correspondences  are  typically  taught  through phonics�based approaches.  In this sub�test, students were asked to provide the sounds  of as many graphemes they could identify within a one�minute period.  A full set of graphemes  in  the  French  language  was  listed  in  random  order,  10  letter  sounds  to  a  row,  for  a  total  of  100  letter sounds.     Scores  in  sub�test  3  are  defined  as  the  number  of  correct  graphemes  identified  per  minute  (CGPM)  On  average,  students  correctly  read  19.1  grapheme  sounds  per  minute,  with  26.1  grapheme  sounds  attempted.  Grapheme  recognition  showed  significant  improvements  from  an  31 average  of  13  graphemes  recognized  in  Grade  2  to  25  at  the  end  of  Grade  3.    Girls  performed  slightly  better  with  an  average  of  21  CGPM  versus  18  CGPM  read  by  boys.  The  gender  difference in performance is statistically significant only in Grade 2.  Sub-test 2 – Grapheme Sound Identification Knowledge of how letters or graphemes (i.e., groups of letters smallest semantically distinguishing unit in a written language) correspond to sounds is another critical skill children must master to become successful readers. Letter-sound correspondences are typically taught through phonics-based approaches. In this sub-test, students were asked to provide the sounds of as many graphemes they could identify within a one- minute period. A full set of graphemes in the French language was listed in random order, 10 letter sounds to a row, for a total of 100 letter sounds. Scores in sub-test 3 are defined as the number of correct graphemes identified per minute (CGPM) On average, students correctly read 19.1 grapheme sounds per minute, with 26.1 grapheme sounds attempted. Grapheme recognition showed significant improvements from an average of 13 graphemes recognized in Grade 2 to 25 at the end of Grade 3. Girls performed slightly better with an average of 21 CGPM versus 18 CGPM read by boys. The gender difference in performance is statistically significant only in Grade 2.   Table 11 – Sub-test 2 Grapheme Identification: Results by Grade and gender. Table 11 – Sub�test 2 Grapheme Identification: Results by Grade and gender.  95% Confidence  Subtest 2 �Number of graphemes  interval  N  Mean  SD  Min  Max  identified per minute (CGPM)  Lower  Upper  bound  Bound   Overall  866  19.09  15.86  0  88  18.04  20.15     Minus zero score students  825  20.04  15.66  1  88  18.97  21.11     Grade   Overall  433  12.97  11.15  0  71  11.92  14.02  Grade 2  Minus zero score students  399  14.08  10.93  1  71  13.00  15.15   Overall  433  25.22  17.45  0  88  23.57  26.86  Grade 3  Minus zero score students  426  25.63  17.28  1  88  23.99  27.27     Gender   Overall  435  17.69  15.57  0  88  16.22  19.15  Boys  Minus zero score students  411  18.72  15.40  1  88  17.23  20.21   Overall  431  20.52  16.05  0  81  19.00  22.03  Girls  Minus zero score students  414  21.36  15.82  1  81  19.83  22.88    Sub-test 33 Sub-test Familiar –– Word Familiar Reading Word Reading Children  who  are  able  to  read  words  that  are  familiar  to  them  often  do  that  by  automatic  recognition. Children who   are Automated   word able to read   recognition words   in  reading that are familiar   allows to them   ado often   beginning   reader  to that by automatic   ‘read’  a  recognition. familiar  word Automated word  not   by  its  letters, recognition   but in reading   as  a a allows whole.     For beginning   this  to reader assessment,   highword ‘read’ a familiar not by  its �frequency familiar   letters, words but   were as a   selected whole. For this  assessment, from  early  high-frequency grade  reading   materials, familiar storyselected   were words   books, from school and early   visits grade   to  reading primary schools materials,  to ask story books,  teachers and school in  Grades visits  1 to 3 schools to primary about the to vocabulary  used ask teachers  during in Grades 1  their lessons. to 3 about the   Words  were  arranged  horizontally,  in  good  separation  from  each  other,  written  in  a  familiar  (lower case) font, comprising 10 rows, five familiar words per line.  Scores  in  sub�test  3  are  defined  as  the  number  of  correct  familiar  words  read  per  minute  32 (CFWPM).    VANEGRA  French  scores  in  sub�test  3  showed  a  weak  automaticity  in  word  reading,  an  ability  closely  associated  to  word  reading  in  the  oral  reading  passage  and  ultimately,  comprehension.  Students  read  an  average  of  10.9  familiar  words  per  minute  vocabulary used during their lessons. Words were arranged horizontally, in good separation from each other, written in a familiar (lower case) font, comprising 10 rows, five familiar words per line. Scores in sub-test 3 are defined as the number of correct familiar words read per minute (CFWPM). VANEGRA French scores in sub-test 3 showed a weak automaticity in word reading, an ability closely associated to word reading in the oral reading passage and ultimately, comprehension. Students read an average of 10.9 familiar words per minute correctly, with 16.5 words attempted. Yet, familiar word reading observed the greatest improvement across sub-tests and grades in the assessment: moving from Grade 1 to 2, the percentage of zero-scorers decreases from 84% to 19% (65% reduction in the number of zero-score students from Grade 1 to Grade 2), to only 5% of Grade 3 students scored zero (an additional reduction of 14%). Girls performed better with an average of 12.2 words per minute, while boys read an average of 9.6 words. Grade differences by gender are statistically significant for Grades 2 and 3.   Table 12 – Sub-test 3 Familiar Word Reading: Results by Grade and Gender Table 12 – Sub�test 3 Familiar Word Reading: Results by Grade and Gender  95% Confidence  Subtest 3 �Number of correct  interval  familiar words read per minute  N  Mean  SD  Min  Max  Lower  Upper  (CFWPM)  bound  Bound   Overall  865  10.91  11.39  0  57.69231  10.15  11.66     Minus zero score students  758  12.44  11.36  1  57.69231  11.64  13.25     Grade   Overall  432  5.62  6.23  0  47  5.03  6.21  Grade 2  Minus zero score students  348  6.98  6.22  1  47  6.32  7.63   Overall  433  16.18  12.85  0  57.69231  14.97  17.39  Grade 3  Minus zero score students  410  17.09  12.61  1  57.69231  15.86  18.31     Gender   Overall  435  9.58  10.51  0  57.69231  8.59  10.57  Boys  Minus zero score students  367  11.36  10.52  1  57.69231  10.28  12.43   Overall  430  12.24  12.09  0  55.4717  11.10  13.39  Girls  Minus zero score students  391  13.46  12.01  1  55.4717  12.27  14.66  Sub-test   4 – Invented Word Reading Invented word reading is a measure of decoding ability. Many children in the early grades learn to memorize Sub-test 4 – Invented Word Reading or recognize by sight a broad range of words. Children’s decoding skills are often assessed using reading Invented  word  reading  is  a  measure  of  decoding  ability.    Many  children  in  the  early  grades  lists learn of invented   to words   memorize   orthat cannot typically   recognize   by  sightbe read   a  broad sight recognition. by  range   of  words.  This allows for Children’s a purer measure   decoding of    skills  are oftenrecognition word   assessed  and using decoding   reading skills   of does than   lists reading invented comprehension   words   that  cannot paragraphs,   typically as children   read are   be by unable sight  recognition. to   This guess the next  allows word from for a context. the purer measure  of word This sub-test  recognition included 50 one- and decoding skills and  two-syllable  than invented  does  words, reading five   comprehension per row,   paragraphs, with the vowel-consonant   as  children patterns   are of letters   unable typical   to in the   guesslanguage. French   the  next  word  from  the  context.  This  sub�test  included  50  one�  and  two�syllable  invented  words,  five  per  row,  with  the  vowel�consonant patterns of letters typical in the French language.    Scores  in  sub�test  4  are  calculated  as  the  number  of  correct  invented  words  read  per  minute  33 (CIWPM).  Students read an average of 8.4 familiar words per minute correctly, with 13.6 words  attempted.  Grade  2  students  decoded  almost  4  words  (48%  reduction  in  the  number  of  zero� score students from Grade 1 to Grade 2), and Grade 3 students decoded an average of 13 correct  Scores in sub-test 4 are calculated as the number of correct invented words read per minute (CIWPM). Students read an average of 8.4 familiar words per minute correctly, with 13.6 words attempted. Grade 2 students decoded almost 4 words (48% reduction in the number of zero-score students from Grade 1 to Grade 2), and Grade 3 students decoded an average of 13 correct invented words per minute (an additional reduction of 30% in the share of zero-score students). Results suggest that while students are able to consolidate familiar word reading skills in Grade 2, the ability to decode is one they become familiar with later in Grade 3. Girls decoded more invented words per minute than boys (9.2 CFWPM versus 7.5 CFWPM), and grade differences by gender are statistically significant only for Grade 3.   Table 13 – Sub-test 4 Invented Word Reading: Results by Grade and Gender Table 13 – Sub�test 4 Invented Word Reading: Results by Grade and Gender   95% Confidence  Subtest 4 �Number of correct  interval  invented words read per minute  N  Mean  SD  Min  Max  Lower  Upper  (CIWPM)  bound  Bound   Overall  865  8.35  10.88  0  76.92308  7.63  9.08     Minus zero score students  572  12.63  11.17  1  76.92308  11.72  13.55     Grade   Overall  432  3.62  5.58  0  29  3.09  4.14  Grade 2  Minus zero score students  221  7.07  6.04  1  29  6.27  7.86   Overall  433  13.08  12.68  0  76.92308  11.88  14.27  Grade 3  Minus zero score students  351  16.13  12.20  1  76.92308  14.86  17.41     Gender   Overall  435  7.46  9.79  0  55.4717  6.54  8.38  Boys  Minus zero score students  275  11.80  10.02  1  55.4717  10.62  12.99   Overall  430  9.25  11.82  0  76.92308  8.14  10.37  Girls  Minus zero score students  297  13.40  12.11  1  76.92308  12.02  14.78    Sub-test 5a – Oral Passage Reading Sub-test 5a – Oral Passage Reading   reading Oralreading Oral   fluency fluency is a  is   a  measure measure   of  overall of overall   reading reading   competence: competence:   the the ability to  translate ability  to   translate letters   letters  into sounds, into  sounds,  unify  sounds  into  words,  process  connections,  relate  text  to  meaning,  and  make  unify sounds into words, process connections, relate text to meaning, and make inferences to fill in missing inferences  to  fill  in  missing  information  (Hasbrouck  &  Tindal,  2006).    As  skilled  readers  information (Hasbrouck & Tindal, 2006). As skilled readers translate text into spoken language, they combine translate  text  into  spoken  language,  they  combine  these  tasks  in  a  seemingly  effortless  manner.  these tasks in a seemingly effortless manner. Because oral reading fluency captures this complex process, it Because  oral  reading  fluency  captures  this  complex  process,  it  can  be  used  to  characterize  can be used to characterize overall reading skill. Poor performance on a reading comprehension tool would overall  reading  skill.    Poor  performance  on  a  reading  comprehension  tool  would  suggest  that  suggest that  the the  student hadstudent had   trouble trouble   with with decoding,   decoding, with reading   with  reading fluently   fluently enough   enough to comprehend,   to  comprehend, or   or with    with vocabulary. vocabulary.  Sub-test 5a produced a 58-word narrative passage from children’s reading materials. The narrative passage Sub�test  5a  produced  a  58�word  narrative  passage  from  children’s  reading  materials.  The  began where the characters are introduced, a middle section containing some dilemma, and an ending section narrative  passage  began  where  the  characters  are  introduced,  a  middle  section  containing  some  with an action dilemma,   and resolving   an  ending dilemma. the  section The   with   action provided passage   an resolvingthe basis   the for the  comprehension   dilemma. questions  The  passage  provided the  basis  for  the  comprehension  questions  presented  in  sub�test  5b.    Scores  in  sub�test  5a  are  calculated as the number of correct words read per minute in the oral reading passage (Table 8).   On 34   average,  students  were  able  to  read  18.6  correct  words  in  a  connected  text  per  minute  (CWCPM).   Students  attempted  22.5  words  on  average.  Looking  at  the proportion  of  zero�score  students in the sub�test, results suggest Francophone students begin consolidating their reading  presented in sub-test 5b. Scores in sub-test 5a are calculated as the number of correct words read per minute in the oral reading passage (Table 8). On average, students were able to read 18.6 correct words in a connected text per minute (CWCPM). Students attempted 22.5 words on average. Looking at the proportion of zero-score students in the sub-test, results suggest Francophone students begin consolidating their reading fluency skills by the end of Grade 2 (a 55% reduction in the number of zero-score students) with the best performance at the end of Grade 3 (an additional 18% reduction). This finding is also observed in terms of word of scores: at the the oral reading end of passage Grade 1, inwere students a minute. able to The read average increases only 1 correct word to 8 at of the the oral end ofpassage reading Grade 2in and a 29 at minute. end of increases theaverage The Girls Grade 3. to 8 at outperformed boys the end of Grade in29 2 and reading a narrative of Grade passage: at the end Boys read 3. Girls outperformed an average of 16.4 words per minute. Girls read an average of 20.7 words per minute. Gender boys in reading a narrative passage: Boys read an average of 16.4 words per minute. Girls read an average of differences are statistically significant only in Grades 2 and 3. 20.7 words per minute. Gender differences are statistically significant only in Grades 2 and 3. Table 14 – Sub-test 5a Oral Passage Reading: Results by Grade and Gender Table 14 – Sub-test 5a Oral Passage Reading: Results by Grade and Gender 95% Confidence Subtest 5a -Number of correct interval words read in a connected text per N Mean SD Min Max Lower Upper minute (CWCPM) bound Bound Overall 865 18.58 23.64 0 201.1765 17.01 20.16 Minus zero score students 724 22.20 24.24 1 201.1765 20.43 23.97 Grade Overall 432 7.93 10.26 0 74.04256 6.96 8.90 Grade 2 Minus zero score students 322 10.64 10.61 1 74.04256 9.48 11.80 Overall 433 29.21 28.04 0 201.1765 26.57 31.85 Grade 3 Minus zero score students 402 31.46 27.86 1 201.1765 28.74 34.18 Gender Overall 435 16.42 22.48 0 201.1765 14.31 18.53 Boys Minus zero score students 351 20.35 23.38 1 201.1765 17.91 22.80 Overall 430 20.77 24.60 0 152.7273 18.44 23.09 Girls Minus zero score students 373 23.94 24.93 1 152.7273 21.41 26.47 Sub-test 5b – Reading Comprehension Sub-test 5b – Reading Average reading Comprehension comprehension levels in sub-test 5b are largely explained by the performance in the oral reading passage. Without sufficient skills to read into the text, students focus on Average reading comprehension levels in sub-test 5b are largely explained by the performance in the oral reading words one-by-one, sometimes letter-by-letter. By the time they reach the end of the reading passage. Without sufficient skills to read into the text, students focus on reading words one-by-one, text, students have already forgotten what they read first. Scores in sub-test 5b are calculated as sometimes the percentage of correctBy letter-by-letter. the time answers inthey thereach the -5 sub-test end of the text, questions instudents total. Thehave already number offorgotten questionswhata they read first. Scores in sub-test 5b are calculated as the percentage of correct student received depended on the number of words read in sub-test 5a, so that students had to answers in the sub-test -5 questions respond only to The in total. number questions of questions related to the student received a segment depended of the text on the The they read. number of words early-stop read in marker was placed sub-test 5a, so at that 9 words – i.e. students had first row in only to respond the text - so that to questions thoseto related unable to correctly the segment read of the text any they of read. first 9 words the early-stop The markerreceived a zero-score was placed at 9 words in– this task. i.e. first rowStudents in the text that read - so that the first those unable9 words with to correctly at read least any of one word the first readreceived 9 words correctly were allowed a zero-score to continue in this task. Students with the the that read exercise. Those first 9 words reading with at least between 9 and 21 words received 1 question. Those reading between 22 one word read correctly were allowed to continue with the exercise. Those reading between 9 and 21 words and 26 received 2 questions, and those reading between 27 and 34 received 3 reading comprehension questions. Students reading between 35 and 47 received 4 questions and those reading above 47 words were asked all 5 reading comprehension questions. 35 On average, students answered 20% of the reading comprehension questions correctly, with received 1 question. Those reading between 22 and 26 received 2 questions, and those reading between 27 and 34 received 3 reading comprehension questions. Students reading between 35 and 47 received 4 questions and those reading above 47 words were asked all 5 reading comprehension questions. On average, students answered 20% of the reading comprehension questions correctly, with the average completed student the exercise, attempting average, understood on comprehension 1 reading slightly question. lesszero-score Excluding than half of the text students they (409 read. 68% students), the of students average scored scores to on zero increase this 44%, section which largely suggests as that a result students of their who reading completed the performance in the exercise, on average, oral reading understood passage. slightly Looking less than half of at textproportion thethe of students students, of zero-score they read. 68% on this suggest results scored zero even section largely though students start to consolidate fluency in reading as early as Grade 2, this skill begins to as a result of their reading performance in the oral reading passage. Looking at the proportion of zero-score consolidate later in Grade 3 (96% of Grade 1 students scored zero, followed by 75% in Grade 2 students, results suggest even though students start to consolidate fluency in reading as early as Grade 2, and 36% in Grade 3). In terms of the percentage of reading comprehension by grade, in Grade 2, this skill begins to consolidate later in Grade 3 (96% of Grade 1 students scored zero, followed by 75% in the average student understands 8% and by Grade 3, the average student understands about Grade 2 and 36% in Grade 3). In terms of the percentage of reading comprehension by grade, in Grade 2, 32% of the text. Boys and girls performed similarly in reading comprehension: boys scored an the average average of student 22% by8% understands 18% versus and girls; by Grade both 3, the average attempting student to answer understands an average of 1 about 32%out of the question of text. Boys and girls performed similarly in reading comprehension: boys scored an average of 5. Girls slightly outperformed boys in Grades 2 and 3 but these differences are not statistically 18% versus 22% by girls; both attempting to answer an average of 1 question out of 5. Girls slightly outperformed boys significant. in Grades 2 and 3 but these differences are not statistically significant. Table 15 – Sub-test 5b Reading Comprehension: Results by Grade and Gender Table 15 – Sub-test 5b Reading Comprehension: Results by Grade and Gender 95% Confidence Subtest 5b -Percentage of overall interval reading comprehension in a N Mean SD Min Max Lower Upper connected text bound Bound Overall 866 0.20 0.28 0 1 0.18 0.22 Minus zero score students 392 0.44 0.25 0.2 1 0.41 0.46 Grade Overall 433 0.08 0.16 0 1 0.06 0.09 Grade 2 Minus zero score students 109 0.31 0.17 0.2 1 0.27 0.34 Overall 433 0.32 0.31 0 1 0.29 0.35 Grade 3 Minus zero score students 283 0.49 0.26 0.2 1 0.46 0.52 Gender Overall 435 0.18 0.27 0 1 0.15 0.20 Boys Minus zero score students 182 0.43 0.25 0.2 1 0.39 0.47 Overall 431 0.22 0.28 0 1 0.19 0.24 Girls Minus zero score students 210 0.44 0.25 0.2 1 0.41 0.48 A closer look at differences in comprehension between students who were asked to answer A closer look at differences in comprehension between students who were asked to answer questions based on questions based on the oral reading passage show important differences in the share of students the oral reading passage show important differences in the share of students in each category and the average in each category and the average comprehension based on questions asked. Table 16 shows the distribution ofbased comprehension on answers correct questionsdepending asked. Tableon16 theshows the distribution number of questionsof correct asked. answers depending on the number of questions asked. 36 As seen in column “0�, 474 Grade 2 and 3 students were not given any questions – zero questions asked – because they could not read correctly any of the first 9 words in the passage. Of the 159 students that correctly read between 1 – 9 words in the first segment of the oral reading passage, 54% (86 students) were able to answer the question correctly. On average, the more fluent in reading students are, i.e. those that received 3 or more questions; the better their comprehension, as they are able to provide correct answers to most, if not all, the questions received. The proportion of students in the 0 and 1 correct answers (rows) is zero for columns 4 and 5. Among the most fluent (column 5), 100% of Grade 2 and 3 students are able to understand all questions they are asked. Table 16 - Percentage of Correct Answers in Sub-test 5b Table 16 - Percentage of Correct Answers in Sub-test 5b # questions asked # questions correct 0 1 2 3 4 5 Total 0 474 73 1 0 0 0 548 1.000 0.459 0.010 0.000 0.000 0.000 0.633 1 86 33 5 0 0 124 0.541 0.333 0.078 0.000 0.000 0.143 2 65 29 0 0 94 0.657 0.453 0.000 0.000 0.109 3 30 21 0 51 0.469 0.477 0.000 0.059 4 23 0 23 0.523 0.000 0.027 5 26 26 1.000 0.030 Total 474 159 99 64 44 26 866 Note: Percentage values reflect column percentages. As seen in column “0�, 474 Grade 2 and 3 students were not given any questions – zero Sub-test 6 – Listening questions asked – because they Comprehension could not read correctly any of the first 9 words in the passage. Of the 159 students that correctly read between 1 – 9 words in the first segment of the oral Assessment of listening comprehension does not involve any reading from the student but involves the reading passage, 54% (86 students) were able to answer the question correctly. On average, the processing more fluentof in oral language reading are, only. information students Testing i.e. those thatofreceived comprehension listening 3 separately the from or more questions; reading better their comprehension is important due to the different ways in which learners approach, comprehension, as they are able to provide correct answers to most, if not all, the questions process, and respond received. to The text. More proportion importantly, of students listening in the 0 and comprehension is an correct answers 1 important (rows) contributor is zerocomprehension, to reading for columns 4 and 5. which Among tends the with to increase most fluent reading (column 5), 100% of Grade 2 and 3 students are able to acquisition. understand all questions they are asked. The narrative passage in VANEGRA’s sub-test 6 was 37 words long and narrated an activity or event familiar to Ni-Vanuatu children. Students then responded to oral comprehension questions – 5 in total. Scores in sub-test Sub-test 6 –calculated 6 are Listening as the number of correct answers in the sub-test. On average, students in the Comprehension Assessment sample of listening responded comprehension to 1.5 of the does (Table 5 questions correctly not involve any reading 17). Excluding from zero-score the student students, but the average involves the processing of oral language information only. Testing of listening comprehension separately from reading comprehension is important due to the different ways in which is learners approach, process, and respond to text. More importantly, listening comprehension 37 an important contributor to reading comprehension, which tends to increase with reading acquisition. student correctly answered 2.3 questions. Students in Grade 3 performed the best with an average of 1.8 Students correct   in  Grade answers (just  3   performed under   the  best 40% average   with  an  average comprehension),   of  1.8 followed by Grade  answers   correct   (just 2 with 1.1   under correct   40%  answers over   average (just comprehension), 20%   followed  by average comprehension).   Grade There were 2 no  with   1.1  correct statistically   answers significant   (just  over differences   20% in the   average  performance boys and girls on  listening comprehension). of There  were   no  statistically comprehension.   significant of The performance differences   in  the boys and girls   performance on listening   of  boys  comprehension and is   girls similar,   on and   listening differences comprehension. in performance   The in each   performance grade   of  boys  and  girls  on  listening  are not significant. comprehension is similar, and differences in performance in each grade are not significant.        Table 17- Sub-test 6 Listening Comprehension: Results by Grade and Gender Table 17� Sub�test 6 Listening Comprehension: Results by Grade and Gender  95% Confidence  Subtest 6 �Number of listening  interval  comprehension questions answered  N  Mean  SD  Min  Max  Lower  Upper  correctly  bound  Bound   Overall  866  1.46  1.45  0  5  1.36  1.55     Minus zero score students  556  2.27  1.20  1  5  2.17  2.37     Grade   Overall  433  1.11  1.32  0  5  0.99  1.23  Grade 2  Minus zero score students  235  2.05  1.13  1  5  1.90  2.19   Overall  433  1.80  1.50  0  5  1.66  1.95  Grade 3  Minus zero score students  321  2.43  1.23  1  5  2.30  2.57     Gender   Overall  435  1.40  1.44  0  5  1.26  1.53  Boys  Minus zero score students  271  2.24  1.20  1  5  2.10  2.39   Overall  431  1.52  1.47  0  5  1.38  1.66  Girls  Minus zero score students  285  2.29  1.21  1  5  2.15  2.44    Sub-test Sub-test 7 Dictation 7––Dictation Dictation assessment is frequently used by teachers to test both oral comprehension and writing  Dictation assessment is frequently used by teachers to test both oral comprehension and writing skills. skills.   Students’  ability  to  hear  sounds  and  correctly  write  the  letters  and  words  corresponding  Students’ ability to hear sounds and correctly write the letters and words corresponding to the sounds they to  the  sounds  they  hear  demonstrates  their  success  with  the  alphabetic  principle.   The  dictation  hear demonstrates their success with the alphabetic principle. The dictation sentence in the VANEGRA sentence in the VANEGRA French was 11 words long (“Je vais au jardin pour planter un taro et  French was 11 words long (“Je vais au jardin pour planter un taro et un bananier�). Students received a un  bananier�).  Students  received  a  weighted  score  capturing  the  accuracy  for  vowel  and  weighted consonant score capturing   sounds, the accuracy   spelling,   spacingfor vowel   and and consonant   direction sounds,   of  text, spelling,     capitalization, spacing and direction and  punctuation. of       In text, capitalization, addition, and punctuation.   we  estimated In addition,   the  number we  and   of  letters estimated the number   full  words of letters   written,   the  and full words percentage written,   of   those  that  were  correctly  written,  and  the  number  of  pictograms   used  by  children  to  represent  the the percentage of those that were correctly written, and the number 12 of pictograms 12 used by children to   sentencethe represent  given.    sentence given.                                                          12 A pictogram is a pictorial representation of words used in Vanuatu to “smooth� student’s transition into alphabetic-based 12 A pictogram is awriting. pictorialAlthough standardized, not aof representation words used inthere is a basic Vanuatu set of pictograms to “smooth� commonly student’s transition used to represent into alphabetic-based the most writing. common Although notwords and verbs used a standardized, therein isthe country. a basic set of pictograms commonly used to represent the most common words and verbs used in the country. 38 41    Scores  in  sub�test  7  are  calculated  using  weights13  to  create  a  variable  with  a  maximum  score  of  100%.   For  easier  interpretation  of  writing  scores,  we  converted  students’  raw  dictation  scores  Scores into in sub-test   a  weighted 7 are calculated   average   of  their using weights13 to   performance   on create   each a  variable question. with a maximum     Each   word score for  the of  100%. spelling For  component   received easier interpretation of  10   points writing   if  answered scores, we converted  correctlystudents’  and raw 5  if dictation the  response scores   was into   partially a weighted   correct. average   Spacing   and  capitalization of their performance on each question. received  3   points Each word   each, for the   while spelling  thecomponent  correct  direction received   and   use  of 10 points if answered the  full  stop   received correctly and 5   2if   points   each  for the response was   apartially   total  of   100  points correct. Spacing   possible   for  the  entire and capitalization   section. received    On each, 3 points average, while  students   wrote   2.6   out   of   8   items   correctly,   with   a   weighted the correct direction and use of the full stop received 2 points each for a total of 100 points possible for the  score   of   26   out   of   100.     As   in   all   sub � tests, entire   student section. On average,  students   performance in  dictation wrote  was2.6 out of 8  in   positive   terms items   of  scores correctly, with   and   overall score a weighted progression. of 26 out    of 100.   As Students Grade in  in   3  performed all sub-tests, student   the   best  with performance in   an   average dictation was   score positive  of in 3.2   out  terms of of   8  items scores and   (41%   overall weighted   score),   followed   by   Grade   2   students   with progression. Students in Grade 3 performed the best with an average score of 3.2 out of 8 items (41%   an   average   score   2   out   of   8   items   (25%   weighted  score).  Girls  outperformed  boys  on  dictation,  and  this  difference  is  statistically  weighted score), followed by Grade 2 students with an average score 2 out of 8 items (25% weighted score). significant in Grades 2 and 3.  Girls outperformed boys on dictation, and this difference is statistically significant in Grades 2 and 3.     Table 18 - Sub-test 7 Dictation:   Results by Grade and Gender Table 18 –Sub�test 7 Dictation: Results by Grade and Gender  Subtest 7 � Score of overall early  writing skills (spelling and basic  N  Mean  SD  Min  Max  conventions)   Overall  866  2.62  1.87  0  8     Minus zero score  751  3.02  1.68  1  8  students     Grade   Overall  433  1.97  1.59  0  8  Grade 2  Minus zero score  350  2.44  1.41  1  8  students   Overall  433  3.26  1.90  0  8  Grade 3  Minus zero score  401  3.52  1.72  1  8  students     Gender   Overall  435  2.37  1.84  0  8  Boys  Minus zero score  363  2.85  1.64  1  8  students   Overall  431  2.86  1.87  0  8  Girls  Minus zero score  388  3.18  1.69  1  8  students    Out of Out the 43 of the letters  in 43 letters inthe  thesentence,  sentence,students  students  were were able to to  able an  average  write write an average  of 23.5 of 23.5  letters, 23.2 23.2 letters,  of  of which which  were written were written  correctly. correctly.   Out Out of the 11 of  the 11 words in  the words sentence,  in the students sentence,  students wrote  wrote of an average an6.7  average  of  full words, 6.7 4.1 full  words, of which  of which  4.1spelled were  were  correctly spelled and about correctly  and about 2.6 phonetically. The 2.6  phonetically. proportion  The using of students proportion  of  pictograms students   using   pictograms   was   very   low   (about   0.17   words   as   a   pictogram). was very low (about 0.17 words as a pictogram). Table 19 shows evidence of progression across grades in     Table   19   shows   evidence of progression across grades in all of the items estimated: the number of correct letters  all of the items estimated: the number of correct letters and full words increases as students go from Grade   full words written) to Grade 3 (30 correct letters and 6 correct full words).                                                       2 (17 correct letters and 3 correct 13 Due to similarities in the components used in the VANEGRA instrument, the weights for this score follow the score used in the Guyana EGRA test. 13 42  Due to similarities in the components used in the VANEGRA instrument, the weights for this score follow the score used   in the Guyana EGRA test. 39 and  full and full  words increases  as words  increases students  go as  students from  Grade go  from Grade  2 (17  correct correct  letters 2  (17 and  3 letters  and correct  full 3  correct words   full  words written)   to  Grade   3   (30  correct   letters   and  6   correct written) to Grade 3 (30 correct letters and 6 correct full words).     full   words).          Table 19 - Sub-test 7 Dictation: Number of letters and full words (total and correct) written by grade Table  19 Table 19  –Sub test  7 –Sub��test Dictation:  Number 7  Dictation: of  letters Number  of letters  and full  words words  (total and  full and  correct) (total  and written  by correct)  written grade   by  grade 2   Grade  2 Grade 3   Grade  3 Grade Total   Total       Mean   Mean    SD   SD Mean   Mean    SD   SD Mean   Mean    SD   SD Number  of Number of  letters written   letters  written 17   17 14   14 30   30 14   14 23   23 15   15 Number  of Number of  correct letters  written correct  letters written   17   17 14   14 30   30 14   14 23   23 15   15 Number  of Number of  full words  written full  words written   5   5 44   8   8 4   4 7   7 44   Number  of Number of  full correct  words full  correct words  written written   3   3 3   3 6   6 3   3 4   4 3   3 Number  of Number of  words written  phonetically words  written phonetically   2   2 2   2 3   3 3   3 3   3 3   3 Number  of Number of  words written  as words  written as  a pictogram   a  pictogram 0   0 1   1 0   0 1   1 0   0 1   1    Boys wrote an average of 21.3 letters, 21 of them correctly; and girls wrote an average of 25.7 letters, 25.5 Boys  wrote Boys wrote  an average  of an  average 21.3  letters, of  21.3 21  of letters,  21 of  them correctly;  and them  correctly; girls  wrote and  girls wrote  an average  of an  average 25.7   of  25.7 of them  correct. letters, Boys   25.5  of  them wrote average an    correct. of 6 full Boys  wrote   anwords, 3.6  correctly   average of  6  fullwritten.   words,   3.6 write Girls an average correctly of 7.4   written. full    Girls letters,  25.5  of  them  correct.   Boys  wrote  an  average  of  6  full  words,  3.6  correctly  written.  Girls  write   with words, 4.7 correctly an   average average of  written. More with  words girls  wrote phonetically. Boys   and girls  wrote less than 1 word  write an    of 7.4   full 7.4 full words,    words,   with 4.7   correctly 4.7 correctly written.    written.      More More girls     girls wrote wrote    words words   phonetically. as a pictogram   phonetically.    onBoys Boys   and girls  wrote and  girls average. wrote  less than  1 less  than 1  word word  asas  a a  pictogram pictogram  on average.   on  average.    Table 20  20 Table Table –Sub-test 20   –Sub 7 –Sub��test test  7 7 Dictation: Number Dictation:   Dictation:  Number of Number  of letters  and of  letters and and full  words full   full words words (total   (total (total and   and and correct) correct)   correct) written written   written byby   by gender gender   gender   Boys   Boys Girls   Girls          Mean   Mean SD   SD Mean   Mean    SD   SD Number  of Number letters  written of  letters written   21.333   21.333 15.622   15.622 25.689   25.689 14.141   14.141 of  correct Number  of Number letters  written correct  letters written   20.991   20.991 15.597   15.597 25.528   25.528 14.159   14.159 Number  of Number full  words of  full written   words  written 5.979   5.979 4.455   4.455 7.414   7.414 3.894   3.894 Number  of Number of  full correct  words full  correct written   words  written 3.623   3.623 3.332   3.332 4.657   4.657 3.362   3.362 of  words Number  of Number written  phonetically words  written phonetically   2.354   2.354 3.038   3.038 2.792   2.792 3.265   3.265 Number  of Number words  written of  words as  a written  as a   0.222   0.222 1.053   1.053 0.117   0.117 0.766   0.766 pictogram pictogram       Forthe For writing   the components   writing assessed   components in the   assessed sub-test,   in the   the  sub rate �test, of student   the   rate  ofnon-response   student  nonis quite high,   isbut �response the    quite For  the  writing  components  assessed  in  the  sub�test,  the  rate  of  student  non�response  is  quite  rate   but  theas decreases high, students   rate progress   decreases   as from Grade students 2 to 3. 26%   progress of     from students in Grade  2 the   to sample   3.     26%(335 students)  did   of  students not  in  the high,  but  the  rate  decreases  as  students  progress  from  Grade  2  to  3.    26%  of  students  in  the  sample write sample students)   (335 (335  students) anything at all, half of did   did them  not in  write not write  anything Grade anything   at 1. Excluding all,  half at  all, students of half  of who  them them   in produced Grade in  Grade a blank1.     Excluding   1. Excluding dictation   students students section, 15%   produced who  produced correctly who blank  dictation a  blank   a used capitalization, dictation 43% used   section, section, spacing 15%   15%   correctly correctly, 75%    correctly used in wrote used  capitalization, capitalization, the right direction, 43%   43% used   used and 7% spacing correctly     spacing correctly, used the full correctly, 75%   75% stop  wrote wrote (Figure  in in   the 5). the  right 14 right  direction, direction,  and and  7% 7%  correctly correctly  used the  full used  the stop  (Figure full  stop 5).14   (Figure  5). 14 While half of the students acquired an understanding of the orientation to write as early as Grade 1 (50%    had acquired this knowledge at the end of Grade 2. Spacing between                                                                                                             correct answer), most of them (81%) 14 14 This section This tends sectionto excludes excludes zero-score zero-score students includes those and includes few Grade 1 those few students who 1 students who produced produced an answer. an answer. words consolidate in Gradeand later students 3 (74%). The use of Grade the full stop, on the other hand, appeared not to develop in most students across grades (Figure 6).15 43   43    14 This section excludes zero-score students and includes those few Grade 1 students who produced an answer. 15 During the pilot of the instrument, the team monitored that enumerators were not dictating the final point to students, as it is often the way teachers do during dictation lessons. 40 While  half  of  the  students  acquired  an  understanding  of  the  orientation  to  write  as  early  as  Grade  1  (50%  correct  answer),  most  of  them  (81%)  had  acquired  this  knowledge  at  the  end  of  Grade 2.  Spacing between words tends to consolidate later in Grade 3 (74%).  The use of the full  stop, on the other hand, appeared not to develop in most students across grades (Figure 6). 15    Figure 8– Distribution of student responses to the writing items (total sample) Figure 8– Distribution of student responses to the writing items (total sample)      Figure 9– Distribution of student responses to the writing items (per grade) Figure 9– Distribution of student responses to the writing items (per grade)                                                           15   the  spelling  component,  students  were  marked  based  on  their  spelling  of  “au�,  “planter�,  For ForDuring the pilot the spelling of the instrument, component, students team monitored the were marked based that enumerators were not on their spelling of dictating the final“et� “au�, “planter�, point andto “et�   and students, it is often the     “bananier�. as Non way �response teachers   is  also do during   high lessons. for  this  component,  with  the  rate decreasing  as  dictation “bananier�. Non-response is also high for this component, with the rate decreasing as students progress students  progress  from  Grade  1  to  later  grades.    Excluding  students  who  left  every  dictation  from Grade 1 to later 44  question   blank,   19%grades.   correctly   spelled students Excluding the  word who left  3%   “au�, every dictation   correctly question   spelled blank,   the   word 19% correctly   “planter�,     spelled the word “au�, 3% correctly spelled the word “planter�, 28% correctly spelled the word “et� and 8% 28% correctly spelled the word “et� and 8% correctly spelled the word “bananier�. Performance  correctly in  spelling spelled   items the word “bananier�.   showed   significant Performance   progression in spelling   across items showed grades,   with significant the  largest progression across    gains  observed at the end grades, with  of  Grade the largest (Figures  3 gains  7 and  observed 8). at    end of Grade 3 (Figures 7 and 8). the   41 Figure 10– Distribution of student responses to spelling items (total sample)   Figure 10– Figure 10–Distribution Distribution ofstudent of student responses to responses to spelling spelling items items (total (total sample) sample) Spelling NR Incorrect Partial Correct 3% 8% 19% 12% 28% 20% 16% 28% 10% 21% 18% 12% 57% 51% 47% 50% "au" "planter" "et" "bananier"     Figure 11– Figure  11–Distribution  ofstudent  Distributionof  student responses to responses to the  the writing  writing items  items (per  (per grade)  grade)   Figure 11– Distribution of student responses to the writing items (per grade) Correct Items in Spelling Grade 1 Grade 2 Grade 3 37% 30% 29% 21% 18% 16% 11% 6% 3% 0% 1% 1% "au" "planter" "et" "bananier"     Differences in Performance by Grade and Gender As previously stated, VANEGRA French results showed evidence of gender differences in some  sub�tests in favor of girls, though the size and significance of the effect varies per skill tested.  45In  order  to  identify  the  skills  and  the  grades  where  gender  differences  take  place,  we  explored  42 performance  differences  by  gender  for  each  grade.  This  analysis  pertains  to  Grades  2  and  3  only.  Differences in Performance by Grade and Gender As previously stated, VANEGRA French results showed evidence of gender differences in some sub-tests in favor of girls, though the size and significance of the effect varies per skill tested. In order to identify the skills and the grades where gender differences take place, we explored performance differences by gender for each grade. This analysis pertains to Grades 2 and 3 only. We observed that differences in performance by gender starts to appear in Grades 2 and 3. These differences suggest that girls step into word-level reading and oral reading fluency faster than boys, as observed in the how boy’s performance lag as grade increases. Moreover, in Grade 3, girls consolidate their advantage in reading fluency, writing and even decoding skills. Figures 9 to 11 below present average scores in VANEGRA French in three important sub-tests. Further exploration of the effect of gender on the acquisition of early literacy skills takes place in Chapter 4. Familiar Word Reading Girls read more familiar words per minute than boys, and this difference is statistically significant for Grades 2 and 3. Girls read an average of 12.2 familiar words per minute, they attempted to read 17.7. Boys read an average of 9.6 correct words per minute; they attempted to read 15.2. In Grade 2, girls read an average of read 1.5  15.2. In familiar  Grade words  2, than more  readand girlsboys in Grade of  an average 3, 1.5  familiar girls read an  additional words more  than boys 4 familiar per in  and words  Grade minute  3,  more girls  than read The boys. differences  4 an  additional in  familiar  words Grades 2 and 3  are per  minute  more statistically   than  boys. significant,   The  differences suggesting   in  Grades that girls transition into  2 and word  3 are statistically reading  significant, skills faster than  suggesting boys. (See Table 12)  that girls transition into word reading skills faster  than boys. (See Table 12)    Figure 12– Differences in performance between boys and girls by grade, measured by the number of Figure 12– Differences in performance between boys and girls by grade, measured by the number of correct  correct familiar familiar words  words  readread per minute  per minute (CFWPM)  (CFWPM)       Oral Passage Reading Oral Passage Reading  Girls outperformed boys in reading a narrative passage. Overall, boys read an average of 16.4 words per Girls  outperformed  boys  in  reading  a  narrative  passage.   Overall,  boys  read  an  average  of  16.4  minute; they words  per attempted   minute; 21.1.   they Girls read   attempted an average   21.1. of 20.7   Girls  read   an words per average minute;   of they attempted   20.7  words 24. This   per  minute; is    they significant attempted in   This  2 Grades   24. is and 3. In Grade significant 2, girls   in  Grades   andalmost   2read   2, more 2 words   3.  In  Grade than  boys girls  read almostin   the oral reading 2  words   more  than boys in the oral  reading passage and girls  in Grade 3 read about 7  words per  minute more passage and girls in Grade 3 read about 7 words per minute more than boys in the same grade. These  than  boys results   in  the suggest   same girls   grade. develop   These fluency   results in reading   suggest faster than  girls boys.  develop  fluency  in  reading  faster  than  boys.  43 Figure 13– Differences in performance between boys and girls by grade, measured by the number of correct  words read in a connected text per minute (CWCPM)  Figure 13– Differences in performance between boys and girls by grade, measured by the number of correct words read in a connected text per minute (CWCPM)     Dictation (Weighted score)   Gender differences in Grades 2 and 3 are statistically significant in favor of girls. In Grade 2, girls read almost Dictation (Weighted score)  2 words more than boys in the oral reading passage and girls in Grade 3 read about 7 words per minute more Gender  differences  in  Grades  2  and  3  are  statistically  significant  in  favor  of  girls.  In  Grade  2,  girlsboys than   readin the same   almost grade.   2  words Boys,   more on average,   than   boys  in   the  orala obtained weighted   reading score of   passage 29%   and and  in   girls girls scored   Grade   3 36%. read  These results suggest that girls acquire writing skills faster than boys do in Grades 2 and about  7  words  per  minute  more  than  boys  in  the  same  grade.  Boys,  on  average,  obtained  a  3.  weighted  score  of  29%  and  girls  scored  36%.  These  results  suggest  that  girls  acquire  writing  skills Figure faster  than boys  14– Differences do in  in Grades 2 performance  and 3.  boys and girls in dictation by grade, as a weighted score between Figure 14– Differences in performance between boys and girls in dictation by grade, as a weighted score             44 Summary of Assessment Results VANEGRA  French  results  show  reading  gains  across  the  three  grades  tested.   As  Francophone  Summary of Assessment Results VANEGRA French results show reading gains across the three grades tested. As Francophone students move from Grade 1 to 3, they improve their competence in all skills tested. Nonetheless, the average rate of progress is slow in the development of word-level reading skills, oral reading fluency and reading comprehension. Looking at differences in performance between boys and girls, assessment results showed that boys and girls perform similarly in Grade 1, but girls outperform boys in word-level skills at statistically significant levels. 45 4   Chapter 4: Performance in Oral Reading Fluency Performance Chapter 4: and in Oral Reading Fluency Reading Comprehension   and Reading Comprehension As  stated  before,  oral  reading  fluency  is  a  measure  of  overall  reading  competence:  the  ability  to  As stated before, translate   into lettersoral   sounds, reading   unify fluency   sounds is a measure   into of   words, overall reading   process competence: the ability     connections, torelate   text translate   to  letters meaning,   and   make   inferences   to   fill  in   missing   information   (Hasbruck into sounds, unify sounds into words, process connections, relate text to meaning, and make inferences to   and   Tindal,   idem;   Fuschs   2001).   This   et  al,information fill in missing   is  not  to (Hasbruck   say, and   however, Tindal, idem;   that   oral Fuschs   reading et   fluency al, 2001). This is  is   the not say,   to  only predictor however,  �  that or the most important � of reading comprehension among readers.  Because oral reading fluency  oral reading fluency is the only predictor - or the most important - of reading comprehension among readers. captures  this  complex  process  and  it  is  strongly  associated  with  both  listening  and  reading  Because oral reading fluency captures this complex process and it is strongly associated with both listening comprehension, it is frequently used as a marker of overall reading ability.  and reading comprehension, it is frequently used as a marker of overall reading ability. Before  VANEGRA, Before VANEGRA,   oral oral   reading reading   fluency fluency had  not had   not  been been   measured measured   in  Vanuatu. in Vanuatu.    Thus, order  to Thus, in in inform order  the to  inform  the  establishment  of  reference  standards  for  early  grade  reading  under  VERM,  we  establishment of reference standards for early grade reading under VERM, we sought to identify the oral sought  to  identify  the  oral  reading  fluency  levels  where  Ni�Vanuatu  beginning  readers  reading fluency levels where Ni-Vanuatu beginning readers demonstrate high reading comprehension levels. demonstrate high reading comprehension levels. VANEGRA French results showed evidence of  VANEGRA French results showed evidence of a positive correlation between oral reading fluency– sub- a  positive  correlation  between  oral  reading  fluency–  sub�test  5a �  and  reading  comprehension  –  test 5a - and reading comprehension – sub-test 5b -: better fluency in reading appears to contribute to sub�test 5b �: better fluency in reading appears to contribute to better comprehension in reading,  better comprehension although in reading,   this  relationship although   is  not this   linear. relationship   In   Figure  12 is not linear.   below, In Figure   average 12 below,   reading average reading   comprehension   comprehension levels  of 60% and levels of 60%  above  startand  at above a minimumstart at fluency a minimum  levelfluency  of 35level  CWCPM.of 35 CWCPM.     Figure 15 –Average Scores in Oral Reading Fluency (sub-test 5a) and Figure 15 –Average Scores in Oral Reading  Reading Comprehension  Fluency (sub-test  (sub�test 5a) 5b)  Comprehension (sub�test 5b)   and Reading   50    46 Using a reference standard of 45 CWCPM to classify students as “fluent readers�, students are able to read between 46 and 152 correct words in a connected text per minute (CWCPM) with an average fluency of Using  a  reference  standard  of  45  CWCPM  to  classify  students  as  “fluent  readers�,  students  are  68.8 CWPM (see Figure 13 below). At this level, 42% of these fluent students were clustered around top able  to  read  between  46  and  152  correct  words  in  a  connected  text  per  minute  (CWCPM)  with  comprehension levels – i.e. correct answers to 4 and 5 questions received. an  average  fluency  of  68.8  CWPM  (see  Figure  13  below).    At  this  level,  42%  of  the  students  clustered around top comprehension levels – i.e. correct answers to 4 and 5 questions received.  Figure 16 – Average Reading Comprehension Levels in Fluent Students (N=110) Figure 16 – Average Reading Comprehension Levels in Fluent Students (N=110)      Using a reference standard of 45 CWCPM to classify students as “fluent readers�, fluent students account Using for  a 13%   reference of   standard the sample.   ofby Differences   45   CWCPM grade   to suggest   classify fluency in   students reading   as  “fluent emerges at the  end readers�,   fluent of Grade 3 but  students only   account among for Grade 24% of  the ofstudents. 13%  3   the  sample.   Differences Of the   by  grade 110 fluent students, 92%  suggest fluency were in  Grade 3,  8% in  reading in Grade  emerges 2. In the  sample at  the  end   of  Grade of Grades but   3 3, 2 and   only 16% of  among the girls  23% and   of  the only 10%  Grade   3boys of the   students. could    beOf   the  110  fluent considered fluent  students, in reading. 92%  were By the end in of Grade 3,  Grade 3, 8% 29% in of Grade girls in 2.   In the Grade  samplethe 3 reached  of Grades  2 and 3,  fluency standard 31% of versus  the of girls 18% the  and  only boys   20% in the same  the   boys  could  be  considered  fluent  in  reading.    By  the  end  of  Grade  3,  29%  of  grade. girls in Grade 3 reached the fluency standard versus 18% of the boys in the same grade.  Differences between fluent and less than fluent students are considerable in both accuracy and fluency. As   expected, important differences in both fluency and reading comprehension emerge between these two groups. While less-than-fluent students – i.e. those reading between 0 and 44 CWCPM - attempted to read about 17.6 words in a minute and got 11.2 of them correct, for an oral reading fluency average of 11.2 CWCPM, fluent students attempted to read 56 words and got 55 words correct (98% accuracy), for an oral reading fluency average of 68.8 CWCPM – a difference of around 58 CWCPM in the oral reading passage between the two groups. In terms of average reading comprehension, less-than-fluent students showed a 47 51    Table 21- Distribution of Students by Fluency Less than Fluent Students Fluent Students Reading 0 to 44 words per >45 correct words per minute minute Frequency Proportion Frequency Proportion SD N Overall 755 0.87 110 0.13 0.011 866 Grade2 424 0.98 9 0.02 0.007 433 Grade3 fluency 0.76 only one 331 them to receive level that allowed 101 0.24 an average comprehension question, reaching 0.020 433 of 13%. Boys 391 0.90 43 0.10 0.014 435 Conversely, fluent students received all questions and were able to understand about 64% of the text. Girls 363 0.84 67 0.16 0.018 431 Table 21- Distribution of Students by Fluency Boys | Grade 2 212 Table 21- Distribution 0.98 4 of Students by Fluency 0.02 0.009 217 Boys | Grade 3 178 Less 0.82 than Fluent Students 0.18 39 Fluent Students 0.026 218 Girls | Grade 2 0.98 per 211 0 to 44 words Reading 4 0.02 0.010 216 >45 correct words per minute Girls | Grade 3 151 minute 0.71 63 0.29 0.031 215 Frequency Proportion Frequency Proportion SD N Overall 755 0.87 110 0.13 0.011 866 Grade2 between Differences 424fluent and0.98 9 students are less than fluent 0.02 considerable in 0.007 433 both accuracy and 331 Grade3 As expected, fluency. 101 in both fluency 0.76 differences important 0.24 and reading0.020comprehension 433 emerge Boys 391 two groups. between these 0.90 While less-than-fluent 43 0.10 students 0.014 – i.e. those 435 reading between 0 and 44 CWCPM363 Girls - attempted to read about 0.84 in a minute and 67 17.6 words0.16 got 11.2 of431 0.018 them correct, for an oral reading fluency average of 11.2 CWCPM, fluent students attempted to read 56 words Boys | Grade 2 got 55 and 0.98 (98% accuracy), 212words correct 4 for an 0.02 0.009 average of oral reading fluency 21768.8 CWCPM Boys | Grade 3 difference – a 178 of around0.82 58 CWCPM 39 in the oral reading passage 0.18 between the 0.026 218two groups. Girls | Grade of average0.98 In2 terms211 reading comprehension, 4 less-than-fluent 0.02 students showed 216 a 0.010 fluency level Girls | Grade 3 that 151 allowed them 0.71 to receive 63 only one question, 0.29 reaching 0.031 an average 215 comprehension of 9%. Conversely, fluent students received all questions and were able to understand about 64% of the text. Table 22 Differences - Average between fluency, fluent accuracy and andfluent less than reading comprehension students levels, by condition are considerable in both of fluency and accuracy Table 22 - Average fluency, accuracy and reading comprehension levels, by condition of fluency fluency. As expected, important differences in both fluency and reading comprehension emerge between these two groups. Less than Fluent While (N=1,152) students less-than-fluent Fluent (N=130) – i.e. those reading between 0 and 44 CWCPM - attempted to Mean read about 17.6 a minute andsd sd words inmean got 11.2 of them Overall: correct, for an > 45 words oral reading fluency 0.872 average of 11.2 0.012 0.128 students CWCPM, fluent 0.012 attempted to read 55 words correct11.272 56 words and got CWCPM (98% accuracy), for an oral 11.316 25.205average of 68.8 reading fluency 68.752 CWCPM – a difference of around 58 CWCPM in the oral reading passage Words attempted 17.646 9.864 55.991 between the two 3.832 groups. In terms of average Words read (raw reading 11.237 comprehension, 11.311 less-than-fluent 54.582 students 4.261 showed a score) fluency level that allowed them to receive only one question, reaching an average comprehension of 9%. Conversely, fluent students received all questions and were able to Reading understand about 64% of the text. 0.13 0.202 0.63 0.2766 comprehension% Table 22 - Average fluency, accuracy and reading comprehension levels, by condition of fluency In summary, there is much variation inLess Fluent (N=1,152) thanfluency reading and comprehension Fluent among(N=130) Ni-Vanuatu Francophone students. Greater oral reading fluency is Mean associated with sd mean higher levels of sd reading comprehension, 52 with fluent students— 13% of the > Overall: 45 words sample of Grade 0.872 0.012 2 and 3 students able to read0.128 0.012words per minute at least 45 correct – reaching an averageCWCPM 69 CWCPM,11.316 11.272 fluency of around 68.752 with an 25.205 which is associated average understanding Words attempted 17.646 9.864 55.991 3.832 of 63% of the text they read. By contrast, less-than-fluent students – i.e. those reading between 0 and 44 Words read (raw CWCPM - read at a fluency level of CWCPM average understanding 11.237 with an 11.311 54.582 of only 13% of the text they 4.261 score) read. The results suggest that reading fluency differs across both grades and gender. In particular, the results provide evidence that schooling increases fluency levels, as students in Grade 3 demonstrated the highest Reading 0.13 0.202 0.63 0.2766 comprehension% fluency levels. Girls also exhibited greater reading fluency than boys, especially by the end of Grade 3. 48 52 5 Chapter 5: Analysis of Student Factors Associated with Better Reading Outcomes We sought to explore the association between students’ characteristics and performance using data Chapter collected 5: Analysis in the student of questionnaire. Student Students Factors were asked Associated a series of questions onwith Better such their background as their parents’ literacy, whether they attended kindergarten, and whether they had books for school and Reading Outcomes other books. Table 23 gives the characteristics of the entire sample.   Table 23 � Characteristics of students in the sample along several student and family factors     Table 23 - Characteristics of students in the sample along several student and family factors    Mean  SD  N  Age of students in the sample (in years)  8  1.7  1,291  Student speaks French at home   26%  44%  1,236  Student speaks Bislama or Vernacular at home  92%  27%  1,262  Student owns the school textbook  68%  47%  1,250  Student has a teacher that reads aloud to him/her  92%  28%  1,259  Student has reading materials at home:  40%  49%  1,212  In English  4%  20%  1,293  In French  23%  42%  1,293  In Bislama  7%  26%  1,293  In other language  1%  7%  1,293  Student has a literate family member:  88%  33%  1,254  Student has a literate mother 55%  50%  1,293  Student has a literate father  52%  50%  1,293  Student has both parents literate 43%  49%  1,293  Student has at least one literate sibling  50%  50%  1,293  Student has other literate family member 9%  28%  1,293  Student does homework and receives help from a family member:  83%  38%  1,238  From his/her mother  41%  49%  1,033  From his/her father  18%  38%  1,033  From his/her sibling  27%  44%  1,033  From another relative  7%  26%  1,033  Someone has a family member that reads with him/her at home  72%  45%  1,237  Mother  4%  20%  891  Father  36%  48%  891  Sibling  15%  36%  891  Other  42%  49%  891  Student attended kindergarten before Grade 1  90%  30%  1,259  Student was absent from school for more than 1 week  53%  50%  1,163  Student is in overage (as a proxy for repetition)  43%  50%  1,293  Note: Missing data explains cases where N is less than 1,293.  49  In  order  to  determine  which  characteristics  have  a  significant  effect  on  acquisition  of  literacy skills,  multiple  Ordinary  Least  Squares  (OLS)  regressions  were  performed  on  the  scores  of  each  In order to determine which characteristics have a significant effect on acquisition of literacy skills, multiple Ordinary Least Squares (OLS) regressions were performed on the scores of each subtest for the subset of Grades 2 and 3. Tables 24, 25, 26 and 27 give the results from each of the eight regressions. Grade progression was a significant determinant in the acquisition of literacy skills with students in Grade 3 expected to read close to 18 CWPM more than students in Grade 2 on average. The variable was significant and positive for all subtests. Gender was not by itself significant, however, there was a significant gender interaction with Grade which suggests that girls in Grade 3 are expected to read 5 CWPM more than their male counterparts. This variable was only significant for oral reading fluency. Gender also played a role with family literacy, which will be discussed below. The language spoken at home was largely insignificant except that speaking Bislama at home had a positive relationship both with oral reading fluency and reading comprehension, with students who spoke the language at home expected to read 3.7 CWPM more than those who did not and score 7% better in reading comprehension on average. Owning the textbook was not a significant determinant of reading acquisition in this model, and having a teacher who the student reported as reading aloud actually had a negative, significant association with phoneme recognition and recognizing familiar words. Further examination of classroom practices or a more complex model may be required to explain such counterintuitive findings. Having books available at home was a positive and significant factor in all subtests except intial phoneme recognition (subtest 1). Students with books available at home are expected to read 5.8 CWPM more, and score between 5.7 and 9% more on the comprehension subtests. Familial literacy had a complex relationship with the development of reading skills. Maternal literacy was insignificant in all subtests, except initial phoneme recognition alone. However, when interacted with gender, it indicates that girls benefit significantly from maternal literacy. Girls with a mother who can read are expected to read 8.9 CWPM more and score between 8.7 and 10% more on the comprehension subtests. The model suggests that paternal literacy had a significant positive effect on phoneme recognition, familiar word recognition, reading and listening comprehension and a negative effect on oral reading fluency. Interacted with gender, the model suggests that girls benefit less from having a literate father than boys only in phoneme recognition. Sibling and the literacy status of other family members were insignificant except for reading comprehension where girls benefitted from a sibling who was literate more than boys did. Students who requested the test format in the Bisclamar language tended to have significantly lower scores across a range of subtests. This may indicate a students’ recognition of their lack of facility with the French language. Likewise, a student who answered the dictation subtest partly in pictograms is expected to have a significantly lower score in reading comprehension, possibly for the same reason. Causation should not be assumed in these particular variables. 50 Table 24 Multiple Table Regression  24 Multiple Results:  Regression Part A  Results: Part A    Sub�test 1  Sub�test 2  Sub�test 3  Sub�test 4  Number of     CSPM  CFWPM  CIWPM  initial sounds  0.3767  1.1262  �0.3145  �1.6538  Gender  0.4960  0.5560  0.8000  0.1510  2.5305  12.2484  8.9018  8.3272  Grade 3  0.0000  0.0000  0.0000  0.0000  0.1097  0.0301  2.3347  1.8346  Girl in Grade 3  0.8220  0.9890  0.0920  0.1670  �0.0752  1.0163  �0.0839  0.0655  English (French) at Home  0.7860  0.3980  0.9170  0.9330  0.1620  1.5701  0.8137  0.7435  Bislama at Home  0.6970  0.2970  0.4300  0.4280  0.4328  1.8138  0.7348  0.7168  Owns Textbook  0.1390  0.1220  0.3360  0.3260  �0.0542  �3.6717  �1.9344  �0.4474  Teacher Reads Aloud  0.9180  0.0570  0.0590  0.6440  0.1986  4.3315  3.1173  2.8135  Books available at home  0.4460  0.0000  0.0000  0.0000  1.0917  0.6993  �0.8503  �0.6557  Mother Literate  0.0080  0.6850  0.4650  0.5400  �0.3460  4.8466  2.0951  1.5983  Father Literate  0.3920  0.0040  0.0680  0.1390  0.4333  0.0469  0.6041  0.3934  Sibling Literate  0.2310  0.9740  0.4940  0.6390  0.9158  3.2554  �0.5283  0.5483  Other family member literate  0.1560  0.2780  0.7380  0.7430  0.2654  4.3245  3.5975  4.3657  Interaction: Mother literate, female  0.6420  0.2780  0.0350  0.0030  0.4785  �4.1675  �1.0004  �0.6855  Interaction: Father literate, female  0.4050  0.0900  0.5480  0.6400  �0.6245  1.1781  �0.2436  0.0804  Interaction: Sibling literate, female  0.2090  0.5550  0.8570  0.9490  Interaction: Other family member literate,  �1.2667  3.4145  2.1897  2.0618  female  0.1760  0.4000  0.3180  0.3980  �0.3299  �3.0373  �1.3202  �1.2700  Test format: Bisclamar  0.2360  0.0030  0.0640  0.0540    51 56    25 OLS Table Table Multiple Regression Results: Part B  25 OLS Multiple Regression Results: Part B    Sub�test 5a  Sub�test 5b  Sub�test 6  Sub�test 7  DICT (weighted    CWCPM  RCOMP (%)  LIST COMP (%)  score)  �2.1189  �0.0425  0.0457  0.2937  Gender  0.3680  0.1960  0.2500  0.2580  17.9717  0.2290  0.1589  1.1657  Grade 3  0.0000  0.0000  0.0000  0.0000  5.0090  0.0106  �0.0461  �0.0004  Girl in Grade 3  0.0950  0.7660  0.2540  0.9990  0.1408  0.0025  �0.0128  0.1168  English (French) at Home  0.9310  0.9030  0.5820  0.4200  3.6549  0.0711  0.0156  0.0454  Bislama at Home  0.0750  0.0040  0.6780  0.8340  0.3500  0.0100  �0.0344  0.1326  Owns Textbook  0.8420  0.6060  0.1530  0.3690  �0.7292  �0.0086  0.0412  �0.1688  Teacher Reads Aloud  0.6920  0.7120  0.2950  0.4460  5.7789  0.0574  0.0890  0.3667  Books available at home  0.0000  0.0030  0.0000  0.0070  �3.4875  �0.0330  0.0301  �0.2006  Mother Literate  0.2220  0.2710  0.3890  0.3000  �3.4875  0.0650  0.0982  0.2903  Father Literate  0.0730  0.0230  0.0040  0.1410  �0.9148  �0.0250  0.0129  �0.0132  Sibling Literate  0.6510  0.2860  0.6420  0.9400  �0.7273  0.0203  0.0728  0.0116  Other family member literate  0.8290  0.7180  0.2840  0.9740  8.9044  0.1074  0.0872  0.5281  Interaction: Mother literate, female  0.0280  0.0150  0.0840  0.0770  �3.0958  �0.0398  �0.0634  �0.2695  Interaction: Father literate, female  0.4300  0.3580  0.2000  0.3740  0.9190  0.0573  �0.0290  �0.0449  Interaction: Sibling literate, female  0.7530  0.0950  0.4690  0.8560  Interaction: Other family member literate,  0.4441  �0.0354  �0.0645  �0.0190  female  0.9200  0.6130  0.4160  0.9640  �1.4186  �0.0088  �0.0025  �0.3236  Test format: Bisclamar  0.3140  0.6320  0.9120  0.0150  52 57    Table 26 Multiple Regression Results: Part C Table 26 Multiple Regression Results: Part C     Sub�test 1  Sub�test 2  Sub�test 3  Sub�test 4  Number of     CSPM  CFWPM  CIWPM  initial sounds  0.1272  0.8893  0.3362  0.0883  Does Homework  0.7630  0.5470  0.6900  0.9150  0.2822  �0.5514  0.2469  �0.4827  Reads At Home  0.3680  0.6330  0.7480  0.5020  Attended  0.4650  1.0570  0.9624  0.2795  Kindegarten  0.2380  0.5000  0.3850  0.7950  �0.6985  �3.9384  �2.1214  �1.7787  Absent >1 Week  0.0050  0.0000  0.0010  0.0050  Answered in  0.0601  �0.2874  �0.2567  �0.0365  Pictographs  0.6500  0.6550  0.6490  0.9470  2.4593  8.5804  3.4495  1.7424  Constant  0.0010  0.0010  0.0290  0.2260                 R^2  0.1954  0.2645  0.2990  0.2763  N  860.0000  860.0000  860.0000  860.0000  F statistic  9.7400  10.3900  13.8200  12.2300  P�value  0.0000  0.0000  0.0000  0.0000  Test for Normality of Residuals  Shapiro�Wilkes  10.0610  7.9890  9.1940  9.3430  P�value  0.0000  0.0000  0.0000  0.0000  Doing homework and reading at home were both insignificant across all subtests in this model. Attending Familial  literacy  had  a  complex  relationship  with  the  development  of  reading  skills.  Maternal  kindergarten was also insignificant for all subtests except listening comprehension and dictation where it had a literacy  was  insignificant  in  all  subtests,  except  initial  phoneme  recognition  alone.  However,  positive effect. Absenteeism of more than 1 week had a negative, significant effect on all subtests. when  interacted  with  gender,  it  indicates  that  girls  benefit  significantly  from  maternal  literacy.  Overall, Girls student-specific   with factors   a  mother  who such   are   can  read as  Grade, familial expected literacy   to  read   8.9 and CWPMsocio-economic factors   more  and  score such as   between the   8.7  presence of books at home had an impact on the acquisition of reading skills. and  10%  more  on  the  comprehension  subtests.  The  model  suggests  that  paternal  literacy  had  a  significant  positive  effect  on  phoneme  recognition,  familiar  word  recognition,  reading  and  listening  comprehension  and  a  negative  effect  on  oral  reading  fluency.  Interacted  with  gender,  the  model  suggests  that  girls  benefit  less  from  having  a  literate  father  than  boys  only  in  phoneme recognition. Sibling and the literacy status of other family members were insignificant  except  for  reading  comprehension  where  girls  benefitted  from  a  sibling  who  was  literate  more  than boys did.  58 53    facility with the French language. Likewise, a student who answered the dictation subtest partly  in  pictograms  is  expected  to  have  a  significantly  lower  score  in  reading  comprehension,  possibly for the same reason. Causation should not be assumed in these particular variables.  Doing  homework  and  reading  at  home  were  both  insignificant  across all subtests  in this  model.  Attending  kindergarten  was  also  insignificant  for  all  subtests  except  listening  comprehension  and  dictation  where  it  had  a  positive  effect.  Absenteeism  of  more  than  1  week  had  a  negative,  significant effect on all subtests.  Table 27 Multiple Regression Results Part D Table 27 Multiple Regression Results Part D     Sub�test 5a  Sub�test 5b  Sub�test 6  Sub�test 7  DICT (weighted     CWCPM  RCOMP (%)  LIST COMP (%)  score)  1.0022  �0.0040  0.0265  0.2046  Does Homework  0.5430  0.8560  0.3520  0.2980  0.1375  0.0216  0.0285  �0.0008  Reads At Home  0.9320  0.2770  0.2210  0.9960  Attended  1.5940  0.0047  0.0497  0.3119  Kindegarten  0.5040  0.8580  0.0680  0.0920  �4.1374  �0.0683  �0.0350  �0.2218  Absent >1 Week  0.0020  0.0000  0.0830  0.0700  Answered in  �0.1010  �0.0142  0.0085  �0.0374  Pictographs  0.9300  0.0020  0.3670  0.4850  1.5602  �0.0055  �0.0055  1.2803  Constant  0.6040  0.8830  0.9190  0.0000                 R^2  0.2669  0.2636  0.1658  0.1747  N  860.0000  860.0000  860.0000  860.0000  F statistic  12.1600  10.6100  7.9200  8.0500  P�value  0.0000  0.0000  0.0000  0.0000  Test for Normality of Residuals  Shapiro�Wilkes  5.3330  10.0460  10.0410  10.0770  P�value  0.0000  0.0000  0.0000  0.0000    Overall, student�specific factors such as Grade, familial literacy and socio�economic factors such  as the presence of books at home had an impact on the acquisition of reading skills.   59    54 6 Chapter 6: Analysis of Teacher Factors Associated with Better Reading Outcomes Chapter 6: Analysis of Teacher Factors Associated with Better Reading Outcomes We sought to explore the association between teacher characteristics –e.g., years of experience, professional We  sought  to  explore  the  association  between  teacher  characteristics  –e.g.,  years  of  experience,  qualification, in-service training, etc. and student reading outcomes using data collected in the teacher professional  qualification,  in�service  training,  etc.  and  student  reading  outcomes  using  data  questionnaire. Table collected  in  the 28 below   teacher summarizes   questionnaire. the profile   Table of teachers   28  below in the  the   summarizes VANEGRA French   profile  sample   in from of  teachers the  Grades 1-3. VANEGRA French sample from Grades 1�3.    Table 28 – Profile of Francophone Teachers in VANEGRA Table 28 – Profile of Francophone Teachers in VANEGRA   Teacher factor  Mean  SD  N  Age (in years)  37  10  105  Percentage  of  teachers  that  hold  a  Certificate  of  Primary  Teaching  95%  21%  106  (0=no; 1=yes)  Teachers by type of professional qualification:           None 2%  14%  108  Certificate of Primary Education 87%  34%  108  Certificate of Education 7%  26%  108  Other 0%  0%  108  Years of experience (in total)  16  12  107  Years of experience (as certified teacher)  12  10  91  26%  44%  102  Percentage that received in�service training last year (0=no; 1=yes)  Percentage  that  received  in�service  reading  training  in  last  2  years  25%  44%  102  (0=no; 1=yes)  Average total number of days spent in training   6  5  24  52%  50%  107  Percentage of teachers that work in a school with library (0=no; 1=yes)  How often does the teacher use it?           Rarely 19%  40%  108  Half the time 8%  28%  108  Most of the time 8%  28%  108  Every lesson 4%  19%  108  Percentage of teachers that supervise students using the library (0=no;  91%  29%  43  1=yes)  78%  42%  105  Percentage of teachers that have a corner library (0=no; 1=yes)  Percentage  of  teachers  that  work  in  a  school  with  a  functioning  PTA  53%  50%  86  (0=no; 1=yes)  Percentage  of teachers  that meet  with  the parents  of  his/her  students  69%  47%  106  (0=no; 1=yes)  Once per term or less 58%  50%  108  Twice 6%  25%  108  Three times 2%  14%  108  Percentage  of  teachers  that  have  the  recommended  Reading  Text  51%  50%  102  (0=no; 1=yes)  How often do you use it?           Rarely 4%  19%  108  Half the time 7%  26%  108  55 60    Teacher factor  Mean  SD  N  Most of the time 19%  39%  108  Every lesson 16%  37%  108  How useful do you find it?           Not very useful 2%  14%  108  Moderately useful 12%  33%  108  Very useful 31%  46%  108  Percentage  of  teachers  that  have  the  teacher  guide  for  reading  57%  50%  105  instruction (0=no; 1=yes)  How useful do you find it?          Not very useful 4%  19%  108  Moderately useful 17%  37%  108  Very useful 35%  48%  108      Most of the teachers in the sample are certified, experienced teachers with an average of 16 years teaching Most   of  the experience,   teachers and 12 years  in ofthe   sample  are experience as  a certified, experienced certified   teacher. 26%  teachers of teachers   with in   an sample the average   of  16  reported years   teaching participating last  experience, year in some   and   12 of form years   of  experience in-service as  a 25% training,  with certified   teacher. of the teachers   26%   of  teachers reporting they had   in  the   sample in participated reported in-service  participating   last  year training for reading in the  inlast   some   form The 2 years. of  in �service average   training, number   with of days   25% the  training for of  the  teachers   reporting   they   had   participated   in   in �service   training   workshops was 6 days. 52% reported having a school library in their schools and 80% of those reported for   reading   in   the   last   2   years.    The  average  number  of  days  for  the  training  workshops  was  6  days.    52%  reported  having  a  using it; less than 10% reported using it half or most of the time. 78% of teachers reported having a learning school library in their schools and 80% of those reported using it; less than 10% reported using it  corner. 53% of the teachers surveyed reported having a PTA, with 69% having parent meetings, most of half or most of the time.  78% of teachers reported having a learning corner.  53% of the teachers  these (58%) at least once per term. 51% of teachers reported having the recommended reading texts but surveyed reported having a PTA, with 69% having parent meetings, most of these (58%) at least  only 19% reported using them most of the time. once   per  term.   51%  of  teachers  reported  having  the  recommended  reading  texts  but  only  19%  reported using them most of the time.  Effect of Teacher Characteristics on Student Performance   In order to explore which teacher characteristics and teaching activities impact on student performance, Effect of Teacher Characteristics on Student Performance we performed individual z-tests on subtest results for Grades 2 and 3. These tests require several stringent   assumptions, but can determine the statistical significance of characteristics. The commonly used t-tests In   order  to  explore  which  teacher  characteristics  and  teaching  activities  impact  on  student  were also performed, performance, but for reasons   we  performed of brevity   individual are  not   z�tests reported on  subtest here. They   results are reported   for  Grades   2  and in  3. Annex 1. 16   These The tests   require  analysis following several concentrates on comparisons stringent  assumptions,   butbetween   can  groups determine for the   the subtests on oral   statistical reading fluency,   significance   of  characteristics.and comprehension   The   commonly dictation. Where   used   t�tests activity, a teaching were  also   performed, meeting or evaluation  but  occurs for  reasons   of  brevity on a regular basis,   are the  not most two reported   here. infrequent   They are groups are in  Annex   reported generally compared:   1.   and “never�16   The   following “1-2 times per   analysis week� for   concentrates teaching activities  on  comparisons and “never� and   between “once per   groups term� for   for   the  subtests meetings   on  oral Exceptions and evaluations. reading  fluency,   comprehension will be noted.   and  to this rule dictation.  Where  a  teaching  activity,  meeting  or  evaluation  occurs  on  a  regular  basis,  the  two  Table 29 gives a  summary most infrequent groups are of   compared: full results these tests, generally are available  “never�  and “1 in �Annex 2 times 1. Overall,  per  week� compared to no  for teaching   certification, holding a Certificate in Primary Teaching as a highest activities and “never� and “once per term� for meetings and evaluations. Exceptions to this rule  qualification had a mixed effect. Insignificant will be noted. for  oral reading fluency and reading comprehension, it had a significant positive effect on listening comprehension scores, but a negative and significant effect on dictation scores. Inservice and training in                                                        16 16The commonly-used t-test The commonly-used requiresfewer t-test requires fewer assumptions assumptions than than the the z-test z-test and and isis less less likely likely to to return return a result a result indicating indicating that that the variable is significant. They are a more conservative test in the variable is significant. They are a more conservative test in this case.this case. 56 61    teaching reading both had negative effects, significantly so in most subtests. Students whose teacher had attended an inservice read at a rate of 5 CWPM less and comprehended between 3-5% less than those whose teachers had not. Experience had a positive and significant effect. Comparing the students of those teachers with 0-4 years experience and those with 5-10 years experience, the more experienced teachers had students who read 11.5 CWPM and comprehended between 11 and 12% more. The effect of further experience was not linear and may be affected by small sample sizes. The presence of a school library, teachers who supervised their students in the library and the presence of a classroom reading corner were all strongly and positively significant. Students at schools with libraries read almost 8CWPM more than those who did not. Students whose teachers supervised them in the library read an astonishing 20.2 CWPM more than those who did not. Students in a classroom with a reading corner read almost 5 CWPM more than those who did not. These results suggest that these resources can assist ni-Vanuatu students in improving their reading skills and that access and appropriate supervision of their use increases their value. The presence of a functioning Parents and Teachers’ Association (PTA) was largely insignificant, but did have a small positive effect on listening comprehension. Teachers who met with parents had a largely insignificant effect, but did have a small positive and significant relationship with listening comprehension. However, comparing groups of students whose teachers met with parents once per term and those who met with parents twice, there was a significant negative relationship between meeting more often and with listening comprehension and dictation. Causality should not be assumed in this case, as it may be a result of poor student performance rather than a cause of it. Schools which own the recommended reading have a strong, significant and positive relationship with better literacy acquisition. Students in these schools read on average 9.7 CWPM more and comprehended 7% more of what they read. A teacher who owned the curriculum guide had a positive, significant relationship with reading comprehension, listening comprehension and dictation. Students with a teacher who owned the guide scored between 3.5 and 6% more on the comprehension subtests. The EGRA teacher survey also enquired into 7 teaching activities and the frequency with which they were undertaken. The following analysis (except where specified) compares students who undertook the activity 1-2 times a week and those who were reported by teachers as never undertaking the activity. Students who practiced identifying the sounds in letters 1-2 times a week had significantly better scores in all subtests except listening comprehension. On average, they read 18 CWPM more and comprehended around 14% more. Repetition of words or sentences after a teacher had a significant effect on listening comprehension and dictation scores, however, these effects were mixed. Students who performed the activity 1-2 times a week scored 15% lower on listening comprehension, but had average scores 0.06 higher on dictation. 57 Table 29 Teacher and School-Specific Factors Associated with Reading Acquisition 58 Table 29 Teacher and School�Specific Factors Associated with Reading Acquisition  ORF  RCOMP%  LCOMP%  DICT%     z�statistic  p�value  z�statistic  p�value  z�statistic  p�value  z�statistic  p�value  Do you hold a Certificate in Primary Teaching?  �0.514  0.304  0.800  0.212  1.775  0.038  �1.670  0.047  Highest Qualification  �0.538  0.295  0.710  0.239  1.732  0.042  �1.673  0.047  Have you attended an in�service?  �2.572  0.005  �1.366  0.086  �2.040  0.021  �4.572  0.000  Have you attended a reading training?  �2.300  0.011  �0.422  0.336  �1.258  0.104  �4.661  0.000  Experience Level  4.060  0.000  4.387  0.000  3.316  0.000  6.084  0.000  Library Present  4.611  0.000  3.488  0.000  1.955  0.025  4.023  0.000  Supervision in Library  8.329  0.000  3.607  0.000  2.090  0.018  4.016  0.000  Reading Corner Present  2.390  0.008  4.030  0.000  4.782  0.000  1.006  0.157  PTA Functioning  �1.115  0.132  �1.191  0.117  3.247  0.001  �0.126  0.450  Meeting with Parents  �0.267  0.395  �0.677  0.249  1.489  0.068  0.705  0.241  How Often do Teachers Meet with Parents  �0.343  0.366  0.766  0.222  �3.518  0.000  �2.088  0.018  Teacher Owns Teachers' Guide for Reading  0.921  0.179  1.865  0.031  2.606  0.005  1.840  0.033  School has Recommended Reading  5.327  0.000  3.346  0.000  0.082  0.467  3.170  0.001  Students Practised Identifying the Sounds in Letters  6.932  0.000  3.336  0.000  0.302  0.381  4.260  0.000  The whole class repeated words or sentences that said first  0.199  0.421  �0.294  0.384  �2.181  0.015  1.403  0.080  Students copied down text from the chalkboard  1.597  0.055  1.609  0.054  �1.471  0.071  0.460  0.323  Students retold a story that they had read  �3.730  0.000  �4.039  0.000  �1.092  0.137  �3.052  0.001  Students sounded out unfamiliar words  �2.137  0.016  �1.365  0.086  �1.855  0.032  �1.357  0.087  Students learned meanings of new words  0.104  0.459  2.029  0.021  2.292  0.011  �0.798  0.213  Students Read Aloud  �2.837  0.002  �0.861  0.195  �0.550  0.291  �2.638  0.004  Students assigned reading on their own  �0.884  0.188  �1.355  0.088  �1.914  0.028  �0.273  0.392  Written Evaluations  4.002  0.000  2.503  0.006  �0.017  0.493  1.178  0.119  Oral Evaluations  2.088  0.018  �0.750  0.226  1.788  0.037  �0.609  0.271  Review of Portfolios and other projects  1.306  0.096  0.706  0.240  0.706  0.240  0.699  0.242  Student Reads aloud from chalkboard  �0.114  0.455  �0.843  0.200  0.238  0.406  �0.441  0.330  Review of homework  0.292  0.385  �0.233  0.408  �2.332  0.010  0.451  0.326  Observation and/or competency checklists  1.977  0.024  1.450  0.074  1.242  0.107  �0.929  0.176  63    Copying text from the chalkboard had a positive, significant relationship with all subtests except dictation when practicing the activity 1-2 times a week is compared to never practicing the activity. However, the effect decreases when the activity occurs more often. Students retelling a story they had heard had a negative, significant association with all subtests except listening comprehension. Students practicing this activity 1-2 times a week read more than 10 CWPM less than those who did not practice the activity. The negative effect increases the more the activity is practiced. Students sounding out unfamiliar words had a negative association with all subtests with students practicing the activity reading 11.8 CWPM less than those who did not. Learning the meanings of new words had positive, significant relationships with both types of comprehension tested, with students having between 5.5-7% higher comprehension scores. Reading aloud had a negative effect on students’ acquisition of oral reading fluency and dictation scores, while assigned reading had a negative association with both types of comprehension. It may be that the students had not yet obtained a degree of fluency high enough to benefit from the activity in the large group or classroom environment. The survey also asked about the practice and frequency of three types of evaluations: written, oral and review of projects. Written evaluations carried out once per term had a significant, positive association with oral reading fluency and reading comprehension when compared to those students who never received such evaluations. This effect increased the more regularly the evaluations were carried out. Oral evaluations were positively associated with reading fluency and listening comprehension, but the effect was not uniformly linear as the frequency increased. Review of portfolios and projects had a positive, significant association with reading fluency that increased as the frequency of the evaluation was carried out. Students who read from the chalkboard on 1-2 days had no significant difference with those who never did so. However, this effect increases rapidly when the activity is performed 3 or more times per week. Review of homework had no significant relationship with any of the subtests except listening comprehension where it had a mild negative association. Observation and/or competency checklists had a positive, significant relationship with oral reading fluency and reading comprehension when performed once or twice a month compared to those teachers who never performed them. Overall, resources such as libraries and reading corners and adequate supervision in them had strong positive relationships with students’ acquisition of basic literacy skills. Although there is an admirable culture of teacher training is being developed in Vanuatu, assistance and improvements in the in-service and training courses provided to teachers will assist in improving student results. 59 Effect of Teacher Expectations on Student Performance Finally, we explored the relationship between teacher’s expectations on reading outcomes and student performance in sub-test 5a (oral reading passage). Table 30 below shows the distribution of teacher expectations about reading outcomes, including the median. For example, although most teachers in Francophone schools in Vanuatu expected students to read aloud a short passage with a few mistake at the end of Grade 3, the median expectation for this outcome falls at the end of Grade 2. On average, teachers expect students to read aloud a short passage in Grade 2, write their names in Grade 1, understand stories they read in Grade 1, recognize the sounds of letters and graphemes in Grade 2, sound out unfamiliar words in Grade 2, and recite the alphabet in Grade 1. We measure teachers’ expectations and fluency by measuring differences in student performance when teachers’ expectations deviate from the median. We regressed student performance against teachers’ expectations, excluding the median expectation. This equation applies to all tasks where the median expectation is Grade 2: result = a+ß1 expect_in_kindergarten + ß2 expect_in_grade1 + ß3 expect_in_grade3 Where result is the subtest under consideration (reading comprehension), expect_in_kindergarten is an indicator for teachers expecting students to perform a given task in kindergarten, similarly expect_in_grade1 and expect_in_grade3. The coefficients ß1 - ß2 give us an average difference in performance between the excluded group of students (students whose teachers have the median expectation) and students whose teachers have different expectations. For example, regressing reading comprehension on teacher expectations on reading aloud a short passage, the coefficient ß2 gives us the difference in average reading comprehension between students whose teachers expect them to read aloud in Grade 1 and students whose teachers expect them to read aloud in Grade 2 (the median expectation). Overall, if a teacher has more “realistic� expectations –i.e. consistent with the median expectation-, student performance is better than those whose teachers fall farthest from the median expectation, either “earlier� if the median expectation is Grade 2 or 3, “later� if the median expectation is either Grade 1 or before, or considers the outcome to be unimportant for reading development in Vanuatu (Tables 30 and 31). Compared to students whose teachers expectations fall on the median expectation for each skill tested: • Students whose teachers expect them to recite the alphabet at the end of Grade 2, read 13 CWCPM more and understood almost 16% more of the text they read; students whose teachers expected them to recite the alphabet at the end of Grade 3 also showed similar improvements in oral reading fluency (12 CWCPM more) and reading comprehension (16% more). Interestingly, students whose teachers considered this skill to be unimportant, also showed better fluency in reading (6 CWCPM more) and reading comprehension (8% more). 60   Table 30 – Teachers median expectations about reading outcomes Table 30 – Teachers median expectations about reading outcomes     Teacher responses  Reading outcome  Grade   Mean   SD  N  Kindergarten  6%  24%  98  Grade 1  39%  49%  98  Read aloud a short passage with  Grade 2  26%  44%  98  few mistakes  Grade 3  23%  43%  98  Not important  0%  0%  98  Kindergarten  30%  46%  98  Grade 1  55%  50%  98  Write name  Grade 2  10%  30%  98  Grade 3  3%  17%  98  Not important  0%  0%  98  Kindergarten  4%  20%  98  Grade 1  32%  47%  98  Understand stories they read  Grade 2  38%  49%  98  Grade 3  24%  43%  98  Not important  0%  0%  98  Kindergarten  15%  36%  98  Grade 1  60%  49%  98  Recognize the sounds of  Grade 2  14%  35%  98  letters/graphemes  Grade 3  9%  29%  98  Not important  0%  0%  98  Kindergarten  1%  10%  98  Grade 1  32%  47%  98  Sound out invented words  Grade 2  35%  48%  98  Grade 3  30%  46%  98  Not important  1%  10%  98  Kindergarten  13%  34%  98  Grade 1  45%  50%  98  Understand stories they hear  Grade 2  18%  39%  98  Grade 3  20%  41%  98  Not important  2%  14%  98  Kindergarten  36%  48%  98  Grade 1  46%  50%  98  Recite alphabet  Grade 2  9%  29%  98  Grade 3  7%  26%  98  Not important  0%  0%  98    Overall,  if  a  teacher  has  more  “realistic�  expectations  –i.e.  consistent  with  the  median  • Students whose teachers expect them to write their name either at kindergarten or at the end of expectation�,  student  performance  is  better  than  those  whose  teachers  fall  farthest  from  the  Grade 2, had better oral reading fluency and comprehension but these differences are not median  expectation,  either  “earlier�  if  the  median  expectation  is  Grade  2  or  3,  “later�  if  the  statistically significant. Interestingly, teachers that allowed their students to write their name “as median  expectation  is  either  Grade  1  or  before,  or  considers  the  outcome  to  be  unimportant  for  readinglate� as at the end   development   inof Grade 3,  read   Vanuatu   30 20 almost (Tables andCWCPM more and     31).  Compared understood 26% to  students more  of   whose the text teachers   they read. expectations  fall on the median expectation for each skill tested:  66    61 • Students whose teachers expect them to understand stories they hear at kindergarten or at the end of Grade 2, also had better oral reading fluency and comprehension but these differences are not statistically significant. These differences were positive and statistically significant in terms of reading fluency (18 CWCPM more) and reading comprehension (17%) if teachers expected students to consolidate this skill at the end of Grade 3. • Students whose teachers expect them to sound out unfamiliar words in Grade 1 or that consider this skill unimportant had lower reading fluency (-6.7 CWCPM and -11 CWCPM, respectively) and comprehended less of the text they read (-7% and -12%, respectively). • Students whose teachers expect them to sound out letters/graphemes before Grade 2 or that considered this to be unimportant is associated with lower reading fluency and comprehension but these effects are not statistically significant. Interestingly, students whose teachers allowed for this skill to consolidate at the end of Grade 3, read 11 CWCPM more and understood 16% more of the text they read. • Students whose teachers expect them to read a short passage with a few mistakes in kindergarten, Grade 1 or that consider this skill to be unimportant, are associated to lower average reading fluency and comprehension. These effects are statistically significant only for those whose teachers consider the skill to be unimportant (11 CWCPM less and 12% fewer comprehension). • Student whose teachers expect them to understand stories they read by Grade 1 or consider this to be unimportant associate to fewer reading fluency and comprehension at statistically significant levels. Students whose teachers allow for this skill to consolidate at the end of Grade 3 had better reading fluency (12 CWCPM more) but improvements in reading comprehension did not show statistical significance. In conclusion, results of the analysis of teacher factors identified a set of teacher characteristics, instructional and assessment methods, and teacher expectations that have an effect on the reading performance of Francophone students in Vanuatu. In terms of teacher characteristics, teacher certification had an average negative effect associated with teachers with a Certificate in Primary. Teacher experience had a significant effect across all subtests. Attendance at in-service training showed negative effects possibly due to the number of courses provided or the scope of content related to other subject areas. Frequent meetings with parents showed negative and statistically significant relationships on student fluency in reading and comprehension. In terms of teacher instruction, students that never practiced letter/grapheme sound recognition or the meaning of new words were associated with lower performance in reading fluency and comprehension. On evaluation methods, the positive and statistical effects associated with oral reviews and the frequent review of projects and homework underscore the importance of frequent feedback and performance monitoring 62   Table 31 - Regression analyses of average effects of teachers’ expectations on fluency in reading Table 31 � Regression analyses of average effects of teachersʹ expectations on fluency in reading  Dependent variable Correct words read per minute Dropped category: median expectation per skill (CWCPM) Independent variables Coeff F R2 N Read aloud a short passage with few mistakes Kindergarten -0.33 . 0.03 1067 Grade 1 -1.70 Grade 3 6.56 Not important -11.03** Write name Kindergarten 1.47 5.43 0.06 1147 Grade 2 4.47 Grade 3 19.69*** Understand stories they read Kindergarten 2.53 . 0.10 1132 Grade 1 -6.191* Grade 3 12.28** Not important -6.733** Recognize the sounds of letters Kindergarten -4.59 . 0.03 1141 Grade 1 -2.40 Grade 3 11.41** Not important -6.772* Sound out unfamiliar words Kindergarten 5.16 10.99 0.06 1126 Grade 1 -6.735* Grade 3 8.42 Not important -11.05** Understand stories they hear Kindergarten 0.38 11.43 0.11 1133 Grade 2 1.99 Grade 3 18.29*** Recite alphabet Kindergarten 4.20 4.44 0.06 1147 Grade 2 13.31** Grade 3 12.00** Not important 6.093**   Significant levels:  * p<0.05   ** p<0.01   *** p<0.001  68    63   Table 32 - Regression analyses of average effects of teachers’ expectations on reading comprehension Table 32 � Regression analyses of average effects of teachersʹ expectations on reading comprehension  Dependent variable Dropped category: median expectation per skill % Reading Comprehension Independent variables Coeff F R2 N Read aloud a short passage with few mistakes Kindergarten 1.33 . 0.03 1069 Grade 1 -1.90 Grade 3 8.43 Not important -11.78** Write name Kindergarten 2.75 13.29 0.08 1149 Grade 2 6.54 Grade 3 25.82*** Understand stories they read Kindergarten 0.12 . 0.06 1134 Grade 1 -8.068** Grade 3 8.52 Not important -8.004** Recognize the sounds of letters Kindergarten -4.33 . 0.05 1143 Grade 1 -3.90 Grade 3 16.17*** Not important -6.769* Sound out unfamiliar words Kindergarten 3.93 9.00 0.06 1128 Grade 1 -6.906* Grade 3 11.83* Not important -11.83** Understand stories they hear Kindergarten 0.88 6.47 0.07 1135 Grade 2 2.52 Grade 3 17.39*** Recite alphabet Kindergarten 6.25 7.52 0.07 1149 Grade 2 15.74*** Grade 3 16.37** Not important 8.393*** Significant levels:  * p<0.05   ** p<0.01   *** p<0.001 69    64 on student learning outcomes. The positive and statistically significant effect of reading from the chalkboard could be related to teachers’ use of blackboard as the main source of print to students and the immediate feedback available to the student. Finally, in terms of teacher expectations, teachers who perceived some skills were best to be developed at a later (or earlier) grade compare to the common expectation of their peers, seem to have a positive effect on the reading outcomes of their students. As expected, teachers that considered key skills in reading development to be unimportant –e.g., understand stories they read- had negative effects on reading fluency and comprehension. Interestingly, when teachers used their professional judgment and allowed for skills to be developed at a later grade, students showed positive and statistical significant effects. This suggests teachers may perceive curricular outcomes to be “too high� for the average student in their classroom and as such, allow students to consolidate skills in later grades. 65 7 Chapter 7 - Next Steps VANEGRA French results call for immediate attention to the way reading development is taking place in the country. As a diagnostic study, its main purpose is to (a) generate data on the extent reading performance in the first cycle of primary education, (b) identify specific skills that could be hampering reading comprehension among Francophone Ni-Vanuatu students, and (c) identify student and teacher characteristics and behaviors that contribute to better reading outcomes in Vanuatu. VANEGRA results indicate that while most students are able to develop some of the basic skills needed to become effective readers, poor knowledge of the alphabetic principle hinders the development of word-level reading, which will slow down fluency development and limit comprehension. Based on the data presented, specific recommendations to be considered are presented to improve the quality of reading instruction in Francophone schools in Vanuatu: • Improve the focus and structure of reading instruction to promote greater fluency in reading by the end of Grade 3. Research has shown that developing fluency in reading is crucial to help students become effective readers in the first years of primary education. As students approach reading at a speed of about 45-60 words per minute, the reader becomes better able to focus on the meaning of the text rather than on individual letters and words. As shown by VANEGRA French results, students reading at least 45 correct words per minute were able to understand about 83% of the text they read. However, less than 1 in 10 Francophone students is able to reach this fluency level at the end of Grade 3. Two factors could contribute to explain these results. On the one hand, poor decoding skills suggest and instruction falls short to develop a solid foundational knowledge of the alphabetic principle. On the other, poor language skills on students entering the Francophone stream may contribute to delay the development of pre-reading skills in Grades 1 and 2 as students struggle to develop language and reading skills simultaneously. As the MoE sets forth to implement the new K-12 curricula, it is fundamental that instructional improvements in the early grades take into account the linguistic diversity of the country and provide adequate strategies to prepare students for reading development in a secondary language (L2). In addition, the new curriculum opens up the opportunity to improve instruction of letter and word-level reading skills to promote a better sequenced instruction of basic reading skills. • Ensure teachers working in the early grades have the knowledge to improve their practice to impact the reading outcomes of their students. In order to improve reading instruction in Vanuatu, teachers will have to improve their knowledge of reading instruction to improve classroom practice. Though most Francophone teachers in VANEGRA use their professional judgment to adjust expectations about reading outcomes, a number of them still consider some of these skills to be unimportant in reading development. Also, while some instructional activities rendered expected outcomes, the fact that other activities typically associated with better reading outcomes showed no statistical significance in Francophone schools suggests the need to review how these activities are carried out in the classroom to better understand the possible factors that are hindering their effectiveness. This is also true of the average effects of teacher methods observed on student outcomes. 66 • Support the reading instruction skills of as many teachers working with beginning readers as possible. Data from the teacher questionnaire showed that only 26% of the teachers in the sample participated in general in-service training courses and only 25% had attended in-service training on reading in the last two years. If only 1 in 4 teachers in the country benefit from learning about specific ways in which they can improve their practice, Ni-Vanuatu teachers will continue to practice their profession in isolation. The role of the newly created In-Service Unit (ISU) at the MoE will be critical to further develop teacher knowledge and practice for reading instruction. As such, it is recommended that VANEGRA findings inform the development of the lesson plans and materials and that ISU staff works in close collaboration with curriculum developers to ensure teachers understand the new curriculum goals and receive support on how to achieve them. • Establish reference reading standards to monitor reading development in the early grades. As the MoE moves on to establish an oral reading fluency standard under VERM, it is important to consider that these indicators should be considered reference standards and not high-stakes benchmarks that would jeopardize additional funding or the promotion of teachers. Since these reference standards are drawn from baseline data, additional measures will be needed in subsequent years to learn about the rate at which Francophone students develop reading abilities. In this sense, reading standards should not be seen as high-stakes but an essential piece to monitor reading progression in the classroom. In order to set up national reference standards to monitor system-level quality improvements, it would be best to use the percentage of zero-score students in selected sub-tests as a marker and track reductions in the shares at least biannually. Monitoring achievements over time will eventually provide more information on the rate and the way in which average fluency develops among Francophone students. A modified version of the test could be used to screen students during the school year that may be in need of additional support. • Help teachers translate national reference standards into easy-to-assess, easy-to-monitor reading goals to monitor the reading progression of their students during the school year. In order for teachers and schools to be able to be held accountable for reading outcomes, teachers, school officials and parents need to understand what these standards mean and how each can support reading development in their own school. School development plans should contain reading improvement goals as part of their minimum service standards, as well as a description of activities aimed at encouraging reading. Parents and the community as a whole should be brought into this effort. • Introduce policy actions that increase student exposure to literacy outside the school. VANEGRA French results showed how students who have reading books at home have better reading outcomes and are more likely to become fluent readers. The effect was positive for both boys and girls and for all basic reading skills. Thus, it is advisable that the MoE promotes increased student access to books at home. However, making more books available to students will not per se ensure better reading outcomes. Along with access to more reading materials, Ni-Vanuatu children will need support to develop a reading habit beyond the requirements of the school curriculum. One way of achieving this would be ensure the books being procured by the ongoing Book Flood program are 67 not only grade-appropriate but that they are accessible to students in and outside of the classroom. Since 52% of the teachers reported having access to a school library, an adequate book-borrowing scheme carries the potential to expose students to print on a more regular basis. Another way of increasing exposure to literacy would be to develop community literacy programs where schools become a focal point of literacy in the community. Teachers and community leaders can start up reading clubs and reading competitions to further promote a reading culture among beginning and more experienced readers. • Promote strategies to assure greater parental and community involvement in the reading development process of children. Research shows that the earlier the parental involvement, the more powerful and long-lasting the effects will be both in terms of academic and behavioral outcomes of children. Moreover, research also shows that the most effective form of parental involvement includes those where parents participated in learning activities at home. However, in order for parental involvement to be more effective, parents need not only to be informed about the academic progress of their children but also about ways in which teachers and the school planned to improve outcomes. If parents and schools communicate regularly on the academic progress of children, parents tend to monitor school and classroom activities, and coordinate efforts with teachers such as helping with homework and carrying out extracurricular activities. For this to happen, it is important that parents and schools commit jointly to the reading development process of children. In addition to providing information on the academic progress of children, schools can advise on –and even facilitate- different ways in which parents can promote reading at home. If reading outcomes are to improve in the country, reading development must be seen as a joint enterprise that extends beyond the teacher and the school classroom environment. • Finally, it is clear that more research is needed to better understand the factors that contribute to differences in reading performance between boys and girls. An analysis of the factors that contribute to these differences is beyond the scope of this survey. However, VANEGRA French data showed that boys and girls finish Grade 1 at similar levels of performance in reading, but in Grade 2 girls transition into word-level reading faster than boys and the difference increases at the end of Grade 3. A better understanding of this phenomenon is critical to inform sector policies and increase the success of future reading development programs. 68 Bibliographical References Abadzi, H. 2006. Efficient learning for the poor: Insights from the Frontier of Cognitive Neuroscience Washington, DC: The World Bank. Chabbott, C. (2006). Accelerating early grades reading in high priority EFA Countries: A desk review. From http://www.equip123.net/docs/E1-EGRinEFACountriesDeskStudy.pdf (Accessed on October 29, 2010) Chiappe, P., L. Siegel, and L. Wade-Woolley. 2002. Linguistic diversity and the development of reading skills: A longitudinal study. Scientific Studies of Reading 6(4): 369–400. Ehri, L. C., Nunes, S. R., Stahl, S. A., & Willows, D. M. (2001). Systematic phonics instruction helps students learn to read: Evidence from the National Reading Panel’s meta- analysis. Review of Educational Research, 71(3), 393-447. Fuchs, L., D. Fuchs, M.K. Hosp, and J. Jenkins. 2001. Oral Reading Fluency as an Indicator of Reading Competence: A Theoretical, Empirical, and Historical Analysis. Scientific Studies of Reading 5(3), 239–256. Hasbrouck, J., & Tindal, G. A. (2006). Oral reading fluency norms: A valuable assessment tool for reading teachers. International Reading Association, 636–644. Jiménez, J. E. and I. O’Shanahan Juan. 2008. Enseñanza de la lectura: de la teoría y la investigación a la práctica educativa. Revista Iberoamericana de Educació, 45(5): 1–22. Available at http://www.rieoei. org/2362.htm (accessed October 28, 2010). Linan-Thompson, S., and S. Vaughn. 2007. Research based methods of reading instruction for French language learners: Grades K-4. Alexandria, VA:Association for Supervision and Curriculum Development. National Reading Panel. 2000. Teaching Children to Read: An Evidence-Based Assessment of the Scientific Research Literature on Reading and Its Implications for Reading Instruction. Washington, DC: National Institute of Child Health and Human Development. Available at http://www.nichd.nih.gov/publications/nrp/ report.cfm (accessed July 13, 2010). RTI. 2009. Early Grade Reading Assessment Toolkit. Prepared by RTI for The World Bank, Office of Human Development. Available at https://www.eddataglobal.org/documents/index.cfm?fuseaction=pubDetail& ID=149 (accessed September 03, 2010). RTI. 2010. Guyana Early Grade Reading Assessment: October 2008 Results. Prepared by RTI for The World Bank, Office of Human Development. Available at https://www.eddataglobal.org/ (accessed August 02, 2010). Sprenger-Charolles, L. 2004. Linguistic Processes in Reading and Spelling: The Case of Alphabetic Writing Systems: French, French, German and Spanish. Pp. 43–66 in Handbook of Children’s Literacy. Edited by T. Nunes and P. Bryant. Dordrecht, the Netherlands: Kluwer Academic Publishers 69 Share, D. L., Jorm, A., Maclearn, R., & Matthews, R. 1984. Sources of individual differences in reading acquisition. Journal of Education Psychology, 76, 1309-1324. Snow, C. E., Burns, M. S., & Griffin, P. (Eds.). 1998. Preventing reading difficulties in young children. Washington, DC: Committee on Preventing of Reading Difficulties in Young Children and National Academy Press. Stanovich, K. E. (2000). Progress in understanding reading: Scientific foundations and new frontiers. New York: Guilford Press. Stores, Christine. 2008. “Language, Literacy and Bilingualism. A Report Presented to MEWAC to support development of the National Language Policy� Vanuatu Education Support Program – Curriculum Reform Project. (Mimeo). Vanuatu Institute of Education (VITE). Diploma of Primary Education Teaching. Curriculum Document. (Mimeo) 70 ANNEX 1/ TABLES Table 33 -VANEGRA French Reliability Matrix, Grades 2 and 3 ANNEX 1/ TABLES Table 33 �VANEGRA French Reliability Matrix, Grades 2 and 3  Correct  Correct  Correct Non  Phonemic  Letters  Oral Reading  Reading  Listening     Words Per  Words Per  Writing  Awareness  Sounds Per  Fluency  Comprehension  Comprehension  Minute  Minute  Minute  Phonemic  1 Awareness  Correct Letters  Sounds Per  0.5259 1 Minute  Correct Words  0.4607 0.7926 1 Per Minute  Correct Non  Words Per  0.4576 0.7944 0.9025 1 Minute  Oral Reading  0.3854 0.7302 0.8737 0.864 1 Fluency  Reading  0.3831 0.6186 0.7094 0.6573 0.7411 1 Comprehension  Listening  0.3402 0.347 0.3292 0.3196 0.2935 0.4288 1 Comprehension  Writing  0.4106 0.6038 0.7 0.623 0.6424 0.5734 0.3303 1   71 77    Table 34 Descriptive Statistics: Grades 2 and 3 72   Table 34 Descriptive Statistics: Grades 2 and 3  Correct  Correct  Correct Non  Phonemic  Letters  Oral Reading  Reading  Listening  Words Per  Words Per  Writing  Awareness  Sounds Per  Fluency  Comprehension  Comprehension  Minute  Minute     Minute  Mean  0.54 19.09 10.91 8.35 18.58 0.20 0.29 0.33 SD  0.37 15.86 11.39 10.88 23.64 0.28 0.29 0.23 Mean/SD  1.47 1.20 0.96 0.77 0.79 0.72 1.00 1.40 Ratio  Min  0 0 0 0 0 0 0 0 Max  1 88 57.6923 76.9231 201.1765 1 1 1 Skewness  -0.39 1.31 1.48 1.71 2.35 1.35 0.74 0.53 Kurtosis  1.63 4.72 4.79 6.40 10.98 3.85 2.57 2.68 Sample Size  866 866 865 865 865 866 866 866     78    Table 35- Cronbach’s Alpha, Grade 1 Subsample   Table 35� Cronbachʹs Alpha, Grade 1 Subsample  average  item�test  item�test  Item  Obs  Sign  inter item  alpha  correlation  correlation  covariance  Phonemic Awareness   430  +  0.46  0.27  0.29  0.74  Correct  Graphemes  Per  430  +  0.66  0.51  0.24  0.70  Minute  Correct Words Per Minute  428  +  0.71  0.58  0.23  0.68  Correct  Non  Words  Per  428  +  0.60  0.43  0.26  0.71  Minute  Oral Reading Fluency  427  +  0.71  0.58  0.23  0.68  Reading Comprehension  430  +  0.59  0.43  0.26  0.71  Listening Comprehension   430  +  0.45  0.26  0.29  0.74  Writing   430  +  0.55  0.38  0.27  0.72    73 79  Table 36- Cronbach’s Alpha, Grade 2 Subsample 74   Table 36� Cronbachʹs Alpha, Grade 2 Subsample  average  item�test  item�test  Item  Obs  Sign  inter item  alpha  correlation  correlation  covariance  Phonemic Awareness  433  +  0.63  0.51  0.47  0.86  Correct Graphemes Per  433  +  0.81  0.74  0.42  0.84  Minute  Correct Words Per Minute  432  +  0.86  0.81  0.41  0.83  Correct Non Words Per  432  +  0.85  0.79  0.41  0.83  Minute  Oral Reading Fluency  432  +  0.83  0.77  0.41  0.83  Reading Comprehension  433  +  0.66  0.54  0.46  0.86  Listening Comprehension  433  +  0.36  0.20  0.54  0.89  Writing  433  +  0.73  0.63  0.44  0.85  Test scale          0.45  0.87      80    Table 37- Cronbach’s Alpha, Grade 3 Subsample   Table 37� Cronbachʹs Alpha, Grade 3 Subsample  average  item�test  item�test  Item  Obs  Sign  inter item  alpha  correlation  correlation  covariance  Phonemic Awareness   433  +  0.56  0.43  0.59  0.91  Correct  Graphemes  Per  433  +  0.84  0.78  0.52  0.88  Minute  Correct Words Per Minute  433  +  0.91  0.87  0.50  0.87  Correct  Non  Words  Per  433  +  0.88  0.83  0.51  0.88  Minute  Oral Reading Fluency  433  +  0.86  0.81  0.51  0.88  Reading Comprehension  433  +  0.80  0.73  0.53  0.89  Listening Comprehension   433  +  0.56  0.43  0.60  0.91  Writing   433  +  0.76  0.67  0.54  0.89  Test scale              0.54  0.90    75 81  Table 38- Cronbach’s Alpha, Grade 1 Subsample, Zero Scores Converted to Missing 76   Table 38� Cronbachʹs Alpha, Grade 1 Subsample, Zero Scores Converted to Missing  average  item�test  item�test  Item  Obs  Sign  inter item  alpha  correlation  correlation  covariance  Phonemic Awareness   430  �  0.53  0.07  �0.01  .  Correct  Graphemes  Per  212  +  0.21  �0.16  0.03  0.19  Minute  Correct Words Per Minute  68  +  0.54  0.31  �0.01  .  Correct  Non  Words  Per  14  +  0.87  0.82  0.00  .  Minute  Oral Reading Fluency  84  +  0.52  0.28  0.00  .  Reading Comprehension  430  +  0.30  �0.07  0.05  0.28  Listening Comprehension   430  �  0.55  0.11  �0.02  .  Writing   430  �  0.46  0.03  0.01  0.05  Test scale              0.01  0.04    82    Table 39- Cronbach’s Alpha, Grade 2 Subsample, Zero Scores Converted to Missing   Table 39� Cronbachʹs Alpha, Grade 2 Subsample, Zero Scores Converted to Missing  average  item�test  item�test  Item  Obs  Sign  inter item  alpha  correlation  correlation  covariance  Phonemic Awareness   433  +  0.65  0.50  0.42  0.83  Correct  Graphemes  Per  399  +  0.80  0.70  0.37  0.80  Minute  Correct Words Per Minute  348  +  0.83  0.76  0.36  0.80  Correct  Non  Words  Per  221  +  0.83  0.75  0.38  0.81  Minute  Oral Reading Fluency  322  +  0.84  0.76  0.37  0.80  Reading Comprehension  433  +  0.65  0.51  0.41  0.83  Listening Comprehension   433  +  0.42  0.22  0.50  0.87  Writing   433  +  0.73  0.61  0.39  0.82  Test scale              0.40  0.84    77 Table 40- Cronbach’s Alpha, Grade 3 Subsample, Zero Scores Converted to Missing 78   Table 40� Cronbachʹs Alpha, Grade 3 Subsample, Zero Scores Converted to Missing  average  item�test  item�test  Item  Obs  Sign  inter item  alpha  correlation  correlation  covariance  Phonemic Awareness   433  +  0.56  0.42  0.57  0.90  Correct  Graphemes  Per  426  +  0.84  0.78  0.49  0.87  Minute  Correct Words Per Minute  410  +  0.91  0.88  0.46  0.86  Correct  Non  Words  Per  351  +  0.85  0.79  0.49  0.87  Minute  Oral Reading Fluency  402  +  0.86  0.81  0.48  0.87  Reading Comprehension  433  +  0.80  0.72  0.50  0.87  Listening Comprehension   433  +  0.56  0.43  0.57  0.90  Writing   433  +  0.76  0.67  0.51  0.88  Test scale              0.51  0.89        84    Table 41 - ANOVA Results: Differences in Means across Sub-tests Table 41 - ANOVA Results: Differences in Means across Sub-tests SUBSCALE Sum of Squares df Mean Square F Sig. Phonemic Awareness Between Groups 61.00 2 30.50 308.25 .000 Within Groups 127.93 1293 .10 Total 188.93 1295 Correct Grapheme Sounds Per Minute Between Groups 110177.65 2 55088.82 371.69 .000 Within Groups 191638.63 1293 148.21 Total 301816.28 1295 Correct Familiar Words Per Minute Between Groups 55960.06 2 27980.03 408.12 .000 Within Groups 88645.64 1293 68.56 Total 144605.69 1295 Correct Unfamiliar Words Per Minute Between Groups 38960.60 2 19480.30 303.22 .000 Within Groups 83067.46 1293 64.24 Total 122028.06 1295 Oral Reading Fluency Between Groups 189370.08 2 94685.04 316.02 .000 Within Groups 387402.77 1293 299.62 Total 576772.84 1295 Reading Comprehension Between Groups 22.71 2 11.36 270.99 .000 Within Groups 54.18 1293 .04 Total 76.89 1295 Listening Comprehension Between Groups 15.36 2 7.68 116.58 .000 Within Groups 85.20 1293 .07 Total 100.57 1295 Writing (score) Between Groups 23.25 2 11.63 329.38 .000 Within Groups 45.64 1293 .04 Total 68.89 1295 79 86 Table 42 Teacher and School Specific Characteristics: Subtest Averages Between Groups   Table 42 Teacher and School Specific Characteristics: Subtest Averages Between Groups  Do you hold a Certificate in Primary Teaching?     ORF  RCOMP %  LCOMP %  DICT %  Mean  20.200  0.160  0.190  0.394  No  SD  15.555  0.167  0.247  0.204  N  20  20  20  20                    Mean  18.359  0.191  0.290  0.316  Yes  SD  23.323  0.273  0.292  0.226  N  746  746  746  746                    Mean  18.407  0.190  0.287  0.318  Total  SD  23.148  0.270  0.292  0.225  N  766  766  766  766  t�statistic  �0.514  0.800  1.775  �1.670  d.f.  12.220  0.001  0.003  0.002     p�value t test  0.308  0.285  0.163  0.172  p�value z test  0.304  0.212  0.038  0.047    80   Table 43 Teacher and School Specific Characteristics: Subtest Averages Between Groups Table 43 Teacher and School Specific Characteristics: Subtest Averages Between Groups  Highest Qualification  RCOMP  LCOMP  DICT     ORF  %  %  %  Mean  20.200  0.160  0.190  0.394  None  SD  15.555  0.167  0.247  0.204  N  20  20  20  20                    Mean  18.273  0.187  0.287  0.316  Certificate in Primary Teaching  SD  23.295  0.271  0.291  0.227  N  716  716  716  716                    Mean  14.371  0.188  0.150  0.320  Certificate in Education  SD  19.215  0.236  0.186  0.223  N  16  16  16  16                    Mean  18.241  0.187  0.282  0.318  Total  SD  23.034  0.268  0.289  0.227  N  752  752  752  752  � t�statistic  �0.538  0.710  1.732  1.673     d.f.  12.248  0.001  0.003  0.002  p�value t test  0.300  0.303  0.167  0.171  p�value z test  0.295  0.239  0.042  0.047    81 Table 44 Teacher and School Specific Characteristics: Subtest Averages Between Groups   Table 44 Teacher and School Specific Characteristics: Subtest Averages Between Groups  Have you attended an inservice?    ORF  RCOMP %  LCOMP %  DICT %  Mean  20.209  0.206  0.309  0.342  No  SD  24.126  0.285  0.303  0.237  N  559  559  559  559              Mean  14.986  0.176  0.260  0.261  Yes  SD  23.361  0.246  0.269  0.192  N  177  177  177  177              Mean  18.953  0.199  0.297  0.322  Total  SD  24.033  0.276  0.296  0.229  N  736  736  736  736  t�statistic  �2.572  �1.366  �2.040  �4.572  d.f.  4.105  0.000  0.001  0.000    p�value t test  0.031  0.201  0.145  0.069  p�value z test  0.005  0.086  0.021  0.000  Have You Attended a Reading Training?    ORF  RCOMP %  LCOMP %  DICT %  Mean  20.371  0.203  0.300  0.348  No  SD  24.458  0.281  0.301  0.240  N  520  520  520  520              Mean  15.982  0.193  0.270  0.271  Yes  SD  22.864  0.267  0.281  0.181  N  210  210  210  210              Mean  19.108  0.200  0.292  0.326  Total  SD  24.077  0.277  0.295  0.227  N  730  730  730  730  t�statistic  �2.300  �0.422  �1.258  �4.661  d.f.  3.626  0.000  0.001  0.000    p�value t test  0.052  0.373  0.214  0.067  p�value z test  0.011  0.336  0.104  0.000    82 Table 45 Teacher and School Specific Characteristics: Subtest Averages Between Groups   Table 45 Teacher and School Specific Characteristics: Subtest Averages Between Groups  Experience Level     ORF  RCOMP %  LCOMP %  DICT %  Mean  13.371  0.116  0.233  0.260  0�4 years  SD  21.611  0.226  0.257  0.194  N  217  217  217  217                    Mean  24.859  0.244  0.347  0.419  5�9 years  SD  25.830  0.264  0.317  0.240  N  114  114  114  114                    Mean  16.584  0.202  0.274  0.324  10�19 years  SD  18.560  0.278  0.295  0.212  N  255  255  255  255                    Mean  34.876  0.329  0.400  0.444  20�29 years  SD  38.157  0.398  0.372  0.332  N  31  31  31  31                    Mean  21.047  0.228  0.326  0.310  30�39 years  SD  23.629  0.275  0.289  0.220  N  149  149  149  149                    Mean  19.104  0.224  0.314  0.364  40+ years  SD  27.176  0.296  0.272  0.284  N  99  100  100  100                    Mean  18.581  0.197  0.291  0.327  Total  SD  23.644  0.275  0.291  0.233  N  865  866  866  866  t�statistic  4.060  4.387  3.316  6.084  d.f.  7.944  0.001  0.001  0.001     p�value t test  0.002  0.071  0.093  0.052  p�value z test  0.000  0.000  0.000  0.000    83 Table 46 Teacher and School Specific Characteristics: Subtest Averages Between Groups   Table 46 Teacher and School Specific Characteristics: Subtest Averages Between Groups  Library Present     ORF  RCOMP %  LCOMP %  DICT %  Mean  15.113  0.164  0.272  0.292  No  SD  18.351  0.250  0.290  0.210  N  415  415  415  415                    Mean  23.074  0.233  0.313  0.358  Yes  SD  28.289  0.296  0.295  0.243  N  369  369  369  369                    Mean  18.860  0.197  0.292  0.323  Total  SD  23.875  0.275  0.293  0.228  N  784  784  784  784                    t�statistic  4.611  3.488  1.955  4.023  d.f.  2.972  0.000  0.000  0.000     p�value t test  0.022  0.089  0.150  0.078  p�value z test  0.000  0.000  0.025  0.000  Supervision in Library     ORF  RCOMP %  LCOMP %  DICT %  Mean  8.025  0.130  0.230  0.259  No  SD  9.718  0.224  0.309  0.197  N  40  40  40  40                    Mean  28.188  0.275  0.339  0.399  Yes  SD  30.684  0.311  0.300  0.251  N  269  269  269  269                    Mean  25.578  0.256  0.325  0.381  Total  SD  29.617  0.305  0.303  0.249  N  309  309  309  309                    t�statistic  8.329  3.607  2.090  4.016  d.f.  5.788  0.002  0.003  0.001     p�value t test  0.000  0.086  0.142  0.078  p�value z test  0.000  0.000  0.018  0.000    84 Table 47 Teacher and School Specific Characteristics: Subtest Averages Between Groups   Table 47 Teacher and School Specific Characteristics: Subtest Averages Between Groups  Reading Corner Present     ORF  RCOMP %  LCOMP %  DICT %  Mean  15.166  0.129  0.203  0.308  No  SD  23.076  0.244  0.273  0.243  N  180  180  180  180                    Mean  19.920  0.216  0.317  0.328  Yes  SD  24.133  0.278  0.295  0.224  N  584  584  584  584                    Mean  21.155  0.250  0.350  0.325  99 missing code  SD  20.821  0.330  0.259  0.216  N  20  20  20  20                    Mean  18.860  0.197  0.292  0.323  Total  SD  23.875  0.275  0.293  0.228  N  784  784  784  784                    t�statistic  2.390  4.030  4.782  1.006  d.f.  3.938  0.000  0.001  0.000     p�value t test  0.048  0.077  0.066  0.249  p�value z test  0.008  0.000  0.000  0.157    85 Table 48 Teacher and School Specific Characteristics: Subtest Averages Between Groups   Table 48 Teacher and School Specific Characteristics: Subtest Averages Between Groups  PTA Functioning    ORF  RCOMP %  LCOMP %  DICT %  Mean  21.535  0.226  0.267  0.330  No  SD  24.699  0.284  0.282  0.236  N  302  302  302  302              Mean  19.376  0.199  0.341  0.328  Yes  SD  24.066  0.275  0.298  0.225  N  335  335  335  335              Mean  9.928  0.127  0.210  0.237  99 missing code  SD  13.172  0.238  0.272  0.187  N  99  99  99  99              Mean  18.991  0.201  0.293  0.317  Total  SD  23.451  0.275  0.291  0.227  N  736  736  736  736              t�statistic  �1.115  �1.191  3.247  �0.126  d.f.  3.737  0.000  0.001  0.000    p�value t test  0.173  0.222  0.095  0.460  p�value z test  0.132  0.117  0.001  0.450      86 Table 49 Teacher and School Specific Characteristics: Subtest Averages Between Groups   Table 49 Teacher and School Specific Characteristics: Subtest Averages Between Groups  Meeting with Parents     ORF  RCOMP %  LCOMP %  DICT %  Mean  19.289  0.207  0.273  0.316  No  SD  23.458  0.279  0.280  0.233  N  275  275  275  275                    Mean  18.813  0.193  0.306  0.329  Yes  SD  24.298  0.274  0.300  0.227  N  498  498  498  498                    Mean  18.982  0.198  0.294  0.324  Total  SD  23.989  0.276  0.294  0.229  N  773  773  773  773                    t�statistic  �0.267  �0.677  1.489  0.705  d.f.  3.177  0.000  0.000  0.000     p�value t test  0.404  0.310  0.188  0.305  p�value z test  0.395  0.249  0.068  0.241    87 Table 50 Teacher and School Specific Characteristics: Subtest Averages Between Groups Table 50 Teacher and School Specific Characteristics: Subtest Averages Between Groups  How Often do Teachers Meet with Parents     ORF  RCOMP %  LCOMP %  DICT %  Mean  19.279  0.194  0.320  0.334  About Once per Term  SD  23.826  0.276  0.303  0.233  N  443  443  443  443                    Mean  17.310  0.233  0.183  0.278  About Twice per Term  SD  33.717  0.300  0.216  0.147  N  36  36  36  36                    Mean  7.000  0.067  0.217  0.198  About Thrice per Term  SD  5.560  0.130  0.276  0.155  N  12  12  12  12                    Mean  18.835  0.193  0.307  0.326  Total  SD  24.449  0.275  0.299  0.227  N  491  491  491  491  t�statistic  �0.343  0.766  �3.518  �2.088  d.f.  31.979  0.003  0.001  0.001     p�value t test  0.367  0.292  0.088  0.142  p�value z test  0.366  0.222  0.000  0.018    88 Table 51 Teacher and School Specific Characteristics: Subtest Averages Between Groups   Table 51 Teacher and School Specific Characteristics: Subtest Averages Between Groups  School has Recommended Reading     ORF  RCOMP %  LCOMP %  DICT %  Mean  14.876  0.170  0.296  0.300  No  SD  19.808  0.263  0.296  0.212  N  426  426  426  426                    Mean  24.583  0.239  0.298  0.354  Yes  SD  27.970  0.291  0.294  0.244  N  326  326  326  326                    Mean  19.084  0.200  0.297  0.324  Total  SD  24.161  0.277  0.295  0.228  N  752  752  752  752                    t�statistic  5.327  3.346  0.082  3.170  d.f.  3.311  0.000  0.000  0.000     p�value t test  0.006  0.092  0.474  0.097  p�value z test  0.000  0.000  0.467  0.001  Teacher Owns Curriculum Guide     ORF  RCOMP %  LCOMP %  DICT %  Mean  17.617  0.171  0.253  0.300  No  SD  23.318  0.244  0.281  0.225  N  251  251  251  251                    Mean  19.303  0.208  0.311  0.333  Yes  SD  24.431  0.288  0.300  0.229  N  503  503  503  503                    Mean  18.742  0.195  0.292  0.322  Total  SD  24.063  0.275  0.295  0.228  N  754  754  754  754                    t�statistic  0.921  1.865  2.606  1.840  d.f.  3.342  0.000  0.000  0.000     p�value t test  0.213  0.157  0.117  0.158  p�value z test  0.179  0.031  0.005  0.033    89 Table 52 Teacher and School Specific Characteristics: Subtest Averages Between Groups   Table 52 Teacher and School Specific Characteristics: Subtest Averages Between Groups  Students Practised Identifying the Sounds in Letters     ORF  RCOMP %  LCOMP %  DICT %  Mean  4.650  0.070  0.270  0.194  Never  SD  7.788  0.163  0.236  0.137  N  20  20  20  20                    Mean  22.744  0.210  0.287  0.345  1 or 2 days  SD  25.123  0.268  0.297  0.230  N  167  167  167  167                    Mean  21.893  0.267  0.375  0.376  3 or 4 days  SD  27.005  0.314  0.304  0.225  N  210  210  210  210                    Mean  15.460  0.153  0.249  0.285  Daily  SD  20.905  0.247  0.281  0.223  N  369  369  369  369                    Mean  18.529  0.194  0.293  0.320  Total  SD  23.759  0.275  0.294  0.228  N  766  766  766  766  t�statistic  6.932  3.336  0.302  4.260  d.f.  6.635  0.002  0.003  0.001     p�value t test  0.000  0.093  0.407  0.073  p�value z test  0.000  0.000  0.381  0.000      90 Table 53 Teacher and School Specific Characteristics: Subtest Averages Between Groups   Table 53 Teacher and School Specific Characteristics: Subtest Averages Between Groups  The whole class repeated words or sentences that you said first     ORF  RCOMP %  LCOMP %  DICT %  Mean  25.661  0.285  0.515  0.332  Never  SD  22.712  0.306  0.300  0.180  N  26  26  26  26                    Mean  26.797  0.264  0.364  0.396  1 or 2 days  SD  31.592  0.317  0.324  0.256  N  78  78  78  78                    Mean  19.078  0.206  0.312  0.339  3 or 4 days  SD  23.354  0.282  0.297  0.225  N  188  188  188  188                    Mean  14.278  0.152  0.253  0.286  Daily  SD  20.441  0.248  0.279  0.218  N  431  431  431  431                    Mean  17.286  0.183  0.290  0.313  Total  SD  23.086  0.270  0.294  0.226  N  723  723  723  723  t�statistic  0.199  �0.294  �2.181  1.403  d.f.  31.703  0.005  0.005  0.002     p�value t test  0.422  0.409  0.137  0.197  p�value z test  0.421  0.384  0.015  0.080    91 Table 54 Teacher and School Specific Characteristics: Subtest Averages Between Groups   Table 54 Teacher and School Specific Characteristics: Subtest Averages Between Groups  Students copied down text from the chalkboard     ORF  RCOMP %  LCOMP %  DICT %  Mean  16.667  0.178  0.360  0.355  Never  SD  17.801  0.236  0.316  0.267  N  55  55  55  55                    Mean  21.386  0.238  0.292  0.372  1 or 2 days  SD  26.336  0.288  0.278  0.216  N  233  233  233  233                    Mean  17.295  0.177  0.321  0.279  3 or 4 days  SD  23.486  0.268  0.296  0.212  N  227  227  227  227                    Mean  18.553  0.185  0.269  0.320  Daily  SD  23.089  0.280  0.301  0.235  N  228  228  228  228                    Mean  18.917  0.199  0.299  0.326  Total  SD  23.961  0.277  0.294  0.228  N  743  743  743  743  t�statistic  1.597  1.609  �1.471  0.460  d.f.  8.620  0.001  0.002  0.001     p�value t test  0.075  0.177  0.190  0.363  p�value z test  0.055  0.054  0.071  0.323    92 Table 55 Teacher and School Specific Characteristics: Subtest Averages Between Groups   Table 55 Teacher and School Specific Characteristics: Subtest Averages Between Groups  Students retold a story that they had read     ORF  RCOMP %  LCOMP %  DICT %  Mean  29.016  0.329  0.315  0.425  Never  SD  22.345  0.315  0.269  0.241  N  73  73  73  73                    Mean  18.044  0.169  0.276  0.330  1 or 2 days  SD  24.807  0.258  0.298  0.236  N  340  340  340  340                    Mean  17.948  0.222  0.327  0.301  3 or 4 days  SD  22.914  0.283  0.301  0.186  N  203  203  203  203                    Mean  15.771  0.154  0.276  0.283  Daily  SD  22.208  0.258  0.282  0.236  N  144  144  144  144                    Mean  18.642  0.196  0.294  0.323  Total  SD  23.814  0.275  0.294  0.227  N  760  760  760  760  t�statistic  �3.730  �4.039  �1.092  �3.052  d.f.  8.551  0.002  0.001  0.001     p�value t test  0.003  0.077  0.236  0.101  p�value z test  0.000  0.000  0.137  0.001    93 Table 56 Teacher and School Specific Characteristics: Subtest Averages Between Groups   Table 56 Teacher and School Specific Characteristics: Subtest Averages Between Groups  Students sounded out unfamiliar words     ORF  RCOMP %  LCOMP %  DICT %  Mean  29.182  0.300  0.427  0.426  Never  SD  24.120  0.348  0.241  0.199  N  22  22  22  22                    Mean  17.390  0.193  0.319  0.362  1 or 2 days  SD  22.196  0.268  0.309  0.234  N  123  123  123  123                    Mean  19.641  0.213  0.292  0.297  3 or 4 days  SD  25.943  0.279  0.287  0.218  N  238  238  238  238                    Mean  17.804  0.179  0.279  0.320  Daily  SD  22.799  0.269  0.294  0.230  N  377  377  377  377                    Mean  18.642  0.196  0.294  0.323  Total  SD  23.814  0.275  0.294  0.227  N  760  760  760  760  t�statistic  �2.137  �1.365  �1.855  �1.357  d.f.  29.210  0.006  0.003  0.002     p�value t test  0.021  0.201  0.157  0.202  p�value z test  0.016  0.086  0.032  0.087    94 Table 57 Teacher and School Specific Characteristics: Subtest Averages Between Groups   Table 57 Teacher and School Specific Characteristics: Subtest Averages Between Groups  Students learned meanings of new words     ORF  RCOMP %  LCOMP %  DICT %  Mean  19.168  0.174  0.270  0.339  1 or 2 days  SD  26.707  0.266  0.292  0.248  N  198  198  198  198                    Mean  19.421  0.230  0.338  0.321  3 or 4 days  SD  21.390  0.280  0.298  0.203  N  196  196  196  196                    Mean  18.255  0.194  0.284  0.316  Daily  SD  23.631  0.278  0.292  0.230  N  357  357  357  357                    Mean  18.800  0.198  0.294  0.323  Total  SD  23.909  0.276  0.294  0.228  N  751  751  751  751  t�statistic  0.104  2.029  2.292  �0.798  d.f.  5.907  0.001  0.001  0.001     p�value t test  0.461  0.146  0.131  0.286  p�value z test  0.459  0.021  0.011  0.213    95 Table 58 Teacher and School Specific Characteristics: Subtest Averages Between Groups   Table 58 Teacher and School Specific Characteristics: Subtest Averages Between Groups  Students Read Aloud     ORF  RCOMP %  LCOMP %  DICT %  Mean  25.897  0.225  0.332  0.375  1 or 2 days  SD  31.523  0.324  0.305  0.270  N  111  111  111  111                    Mean  16.121  0.193  0.312  0.297  3 or 4 days  SD  23.002  0.277  0.295  0.199  N  181  181  181  181                    Mean  17.724  0.187  0.276  0.317  Daily  SD  21.562  0.261  0.290  0.225  N  474  474  474  474                    Mean  18.529  0.194  0.293  0.320  Total  SD  23.759  0.275  0.294  0.228  N  766  766  766  766  t�statistic  �2.837  �0.861  �0.550  �2.638  d.f.  11.778  0.001  0.001  0.001     p�value t test  0.008  0.274  0.340  0.115  p�value z test  0.002  0.195  0.291  0.004    96   Table 59 Teacher and School Specific Characteristics: Subtest Averages Between Groups Table 59 Teacher and School Specific Characteristics: Subtest Averages Between Groups  Students assigned reading on their own     ORF  RCOMP %  LCOMP %  DICT %  Mean  18.034  0.213  0.338  0.320  Never  SD  19.573  0.309  0.246  0.218  N  32  32  32  32                    Mean  14.302  0.129  0.228  0.308  1 or 2 days  SD  19.481  0.228  0.302  0.206  N  65  65  65  65                    Mean  17.260  0.208  0.335  0.318  3 or 4 days  SD  20.309  0.277  0.298  0.191  N  198  198  198  198                    Mean  20.879  0.210  0.293  0.338  Daily  SD  26.355  0.282  0.294  0.247  N  436  436  436  436                    Mean  19.190  0.202  0.300  0.329  Total  SD  24.076  0.278  0.295  0.228  N  731  731  731  731  t�statistic  �0.884  �1.355  �1.914  �0.273  d.f.  17.346  0.004  0.003  0.002     p�value t test  0.194  0.202  0.153  0.415  p�value z test  0.188  0.088  0.028  0.392      97 Table 60 Teacher and School Specific Characteristics: Subtest Averages Between Groups   Table 60 Teacher and School Specific Characteristics: Subtest Averages Between Groups  Written Evaluations     ORF  RCOMP %  LCOMP %  DICT %  Mean  5.826  0.049  0.195  0.228  Never  SD  9.538  0.130  0.247  0.177  N  86  86  86  86                    Mean  16.131  0.105  0.195  0.261  Once per term  SD  25.209  0.189  0.249  0.216  N  114  114  114  114                    Mean  16.545  0.192  0.354  0.313  Once or twice per term  SD  24.490  0.289  0.317  0.258  N  125  125  125  125                    Mean  21.811  0.243  0.321  0.353  Weekly or Monthly  SD  24.072  0.291  0.300  0.221  N  409  409  409  409                    Mean  18.159  0.190  0.292  0.317  Total  SD  23.641  0.272  0.296  0.227  N  734  734  734  734  t�statistic  4.002  2.503  �0.017  1.178  d.f.  6.571  0.001  0.001  0.001     p�value t test  0.004  0.121  0.494  0.224  p�value z test  0.000  0.006  0.493  0.119    98 Table 61 Teacher and School Specific Characteristics: Subtest Averages Between Groups   Table 61 Teacher and School Specific Characteristics: Subtest Averages Between Groups  Oral Evaluations    ORF  RCOMP %  LCOMP %  DICT %  Mean  16.182  0.245  0.218  0.386  Never  SD  10.187  0.246  0.282  0.204  N  22  22  22  22              Mean  27.689  0.197  0.345  0.351  Once per term  SD  38.578  0.294  0.285  0.288  N  58  58  58  58              Mean  16.167  0.192  0.359  0.302  Once or twice per term  SD  23.522  0.292  0.315  0.241  N  145  145  145  145              Mean  18.103  0.191  0.276  0.321  Weekly or Monthly  SD  21.986  0.269  0.292  0.213  N  465  465  465  465              Mean  18.441  0.193  0.297  0.322  Total  SD  23.999  0.275  0.298  0.226  N  690  690  690  690  t�statistic  2.088  �0.750  1.788  �0.609  d.f.  29.716  0.004  0.005  0.003    p�value t test  0.023  0.295  0.162  0.326  p�value z test  0.018  0.226  0.037  0.271    99 Table 62 Teacher and School Specific Characteristics: Subtest Averages Between Groups   Table 62 Teacher and School Specific Characteristics: Subtest Averages Between Groups  Review of Portfolios and other projects     ORF  RCOMP %  LCOMP %  DICT %  Mean  9.675  0.090  0.216  0.224  Never  SD  15.757  0.189  0.258  0.171  N  98  98  98  98                    Mean  12.397  0.106  0.239  0.240  Once per term  SD  20.314  0.199  0.268  0.196  N  228  228  228  228                    Mean  29.618  0.323  0.393  0.396  Once or twice per term  SD  30.822  0.320  0.331  0.231  N  117  117  117  117                    Mean  25.077  0.314  0.346  0.400  Weekly or Monthly  SD  22.257  0.312  0.300  0.211  N  142  142  142  142                    Mean  18.463  0.197  0.292  0.307  Total  SD  23.934  0.277  0.296  0.218  N  585  585  585  585  t�statistic  1.306  0.706  0.706  0.699  d.f.  4.310  0.001  0.001  0.000     p�value t test  0.131  0.304  0.304  0.306  p�value z test  0.096  0.240  0.240  0.242    100 Table 63 Teacher and School Specific Characteristics: Subtest Averages Between Groups   Table 63 Teacher and School Specific Characteristics: Subtest Averages Between Groups  Student Reads aloud from chalkboard     ORF  RCOMP %  LCOMP %  DICT %  Mean  4.650  0.070  0.270  0.194  Never  SD  7.788  0.163  0.236  0.137  N  20  20  20  20                    Mean  4.385  0.036  0.287  0.176  1 or 2 days  SD  9.626  0.111  0.307  0.156  N  39  39  39  39                    Mean  18.634  0.188  0.218  0.364  3 or 4 days  SD  19.507  0.264  0.271  0.201  N  33  33  33  33                    Mean  19.706  0.211  0.300  0.328  Daily  SD  24.817  0.282  0.300  0.229  N  624  624  624  624                    Mean  18.401  0.196  0.294  0.318  Total  SD  24.035  0.275  0.297  0.226  N  716  716  716  716  t�statistic  �0.114  �0.843  0.238  �0.441  d.f.  5.196  0.002  0.005  0.002     p�value t test  0.457  0.277  0.426  0.368  p�value z test  0.455  0.200  0.406  0.330    101 Table 64 Teacher and School Specific Characteristics: Subtest Averages Between Groups   Table 64 Teacher and School Specific Characteristics: Subtest Averages Between Groups  Review of homework  DICT     ORF  RCOMP %  LCOMP %  %  Mean  18.500  0.233  0.533  0.292  Once per term  SD  16.257  0.294  0.207  0.171  N  6  6  6  6                    Mean  21.000  0.200  0.267  0.333  Once or twice per month  SD  20.973  0.302  0.299  0.235  N  15  15  15  15                    Mean  18.231  0.191  0.291  0.320  Weekly or more often  SD  23.900  0.273  0.297  0.227  N  703  703  703  703                    Mean  18.291  0.191  0.292  0.320  Total  SD  23.772  0.273  0.297  0.227  N  724  724  724  724  t�statistic  0.292  �0.233  �2.332  0.451  d.f.  63.880  0.018  0.011  0.007     p�value t test  0.386  0.427  0.129  0.365  p�value z test  0.385  0.408  0.010  0.326    102 Table 65 Teacher and School Specific Characteristics: Subtest Averages Between Groups Table 65 Teacher and School Specific Characteristics: Subtest Averages Between Groups  Observation and/or competency checklists     ORF  RCOMP %  LCOMP %  DICT %  Mean  17.136  0.175  0.305  0.335  Once per term or more  SD  21.691  0.263  0.294  0.228  N  174  174  174  174                    Mean  22.597  0.217  0.344  0.312  Once or twice a month  SD  30.431  0.292  0.304  0.241  N  188  188  188  188                    Mean  16.955  0.190  0.265  0.318  Weekly or more often  SD  20.490  0.272  0.290  0.219  N  376  376  376  376                    Mean  18.435  0.193  0.295  0.320  Total  SD  23.769  0.275  0.296  0.227  N  738  738  738  738  t�statistic  1.977  1.450  1.242  �0.929  d.f.  7.588  0.001  0.001  0.001     p�value t test  0.044  0.192  0.216  0.262  p�value z test  0.024  0.074  0.107  0.176    103