Report No: AUS0000649 Solomon Islands Early Grade Reading Assessment (SIEGRA) Results Report 2018 EDU 1 © 2017 The World Bank 1818 H Street NW, Washington DC 20433 Telephone: 202-473-1000; Internet: www.worldbank.org Some rights reserved This work is a product of the staff of The World Bank. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of the Executive Directors of The World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this work is subject to copyright. Because The World Bank encourages dissemination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. Attribution—Please cite the work as follows: “World Bank. 2018. Solomon Islands Early Grade Reading Assessment (SIEGRA) Results Report. © World Bank.� All queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@worldbank.org. 2 Table of Content Executive Summary ............................................................................................................ 6 Key SIEGRA findings .................................................................................................... 8 Factors contributing to greater fluency and comprehension in Solomon Islands ......... 10 Conclusions ................................................................................................................... 11 Recommendations ......................................................................................................... 11 Chapter 1 / Introduction ................................................................................................... 18 1.1 Country Context ................................................................................................ 18 1.2 Why Testing Early Grade Reading Abilities is important ................................ 18 Chapter 2 / Evaluation Design and Methodology ............................................................ 20 2.1 Objectives of Study ........................................................................................... 20 2.2 Research Questions ........................................................................................... 20 2.3 Overview of EGRA instrument and SSME tools ............................................. 21 Chapter 3 / Instrument Development and Implementation ............................................... 22 3.1 Instrument Adaptation ...................................................................................... 22 3.2 Sample Design .................................................................................................. 24 3.3 Enumerators Training ....................................................................................... 25 3.4 Data Collection ................................................................................................. 25 3.5 Limitations ........................................................................................................ 26 Chapter 4 / EGRA Results ............................................................................................... 27 4.1 Results by Subtest ............................................................................................. 27 4.2 Summary of findings......................................................................................... 45 Chapter 5 / Performance in reading fluency and comprehension .................................... 50 Chapter 6 / SSME Findings and Analysis of Factors Associated with Reading Fluency 54 6.1 Association of student characteristics to student reading performance ............ 60 6.2 Association of teacher characteristics to student reading performance ............ 62 6.3 Association of teacher training and teaching guides to student reading performance .................................................................................................................. 63 6.4 Association of Classroom Environment to student reading performance ........ 64 6.5 Association of Teacher Instructional and Assessment Methods to Student Performance .................................................................................................................. 65 Chapter 7 / Conclusions and Next Steps .......................................................................... 69 Recommendations ......................................................................................................... 70 Annex 1 / Tables .............................................................................................................. 77 Annex 2 / Instruments ...................................................................................................... 87 Annex 3 / Test Reliability Measures .............................................................................. 105 3 List of Tables Figure 1: Letter Name Results by Region and Year ......................................................... 28 Figure 2: Initial Sound Results by Region and Year ........................................................ 30 Figure 3: Letter Sound Results by Region and Year ........................................................ 31 Figure 4: Familiar Words Results by Region and Year .................................................... 33 Figure 5: Non-words Results by Region and Year ........................................................... 34 Figure 6: Oral Passage Reading Results by Region and Year .......................................... 36 Figure 7: Reading Comprehension Results by Region and Year ..................................... 38 Figure 8: Listening Comprehension Results by Region and Year .................................... 39 Figure 9: Dictation Orthography Results by Region and Year ......................................... 41 Figure 10: Dictation Convention Results by Region and Year......................................... 42 Figure 11: Vocabulary Results by Region and Year ........................................................ 44 Figure 12: Percentage of Students Scoring Zero by Subtest and Year ............................. 45 Figure 13: Summary of Results: Number of correct answers for timed subtests by Year 46 Figure 14: Summary of Results: Number of correct answers for untimed sub-tests by Year Level ......................................................................................................................... 47 Figure 15: Summary of Results: Percentage of Correct Responses for Timed sub-tests by Gender ....................................................................................................................... 47 Figure 16: Summary of Results: Number of correct answers for untimed sub-tests by Gender ....................................................................................................................... 48 Figure 17: Summary of Results: ORF Distribution by Reading Comprehension Level .. 52 Figure 18: Distribution of ORF scores for students reading with at least 80% Comprehension ......................................................................................................... 53 4 List of Figures Figure 1: Letter Name Results by Region and Year ......................................................... 28 Figure 2: Initial Sound Results by Region and Year ........................................................ 30 Figure 3: Letter Sound Results by Region and Year ........................................................ 31 Figure 4: Familiar Words Results by Region and Year .................................................... 33 Figure 5: Non-words Results by Region and Year ........................................................... 34 Figure 6: Oral Passage Reading Results by Region and Year .......................................... 36 Figure 7: Reading Comprehension Results by Region and Year ..................................... 38 Figure 8: Listening Comprehension Results by Region and Year .................................... 39 Figure 9: Dictation Orthography Results by Region and Year ......................................... 41 Figure 10: Dictation Convention Results by Region and Year......................................... 42 Figure 11: Vocabulary Results by Region and Year ........................................................ 44 Figure 12: Percentage of Students Scoring Zero by Subtest and Year ............................. 45 Figure 13: Summary of Results: Number of correct answers for timed subtests by Year 46 Figure 14: Summary of Results: Number of correct answers for untimed sub-tests by Year Level ......................................................................................................................... 47 Figure 15: Summary of Results: Percentage of Correct Responses for Timed sub-tests by Gender ....................................................................................................................... 47 Figure 16: Summary of Results: Number of correct answers for untimed sub-tests by Gender ....................................................................................................................... 48 Figure 17: Summary of Results: ORF Distribution by Reading Comprehension Level .. 52 Figure 18: Distribution of ORF scores for students reading with at least 80% Comprehension ......................................................................................................... 53 5 Acknowledgement This report was prepared by Education Technology for Development. The overall research quality was monitored and advised by the World Bank and was funded through the Global Partnership for Education. The authors would like to express their gratitude to the Solomon Islands Ministry of Education and Human Resources Development. Sincere thanks are given to the supervisors and enumerators who helped support data collection, and the teachers and students who participated in the survey. 6 Executive Summary Early Grade Reading Assessment (EGRA) is a simple instrument that measures foundational reading skills in Grades (Years) 1-3. The results are used to identify progress towards achieving reading fluency and comprehension, which are essential skills for learning and completing primary education. This report summarizes the results of EGRA conducted in the Solomon Islands from September 11 to October 20, 2017. With funding from the Global Partnership for Education (GPE), the World Bank and Education Technology for Development (ET4d) carried out the assessment in collaboration with the Pacific Community (formerly known as the Secretariat of the Pacific Community) and the Solomon Islands Ministry of Education and Human Resources Developments (MEHRD). The findings of this assessment are expected to assist policymakers with designing effective early grade reading intervention strategies to improve student’s literacy outcomes. This activity is part of the Pacific Early Age and Readiness Program (PEARL), which was established to support Pacific Island countries improve policy and programming decisions around early grade literacy and school readiness. The Solomon Islands EGRA (SIEGRA) was administered to a nationally representative sample of students enrolled in Years1 1, 2 and 3. A total of 1,159 students (568 girls and 591 boys) participated in the assessment. The SIEGRA tool consisted of eight reading skills tests and two reading-related tests (listening comprehension and dictation). Unlike most EGRAs, which primarily test reading and listening skills, the SIEGRA included a short dictation exercise to assess early writing skills. A vocabulary subtest was also included to determine students’ knowledge of and ability to identify vocabulary words in both English and Pijin from two categories: words for everyday surroundings and spatial terms. In addition, a head teacher, teacher and student surveys to collect information on characteristics associated with reading outcomes were administered to identify factors contributing to reading fluency. The assessors also carried out a classroom observation in each school visited to assess the classroom environment and teaching resources available. The analysis of SIEGRA data included descriptive statistics (means and standard deviations) to measure average levels in basic reading skills, an analysis of variance to 1 In the Solomon Islands education system, grade levels are called years. Year 1 is a synonym for Grade 1. 7 determine statistical significance of gender and regional differences, and regression analysis to estimate the association of a given teacher, student or classroom characteristic and reading fluency outcomes. The key findings, factors associated with reading performance and recommendations are detailed below. Key SIEGRA findings • Students have good phonemic awareness Students are showing relatively good progress in phonemic awareness measured through the initial sound in a word subtest. In terms of overall mean scores, students could correctly identify an average of 8 out of 10 initial sounds. Students in Year 1 identified an average of 7 initial sounds correctly, and 8 correct initial sounds for Years 2 and 3. About 16% of Year 1 and less than 10% of Year 2 (8%) and Year 3 (6%) students could not identify one initial sound. Thus, there is progress of learning across the years and overall, students are showing advancement towards mastering this fundamental skill necessary for decoding new words. • Students have good letter name knowledge and letter sounds fluency Students’ ability to provide the sound (i.e., phoneme) related to an individual letter is foundational literacy that contributes to word identification. Year 3 students scored an average of 46 correct letter sounds per minute (clspm), those in Year 2 were not far behind with a mean score of 44 clspm and Year 1 had a mean score of 28 clspm. Analysis of zero scores showed that overall, only 9 % of students could not identify a single letter sound. For Year 1, 13% of students received zero scores, 7% for Year 2 and 6% for Year 3. In terms of letter names knowledge, there is significant learning progress between the years and less than 5% of the students had zero scores. Results for these two subtests therefore indicate that students are making good progress to mastering these foundational reading skills. • Students have difficulties in decoding words To read text fluently, students must be able to decode unfamiliar words by sounding out individual letters and syllables. Overall, students correctly identified an average of 13 non-words per minute (cnwpm). Analysis by year showed that students in Year 1 correctly decoded an average of 7 non-words per minute, 13 cnwpm for Year 2 and 18 cnwpm for Year 3. More than half (64%) of Year 1 students, almost half (43%) of Year 2 students and 30% of those in Year 3 could not correctly identify a single non-word. These results suggest that students have not quite mastered decoding skills in English, which helps to explain the low reading fluency and comprehension scores described below. 8 • Students are not yet fluent readers Oral reading fluency (ORF) is a measure of overall reading competence and defined by the ability to translate letters into sounds, unify sounds into words, process connections, relate text to meaning, and make inferences. Research 2 suggests a minimum fluency rate of 45-60 correct words per minute (cwpm). The overall mean score for oral reading fluency3 is 21 correct words per minute (cwpm). The bulk of the low scores was in Year 1, where students read an average of only 7 cwpm. An additional 12 cwpm was achieved by students in Year 2 with an average score of 19 cwpm, and an increase of 17 cwpm for Year 3 students with an average score of 36 cwpm. These results therefore indicate that the Year 3 students are making good progress towards achieving the international fluency benchmark with an average of 36 cwpm. It is important to keep in mind while interpreting these results that the students are learning in a language that they are less familiar with and would therefore take longer than the mother tongue. • Reading comprehension levels are very low Reading with comprehension is the ultimate goal of literacy instruction. About 84% of students in Years 1, 53% in Year 2 and 41% in Year 3 scored zero in reading comprehension. The majority of students’ reading comprehension levels are well below the international reading comprehension benchmark of 80%. Only 12% of all students tested and 14% of Year 3 students could comprehend 80% of the text. Poor performance on a reading comprehension tool would suggest that the students had problems with decoding, or with reading fluently enough to comprehend, or with vocabulary. Delay in mastering reading comprehension skills might be expected given that the students are learning in a less transparent language and therefore takes longer than the mother tongue. Findings from the student questionnaire revealed that most of the students (85.2%) reported speaking vernacular at home, 32.9% said they speak Pijin and 3.5% use English. The vocabulary subtest also revealed that students have better vocabulary skills in Pijin than English where students recognized 87% of the items in Pijin compared to only 52% of the English items. However, it is important to note that there are other possible contributing factors to this delay in mastering reading comprehension skills such as the quality of reading instruction and the availability and appropriateness of resources used. • Students show progression in reading skills between the years There was measurable progress between Years 1-3 across most sub-tests, including: oral reading fluency, letter names, letter sounds, familiar words and dictation 3 Oral Reading Fluency refers to the Oral Reading Passage subtest 9 orthography sub-tests. Students demonstrated the most improvements between the years in the letter names and oral reading fluency subtests. For the letter names subtest, the mean score increased by 18 correct letters per minute (clpm) from Year 1 to Year 2 (from a mean score of 45 clpm to 63 clpm) and an additional 10 clpm for Year 3 students compared to those in Year 2 (from a mean score of 63 clpm to 73 clpm). Scores in the ORF subtest increased by an average of 12 correct word per minute (cwpm) between Years 1-2 (from a mean score of 7 cwpm to 19 cwpm per minute), and an additional 17 cwpm between Years 2 and 3 (from a mean score of 19 cwpm to 36 cwpm). The letter sounds subtest also showed progression in learning between the years with an increase in average of 16 correct letter sounds for Year 2 students and an additional 2 correct letter sounds from Year 2 to Year 3. For Familiar Word subtask, an increase of 8 cwpm was recorded from Year 1 to Year 2 and 14 more cwpm from Year 2 to Year 3. Dictation orthography also recorded improvements in student skills with an increase of 10.3 points from Year 1 to Year 2 and an additional 4.3 points from Year 2 to Year 3. Year 3 students recorded the lowest percentage of zero scores ranging from 0%-48% across the 10 subtests. • Girls have higher reading fluency and comprehension The difference in girls’ and boys’ performance was somewhat significant. Girls had a lower percentage of zero scores and performed better than boys across all sub- tests. • Regional differences were statistically significant The Honiara region scored the highest across the majority of subtasks with the Western region not far behind. The Choiseul region scored the lowest for 8 subtests. The students with the lowest average scores for the remaining two subtests (Reading and Listening Comprehensions) were from the Isabel region. The difference between the highest and the lowest scores across the regions for all subtests is noteworthy. The differences in scores ranged from 0.6 to 21.2 points. Factors contributing to greater fluency and comprehension in Solomon Islands The following factors are statistically significant and strongly associated with gains of 6- 19 more cwpm in ORF scores: Student characteristics: • Student has books, newspapers or other reading materials at home in Pijin (average increase of 15 cwpm) • Student reads aloud to someone at home (average increase of 12 cwpm) • Student reads to himself/herself at home (average increase of 11 cwpm) 10 • Student speaks Pijin at home (average increase of 10 cwpm) • Student speaks vernacular at home (average increase of 9 cwpm) • Student receives help with homework from father (average increase of 7 cwpm) • Student has books, newspapers or other reading materials at home in English (average increase of 6 cwpm) Classroom environment: • Reading corner in the classroom (average increase of 11 cwpm) Teaching pedagogy and assessment: • Students take books home to read with their parents (average increase of 19 cwpm) Conclusions • Although the overall performance for oral fluency reading is low when compared to the internationally accepted standard of 45-60 cwpm, students in Year 3 are well on their way to meeting this benchmark. • Given the low reading comprehension scores and the significant number of students who cannot comprehend one question across the three years, it is most likely that the breakthrough point for reading fluency and comprehension will be much later than Year 3 for the majority of students thus compromising their ability to cope with content and curriculum in upper grades. While it is recognized that the students are learning in a second language and the delay in mastering reading skills is somewhat expected, low achievements in reading fluency and comprehension is not a desirable outcome for any education system regardless of the language of instruction. • Of all the contextual factors assessed, student characteristics had the strongest influence on reading fluency scores. Specifically, the three factors most strongly associated with improved ORF scores were: students who speak Pijin or vernacular at home, have reading materials in Pijin and English at home, and students who read aloud to someone or by themselves at home. Students with these characteristics scored between 6-15 cwpm higher than other students assessed. Recommendations Based on the findings, the following recommendations are presented for consideration by MEHRD, as means to improve the quality of early grade reading instruction for Solomon Islands students: 11 1. Teaching and Learning 1a. Teachers should provide explicit instruction in reading comprehension skills for students in early grades. Given that only 12% of students were able to comprehend 80% or more of grade level text, teachers should focus on improving reading comprehension levels for all years. Research shows that teachers are often the ones to ask questions orally and that students are expected to respond, but rarely are students asked to develop their own questions as they read (think aloud) or to make predictions and then check their predictions. Students need to be empowered to make meaning from text and to use an inquiry-based approach to reading. Teachers could ask students to make predictions before, during and after reading, to retell stories, or to identify problems within the story and provide possible solutions. 1b. Provide remedial instruction for non-readers. An analysis of zero scores revealed that 62% of students in Year 1, 37% in Year 2 and 22% in Year 3 could not correctly read a single word in the reading passage. Given this significant number of zero scores, it is important for the MEHRD to conduct classroom level assessments to identify non-readers in Years 1-3, diagnose the causes, and design targeted intervention strategies to address deficiencies. For instance, teachers may group students according to ability and provide remedial activities and appropriately levelled text. This “catch-up approach� is being used by UNICEF in Zambia using J-Pal’s research in India, which demonstrated that students grouped by ability is more effective than mixed- ability grouping. UNICEF will assess students in all three grades and group them according to reading abilities (non-readers, those who can read letter sounds, syllables, words, passages, etc.). Grouping students by reading ability rather than grades has produced dramatic results in India, Kenya and Ghana. Teachers and school administrators should further determine whether non-readers have learning disabilities (e.g., dyslexia) and design relevant intervention strategies for special needs students. 1c. Develop and implement activities that specifically focus on raising boys’ performance and interest in reading. The results illustrate that boys consistently performed lower than their female counterparts across the subtests. There may be cultural or gender barriers that affect boys’ interest and engagement in reading activities. In addition to designing strategies to address low reading abilities of boys and girls, stakeholders at all levels should discuss the potential challenges specific to boys and design strategies to improve boys’ reading achievement. Successful strategies that have worked in other countries include developing gender-sensitive materials that attract boys’ attention (such as sports, science fiction, fantasy, comic books, digital text, and stories that are humorous) and integrating reading into extracurricular activities (e.g., sports, health clubs). 12 2. Teacher Training 2a. Train Years 1-3 teachers in reading instruction and assessment with a focus on decoding skills and reading comprehension. Only 5% of teachers surveyed received pre-service training on early grade literacy and teaching English as a second language. Furthermore, only 11% of the teachers were trained on this in the last year and 14% in the last two years. With regards to reading assessment, only 13% of the teachers received training and 7% of this number received the relevant tools for conducting assessment. SIEGRA results indicate that students are particularly weak in decoding unfamiliar words, and have low reading comprehension skills. Students also have very low writing skills as shown in the dictation orthography and dictation convention subtest results. Therefore, teachers need to be trained on early grade reading instruction and assessment. Training should concentrate on providing teachers with instructional strategies for building foundational reading skills such as phonics, decoding and comprehension. Additionally, training should incorporate curriculum expectations, highlight writing and its relationship with reading, and provide teachers with the specific approaches, classroom activities and assessment methods to achieve results. 2b. Ensure that pre-service course content provides new teachers with essential knowledge and skills related to improving reading and literacy outcomes. One of the main objectives of primary teacher education courses should be to equip pre- service teachers with the skills to teach and support emergent skills of students. Pre-service training programs therefore need to assist teacher trainees to understand and be able to use various strategies to develop foundational literacy skills for early grade students. Emphasis in pre-service should include training future teachers on how to deliver explicit teaching about phonemic awareness, alphabetic principle/phonics, vocabulary, reading fluency and comprehension and writing. There should also be opportunities for teacher trainees to observe and engage with effective reading teaching practices before they are sent out into the field of teaching. 3. Reading Materials 3a. Ensure that teachers have adequate and appropriate instructional materials for quality instruction. The development of reading skills in any language depends on many factors and among them is the availability and use of appropriate instructional materials. The findings show that only 32% of teachers had teaching guides and 28% did not have curriculum statements. As well, only 33% reported to have sufficient materials for teaching reading. The curriculum statement is a must-have tool in the classroom for every teacher as it outlines the philosophy, goals, subject content, learning standards, teaching strategies and teaching resources for any area of learning. An equally important tool is the teaching guide which serves as an important source of support in focusing early grades reading instruction on what students need to know and be able to do based on the literacy goals and other learning 13 outcomes in the prescribed curriculum. These important teaching and learning tools together with other reading instructional materials provide strong support for the effective teaching and learning of reading. In light of the above findings, it is recommended that the teachers are provided with adequate and varied instructional materials for teaching and learning of English reading skills in primary schools. 3b. Ensure that classrooms have libraries/reading corners with a variety of reading materials and that these are used during reading instruction. Reading corners in the classroom was a statistically significant factor associated with improved ORF average scores of 11 cwpm. Results show that 41% of classrooms had reading corners and only 51% of the schools had libraries. The majority of the classrooms (37%) had 1-5 copies of student textbooks, only 2% had 11 or more instructional readers and 15% had none at all. Classrooms are therefore not well equipped with the necessary reading resources to support reading instruction. Thus, it is recommended to develop reading corners in the classrooms with a variety of books (fiction and non-fiction) including graded readers and challenging materials for the experienced readers. Teacher guides for reading should be available to assist teachers in effectively using these books for reading instruction. The International Reading Association (IRA) recommends that classroom libraries start with at least seven books per child and purchase two additional new books per year. The optimal number of books in a classroom library is 300-600, depending on the grade level and number of copies4. The number of books teachers should expect children to read during the school year is 100-125 picture books by the end of Grade 1 and 50-75 chapter books by the end of Grade 2. Ideally, reading materials should include a variety of narrative and expository books at different levels of difficulty and reflect the interests of the pupils. A low-cost option is to provide e-readers if the schools can make available the required equipment to support student access to such resources. E-readers allow students and teachers to choose from a variety of genres, it is portable so students can read from home or school, and it’s read aloud features provide additional support for emergent readers56. In addition to provision of an increased number of hard and soft copy books, teachers should be trained on how to better integrate materials into their instruction and on how to develop attractive reading corners7. 3c. Encourage family involvement as an integral part of early grades reading. 4 Neuman, S. (undated). The importance of the classroom library. Available at: http://teacher.scholastic.com/products/paperbacks/downloads/library.pdf 5 Adams, A. & van der Gaag, J. (2011). First Step to Literacy: Getting Books in the Hands of Children Available at: https://www.brookings.edu/research/first-step-to-literacy-getting-books-in-the-hands-of-children/ 6 UNESCO (2014). Reading in the mobile era: A study of mobile reading in developing countries. http://unesdoc.unesco.org/images/0022/002274/227436E.pdf 7 Neuman, S. (undated). The importance of the classroom library. 14 Analysis of the contextual information collected show that students’ characteristics had the greatest association with improved ORF scores. Students that have access to reading materials in Pijin at home is critical to increasing oral reading fluency by 15 cwpm and 6 cwpm if the materials are in English. Results also show that students who read aloud to himself/herself at home had an additional 11 cwpm. As well, students taking books home on a daily basis to read with their parents was statistically significant and associated with higher ORF scores of 19 more cwpm. It is therefore important to engage parents in reading activities to ensure that students and parents are spending the recommended time (20-40 minutes per day) reading at home. Parents should also be encouraged to have reading materials at home to support students in practicing reading outside of school. Establishing partnerships with parents is a huge challenge and one that is often avoided in literacy initiatives, but it is critical. 4. Language 4a. Language of Instruction Policy. As previously mentioned, students who had access to reading materials in Pijin at home scored 15 cwpm more than their peers. As well, students speaking Pijin or vernacular at home was associated with higher ORF scores of 10 more cwpm and 9 more cwpm respectively and both factors were statistically significant. Findings also showed that while the language of instruction is English, students’ vocabulary skills are stronger in Pijin than English. Many researches argue that children most easily acquire reading skills in their mother tongue and that, with appropriate instruction, materials, and other supportive resources and effective guidance, they can successfully transfer those skills to a second language, and in this case English, resulting eventually in better achievement in both languages. The SIEGRA results is critical evidence that students are learning better in their mother tongue hence MEHRD might need to reconsider its language of instruction policy for primary schools. 4b. Support students’ language skills in their mother tongue or dominant language. As previously mentioned, results have shown that students have better vocabulary skills in Pijin (an average of 87% correct responses) than English (an average of 52% correct responses). The vocabulary subtest measures receptive language skills to better gauge students’ expressive oral language skills. The results suggest that the students need to develop vocabulary skills in English to support comprehension. It is therefore recommended that teachers place more emphasis on building students English language vocabulary. 4c. Training of teachers to improve their skills in reading and writing in English and on the principles of code switching. An alarming finding of this study was that only 11% of the teachers said they were able to read grade level English passages fluently with comprehension and only 5% reported they could write stories in perfect English. The study also found that the majority of teachers 15 were in fact teaching in Pijin (87%) and 88% said they switch between English and Pijin in class. Given the above results, it is therefore recommended that teachers be trained on reading and writing skills in the English language if it is retained as the language of instruction. Teachers also need training on the principles of code switching to ensure that teaching and learning of the target language (English) is given the prominence it requires. 5. National Benchmarks for Reading 5a. Establish national benchmarks for reading skills in early grades A benchmarking system provides critical evidence to redirect the education system as a whole to get as many students as possible to achieve the approved standards. This requires improving classroom instruction, ensuring a culture of shared accountability over learning at the school level as well as strengthening practices and support for reading at home. The findings of this study can guide the MEHRD and its stakeholders in developing specific benchmarks for reading in Years 1 to 3. Data shows that only 12% of students assessed met the international reading benchmark of 80% and for ORF, students scored 21 correct words per minute which is also below the international standard of 45-60 correct words per minute. The Ministry of Education together with its stakeholders should therefore decide on the level of comprehension that is acceptable as demonstrating full understanding of a given text. Most countries have settled on 80% or higher (4 or more correct responses out of 5 questions) as the desirable level of comprehension. With a reading comprehension benchmark confirmed, SIEGRA data can then be used to show the range of oral reading fluency (ORF) scores, measured in correct words per minute (cwpm), obtained by students to be able to achieve the desired level of comprehension. Discussion is needed to determine the value within that range that is put forward as the benchmark for ORF. SIEGRA results show that for those students who met the 80% benchmark, their reading fluency rate was 76 cwpm. This can be the starting point for MEHRD in determining the national benchmark. Alternatively, a range can indicate the levels of skill development that are acceptable as “proficient� or meeting a grade level standard (for example, 40 to 50 cwpm). The process then proceeds in the same manner for each skill area assessed in the SIEGRA subtests. Once the benchmark and targets are set, the MEHRD can then conduct a national reading campaign to inform all stakeholders of the new benchmarks, train teachers and then regularly monitor and report progress towards achieving the targets at all levels (national, regional and schools). Equally important is for MEHRD to ensure that if a benchmark system is introduced, it should include adequate mechanisms to identify struggling and non-readers in order for them to receive the necessary support to reach the grade standards before completion of a school year. 6. Additional research on findings not well understood 6a. Conduct a follow-up study on teaching practices and assessment methods to identify strengths and weaknesses. Several teaching methods showed a negative association with students’ ORF scores. Daily shared reading recorded an average of 14 less words and, when this activity is done 16 regularly (3-4) days, students scored an average of 21 less words. As well, when teachers work regularly on word building with students, results showed that students had lower ORF scores of 19 less words. It is also interesting to note that results indicate that the most frequently used instructional method was reading comprehension and yet, students’ scores in this subtest were very low. It is therefore important to understand why these approaches have an adverse relationship. It could be because teachers were not adequately supported through pre-service and in-service training to master the new teaching methodologies or that training was not aligned with best practices in reading instruction. Additionally, teachers may not understand how to accurately assess students and utilize the results for reflection and lesson planning. A further investigation into the teaching practices and assessment methods used in the classrooms will better inform professional development intervention strategies and other ongoing support in these areas for effective reading instruction. 6b. Identify the causes for the differences in performance across regions and develop context-relevant interventions. The Honiara region scored the highest across the majority of subtasks while the Choiseul region scored the lowest for 8 subtests. Analysis of the findings by region also showed that students from the Isabel area were the weakest in reading comprehension, listening comprehension and shared the bottom ranking with Choiseul for the initial sounds subtest. The difference between the highest and the lowest scores across the regions for all subtests is significant and warrants an investigation into the causes. Once the causes are identified, interventions specific to the regions’ contexts can then be developed and implemented to address the gaps. It is also recommended that an investigation be conducted to determine the factors contributing to high reading achievements in Honiara and the Western regions and to identify lessons learned and best practices that may be replicated to improve scores in other regions. 17 Chapter 1 / Introduction 1.1 Country Context Solomon Islands is a country in the southwestern part of the Pacific Ocean. It consists of six major and approximately 900 smaller volcanic islands, coral atolls and reefs, more than 300 of them are inhabited. They stretch about 900 miles in a south-easterly direction from Papua New Guinea towards Fiji. The Solomon Islands education system is structured so that the primary school cycle lasts 6 years, lower secondary for 3 years and upper secondary for 4 years. Primary education is not compulsory in the Solomon Islands but the government provides an annual grant to support the schools financially. Repetition is also high and survival rates show that only 67% of the children who enter primary may reach Year 6 without repeating or dropping out8. 1.2 Why Testing Early Grade Reading Abilities is important Teaching students to read at a young age is the cornerstone of improving educational outcomes. Students who do not learn to read in the early years of schooling often struggle and repeat grades. Not understanding printed information and not being able to follow written instructions and communicate well in writing, these children risk falling further and further behind those who can read effectively in later grades 9 . Without effective interventions, the literacy gap between good and poor readers increases over time. Effective readers absorb increasing amounts of written information, enhancing their vocabularies and improving their comprehension, while ineffective readers lose motivation, reading a fraction of the amount and remaining unable to comprehend more complex information referred to as the Matthew Effect10. Research has found that students who are unable to read proficiently by the end of Year 3 or grade 3 are four times more likely to leave school without a diploma than proficient 8 Solomon Islands National Education Action Plan 2016-2020 9 Give, A. and A. Wetterberg, 2011. The Early Grade Reading Assessment: Applications and Interventions to Improve Basic Literacy. Research Triangle Park, NC, USA: Research Triangle Institute 10 Stanovich, K. 1986. Mathew Effects in Reading: Some consequences of Individual Differences in Acquisition of Literacy. Reading Research Quarterly. 11(4). 18 readers 11 . Research also suggests that poor readers are more likely to experience behavioural and social problems in subsequent grades and are more likely to repeat grades12. This affects the social and economic wellbeing of these children in future life. MEHRD has made impressive efforts to improve the quality of teaching and learning during the previous Education Strategic Framework 2007-2015 (ESF). However, if quality is measured in terms of student performance, the situation remains a concern. Although there has been some improvement in literacy achievements during the previous ESF, this is considered unsatisfactory. Improving student achievement is now recognized as the most important area where further progress is required. An unacceptable number of students are still below the expected level of literacy. The latest Solomon Islands Standardised Test of Achievements (SISTA) results confirm a positive trend and that some moderate improvements are happening in Year 4 and Year 6 literacy but the issue of critical level of underachievement remains. The Pacific Early Age and Readiness Program (PEARL) was established to improve policy and programming decisions for school readiness and early grade literacy throughout the Pacific region. The SIEGRA results provide an initial measure of students reading abilities in Years 1 to 3. 11 Annie E. Casey Foundation. 2010. Early Warning: Why Reading By the End of Third Grade Matters. Baltimore, MD, USA: Annie E. Casey Foundation. 12 Hernandez, D. 2011. Double Jeopardy: How Third-Grade Reading Skills and Poverty Influence High School Graduation. New York, USA: The Annie E. Casey Foundation 19 Chapter 2 / Evaluation Design and Methodology 2.1 Objectives of Study The purposes of SIEGRA are to: • ascertain reading levels of students attending Years 1 to 3 • use information collected to monitor student’s reading levels in these grades • establish a baseline on student’s reading levels to inform and improve teaching and learning, address curriculum gaps, inform resource distribution, and other policy decisions pertaining to learners in the early primary year levels 2.2 Research Questions Aligned with the above purposes, the survey was developed specifically to answer the following research questions: 1. What are children’s reading fluency and comprehension levels in grades 1-3? 2. What is the difference in performance between girls and boys? 3. What reading skills need to be strengthened? 4. What student, teacher and classroom environmental factors are associated with reading achievement? a. Are teachers adequately trained and supported to teach early grade reading? (teach) b. Does the classroom environment enhance reading (e.g., student work displayed, reading corners, visual aids)? c. Are there sufficient reading and teaching materials to support student learning (e.g. at least one textbook per student, additional supplementary materials)? (text) d. Do teachers spend sufficient time on reading activities according to the lesson plan and teacher guide? (time) e. Are teachers fluent in the language of instruction? (tongue) f. How does using English (student’s non-native language) as the medium of instruction affect reading outcomes? (tongue) g. Do teachers assess students’ reading skills and modify instruction according to students’ reading levels? (test) h. Do parents, siblings or other family members support students to read at home? Do students have access to reading materials in the language of instruction? 20 5. What additional resources, training, policies or support is required to improve overall reading achievement? 2.3 Overview of EGRA instrument and SSME tools EGRA is not a high-stakes accountability measure to determine whether a child should advance to the next grade. As well, it is not to evaluate individual teachers. It is a diagnostic tool to assess early grade reading outcomes. The EGRA subtests measure predictors of early grade reading which are the skills found to be the most reliable evidence of later reading success. As a formative assessment tool, teachers can either use EGRA in its entirety or select subtasks to monitor classroom progress, determine trends in performance, and adapt instruction to meet children's instructional needs. EGRA’s theoretical framework serves to support its adaptation to other languages. It was decided by MEHRD that SIEGRA would be conducted in English. Although the reading assessment tools were developed in English, it was recognised that in Grades 1-3, many children were still in the early stages of learning English so Pijin was selected to be used for the purpose of communication and giving instructions. The questionnaire component of an assessment includes the collection of background and contextual data at different levels. The information gathered can be a powerful tool in providing explanations of the outcomes of an assessment’s cognitive component. This information enables a more in-depth understanding of the observed test outcomes and the implications of these outcomes for designing improvement strategies. SIEGRA included a set of contextual data collection tools to provide a Snapshot of School Management and Effectiveness (SSME). SSME focus is on specific aspects of school environments teacher- student interactions, student-to-student interactions, resources available, and pedagogical practices. 21 Chapter 3 / Instrument Development and Implementation 3.1 Instrument Adaptation An instrument design workshop was conducted by ET4D in the Solomon Islands from 5th to 9th June 2017. The workshop was facilitated by consultants from the University of the South Pacific and the Pacific Community (SPC) working with ET4D. Thirteen in-country participants drawn from MEHRD’s Curriculum unit, Assessment Unit, Literacy Development Unit and school teachers worked with the consultants to provide local knowledge, context, advice and language input for the instruments and administration process. The workshop also provided capacity building for the participants in developing reading assessment instruments, the ability to consider the various issues relevant to instrumentation, and to administer the field trial and review instruments based on results from the pilot. Participants also identified a number of key areas and skills learned at the workshop that they considered useful for their own work, including the ability to develop reading assessment instruments, prepare and critique comprehension passages for assessment, actively participate in the entire process of developing, trialing and refining the instruments, and analyse pilot results. In addition, the teachers among the participants welcomed the opportunity to have a set of tools to inform classroom instruction and assessment of early grade reading. The SIEGRA tool was adapted to the Solomon Islands context and administered nationally from September 11th to October 20th, 2017. 3.1.1 SIEGRA Tools The SIEGRA tools were administered in English, which is the language of instruction for primary schools in the Solomon Islands. Although the assessment tools were developed in English, it was recognised that the majority of students in Years 1-3 were still in the early stages of learning English so Pijin was used for the purpose of communication and giving instructions. Prior to the workshop, a language analysis of the curriculum materials in the early grades of primary schools was conducted to get a good understanding of any language issues to be considered in the development of the SIEGRA tools. The letter frequency and word frequency lists were also developed. For the letters and words subtasks (subtasks 1-6), only one instrument each was developed for trialing. The reading and listening passages were 22 developed according to specific criteria relating to length, text type, structure and language level. Five comprehension questions were also developed to assess students’ understanding of the passage they read or heard, the first four of which were text-based explicit information and the final was an inferential question. A vocabulary subtest, in which students were asked to identify words through their pictorial representation, was also included to determine students’ knowledge of and ability to identify vocabulary words in both English and Pijin. The subtest measured students’ receptive vocabulary (ability to understand words that they hear or read) from two categories: words for everyday surroundings and spatial terms. The purpose of the subtest was to compare students’ different levels of vocabulary in each language. Each text, comprehension questions and dictation sentences were presented to and reviewed by the whole group before finalizing them for the first pilot. See Annex 2 for copies of the SIEGRA instruments. The SIEGRA instruments consisted of the following ten subtests: Table 1: EGRA instrument structure and early skills tested Test Skills Measured Timed Letter Name Knowledge Ability to read alphabet letters with accuracy and fluency Yes Initial Sound Identification Phonemic awareness – the ability to identify sounds in spoken No words Letter Sound Knowledge Phonics – the ability to identify sounds of letters with Yes accuracy and fluency Familiar Word Reading Ability to read familiar words with fluency and accuracy Yes Nonword Reading Ability to decode linguistically sound invented words Yes Oral Passage Reading Ability to read a short passage with fluency and accuracy Yes Reading Comprehension Ability to respond to several comprehension questions based No on passage Listening Comprehension Ability to comprehend a short story read aloud No Dictation Orthography and Convention skills No Vocabulary Receptive language skills - ability to identify vocabulary No words in both English and Pijin from two categories - words for everyday surroundings and spatial terms 3.1.2 Survey Tools In order to get a broader picture of the contextual environment around students’ development of reading skills, additional tools were developed to collect background information from students, teachers and head teachers as well as classroom observations. Workshop participants reviewed and validated questionnaires designed for students, teachers, head teachers and classroom observations. Student survey questionnaires were 23 also translated into Pijin and validated and administered after the EGRA subtasks. The survey tools were revised where necessary as a result of feedback after the trials. 3.1.3 Instruments Piloting On the third day of the workshop, participants piloted the materials at two nearby schools. The purpose of the pilot was to test the instruments in order to identify any necessary modifications based on observation of the process, reports from the participants and from the pilot results. The results also assisted in selecting the version of parallel items to be used in the assessment. Student samples of 15 from each school were randomly selected from separate boys’ and girls’ class lists across grades 1 to 3 to get a gender and year-level balance. After the pilot, the group reconvened to provide feedback on the process and review the results. Changes were made to the instruments before returning to the same schools for a second pilot test. This time the classroom observation, teacher questionnaires and head teacher questionnaires were also field-tested. 3.2 Sample Design The sample included 40 schools. The statistician provided a target number of schools to select for each island. Using a list of total schools on each island, the number randomizer was used to select the ones that were evaluated. In the remote islands, schools that were not accessible were replaced. If there was more than one classroom for a year, and each class had their own teacher, the supervisor assigned each class a number (starting from 1 up to the total number of classes currently in session) and selected one class using a random number generator. The general rule was to randomly select five girls and five boys from each grade for a total of 30 children per school. A total of 1,160 students, 568 girls and 592 boys, participated in the assessment (see Table 2). Table 2: SIEGRA Sample by Region, Year and Gender Province Year 1 Year 2 Year 3 Total Girl Boy Total Girl Boy Total Girl Boy Total Choiseul 13 15 28 11 11 22 10 9 19 69 Guadalcanal 17 17 34 18 14 32 15 15 30 96 Honiara 9 9 18 9 10 19 9 9 18 55 Isabel 4 4 8 4 4 8 4 4 8 24 Malaita 82 95 177 95 106 201 108 100 208 586 Western 47 51 98 51 53 104 62 66 128 330 Total 172 191 363 188 198 386 208 203 411 1160 24 3.3 Enumerators Training Training was conducted over a five-day period. The first two days were spent on introducing the project, where an overview of the testing that was given, along with some of the research and pedagogy underlying the EGRA instrument and methods used. The tablets were also introduced and a review of the use of the tablets was given. The morning of day three was spent in a local school where the enumerators worked directly with children. In the afternoon, the enumerators were given their first assessor’s accuracy measure (AAM) test. The AAM is a measure to determine the degree of agreement among multiple enumerators administering the same test at the same time to the same student. This type of test or measurement of assessors’ skills determines the trainee’s ability to accurately administer EGRA. Assessors who are unable to reach the 90% or above accuracy standard in the final AAM are not selected for data collection. Day four was also spent in a school followed by a second AAM in the afternoon and a general discussion of mistakes that were made and areas where improvement was needed. The final AAM was given on the morning of day five and the rest of the time was partly spent explaining the logistical details involved in the upcoming travel. There was also a practice session where the enumerators simulated working with an interpreter as communication difficulties was anticipated in the more remote schools where students do not speak English fluently. 3.4 Data Collection Fifteen enumerators selected during the training session participated in the data collection which took place over a six week period, 11th September to October 20, 2017. One of the six weeks was school holiday and no work was possible. The data collection teams consisted of either four or two members for remote islands with less than four schools. For the teams of four, one person was assigned the supervisor role and was in charge of team organization, classroom observations, and teacher and head teacher interviews. The supervisor was also trained in SIEGRA administration and was available to help when time allowed, though in practice almost all student reading assessments were performed by the other three members of the team. Each school visit required 4 days of work, of which 3 days were for SIEGRA administration and one day for interviews and observation. The Solomon Islands presented a diverse and challenging set of conditions. The country is spread over many large and small islands covering a large territory, often with difficult access. Additionally, there is a great diversity of languages spoken and, contrary to the assumptions of the instrument design team that Pijin was widely spoken, Pijin was not always spoken or understood well by the children. A great deal of logistical and technical (regarding language difficulties) planning was needed. While there were initial difficulties 25 with obtaining good enumerator candidates, the final teams performed well and effectively dealt with the various problems they encountered. The planning of the data collection was changed several times because of problems with enumerator availability and physical access issues that arose after the initial schools were selected. The final sample closely followed the initial planning, however, in a few cases it was determined that access to selected schools would have been unreasonably difficult, so similar schools were chosen with the MEHRD, as replacements. 3.5 Limitations Language was one of the key challenges of this survey. There are over 70 languages spoken in the Solomon Islands, and although one of the criteria for determining team sizes and composition for data collection was the individual assessors’ local knowledge and language skills of the island, it was impossible to have every language represented in our pool of trained and qualified assessors. Therefore, we had to rely on locally available resources in some cases. Local knowledge was a great asset in regions like Malaita and Choiseul in particular. In the event of further data collection in the Solomon Islands, it is recommended to sample the islands and the schools early in the process, so that enumerators can be pre- selected based on the language spoken by the children from the selected schools. As well, access to participating schools was an issue. On the outer islands, road transportation for the most part does not exist. Most of the inland schools are accessed by foot and it can take several hours to reach them. Along the coast, schools are mostly accessed by boat. The geography of the Solomon Islands can make data collection extremely time consuming and expensive. 26 Chapter 4 / EGRA Results This section provides information on how students performed in each of the ten SIEGRA subtests. Most importantly, it gives comparative information of performance for year levels, gender and regions because the tool is the same for all grades. The average scores of the sub-tests are provided and disaggregated by class, gender and region. Mean scores are presented for the entire sample and the overall results therefore are the actual reading performance of the entire population, including those who can and cannot read. 4.1 Results by Subtest 4.1.1 Sub-test 1 – Automatic Letter Name Recognition The test of automatic letter name recognition is the most basic assessment of reading skills. It measures students’ ability to read the names of the letters of the alphabet with accuracy. Automaticity and fluency of letter name knowledge is a predictive skill for later reading success. During the SIEGRA, students were given a page of 100 randomly distributed upper- and lowercase letters and asked to say the names of as many letters as possible within one minute. The test was scored by the number of letters that students correctly named in one minute (correct letters per minute - clpm). The overall scores for sub-test 1 by year is presented in Table 3. The results show that students were able to identify an average of 61 clpm. Year 1 students recognised an average of almost half of the letters (45 out of 100), those in Year 2 named an average of 63 clpm and an average of 73 clpm were identified by the Year 3 students. Overall, there is significant learning progress in letter name knowledge between the years and less than 5% of the students had zero scores. Table 3: Sub-test 1 Letter Names: Results by Year Letter Name N Mean SD Min Max % zero scores (clpm) Overall 1,159 60.6 29.0 0 162.2 2.1% Year 1 363 45.0 23.8 0 150.0 1.7% Year 2 385 62.5 26.7 0 162.2 4.2% Year 3 411 72.6 29.0 0 146.3 0.5% 27 Table 4 shows automatic letter recognition results by gender. Overall, girls scored higher than boys, identifying correctly an average of 65 letters compared to 57 letters for the boys. Table 4: Sub-test 1 Letter Names: Results by Gender Letter Name N Mean SD Min Max % zero (clpm) scores Overall 1,159 60.6 29.0 0 162.2 2.1% Girls 568 64.7 28.6 0 146.3 1.6% Boys 591 56.7 28.8 0 162.2 2.5% Figure 1 below is a graphical representation of results by region and year. The region with the highest overall mean score of 65 clpm was Honiara and the lowest was Choiseul with 44 clpm. Two out of the 6 participating regions had averages below 50 clpm. Students in the other four regions correctly named an average of 60 clpm. Figure 1: Letter Name Results by Region and Year Letter Name average results by Region and Year 90 letters per minute (clpm) 80 Number of correct 70 60 50 40 30 20 10 0 Choiseul Guadalcanal Honiara Isabel Malaita Western Yr1 29.3 52.9 52.5 39.4 44.0 47.5 Yr2 49.2 66.3 62.3 46.5 63.0 64.4 Yr3 58.9 70.5 80.7 61.0 79.3 64.0 Overall 43.8 62.9 65.0 49.4 63.0 59.2 4.1.2 Sub-test 2 – Initial Sounds of Words Identification Phonemic awareness is an important precursor to both reading and writing where students learn to identify the sounds in words and match the sounds to the corresponding letter. Being able to correctly identify the sound made by the first letter of each word will help children figure out what the word says. Students are able to hear and isolate beginning sounds in a word before they can hear sounds in the middle or at the end of a word. At its very core phonemic awareness is a listening and speaking skill rather than a reading skill and so in this test, the assessor read aloud a word twice and then asked the student to 28 identify the first sound in the word. The test was comprised of 10 words and this was not a timed exercise. Based on the figures in Table 5 below, about 16% of Year 1 students and less than 10% of Year 2 (8%) and Year 3 (6%) students could not identify one initial sound. Thus, there is progress across the years. In terms of mean scores, overall students could correctly identify an average of 8 out of 10 initial sounds. Students in Year 1 identified an average of 7 initial sounds correctly, and Years 2 and 3 students identified an average of 8 correct initial sounds. Results show very minimal progression of learning between the years with only 1 point between Year 1 and Year 2 and no improvement shown between Year 2 and Year 3. Overall, students performed well in this initial decoding subtest. Table 5: Sub-test 2 Initial Sounds of Words: Results by Year Initial Sounds N Mean SD Min Max % zero scores Overall 1,159 8.0 3.2 0 10 9.6% Year 1 363 7.3 3.8 0 10 15.9% Year 2 385 8.2 3.0 0 10 7.5% Year 3 411 8.4 2.7 0 10 5.9% Table 6 shows the results of initial sounds subtest by gender. The difference between girls and boys performance is minimal, with mean scores of 8.2 and 7.8 respectively. Table 6: Sub-test 2 Initial Sounds of Words: Results by Gender Initial Sounds N Mean SD Min Max % zero scores Overall 1,159 8.0 3.2 0 10 9.6% Girls 568 8.2 3.2 0 10 10.4% Boys 591 7.8 3.2 0 10 8.8% Analysis by region shows that students from the Malaita, Western, Honiara and Guadalcanal regions scored an average of 8 initial sounds and the remaining 2 areas were not far behind with averages of 7.5 (refer to Figure 2). Year 3 students across the 6 regions averaged 8.0 – 9 correct initial sounds out of 10. 29 Figure 2: Initial Sound Results by Region and Year Number of correct initial sounds Initial sounds average results by Region and Year 10 9 8 7 6 5 4 3 2 1 0 Choiseul Guadalcanal Honiara Isabel Malaita Western Yr1 6.7 7.5 7.3 6.7 7.5 7.2 Yr2 7.2 7.8 8.1 7.2 8.2 8.5 Yr3 8.9 8.7 8.5 8.6 8.6 8.0 Overall 7.5 8.0 8.0 7.5 8.1 8.0 4.1.3 Sub-test 3 – Letter Sounds Letter sound knowledge or graphemic knowledge is an awareness of the letters or groups of letters which represent the individual speech sounds in language. Knowledge of how letters correspond to sounds and to match sound and symbol with automaticity are critical skills students must master to become successful readers. The letter sound knowledge test was administered similarly to the letter name knowledge subtest. Students were provided a page of 100 randomly distributed upper and lowercase letters of the alphabet and asked to provide the sounds (not the names) of as many letters as they could identify within a one-minute period. This subtest is timed and scored as the total number of correct letter sounds per minute (clspm). On average, students were able to identify 39 correct letter sounds per minute (clspm) Year 1 students correctly identified an average of 26 clspm. Students in Year 2 could accurately sound an additional 16 letters, for a total average score of 42 clspm. Scores increased by only 3 clspm for Year 3 students who correctly sounded an average of 45 clspm. The percentage of students scoring zero reduced from 13% in Year 1 to 10% in Years 2 and 3. The increase in average scores from Year 1 to Year 3 and reduction in the percentage of zero scores is an indication of learning progression from Years 1-3. Table 7: Sub-test 3 Letter Sounds: Results by Year Letter Sounds N Mean SD Min Max % zero scores Overall 1,159 39.4 25.3 0 230.8 8.5% Year 1 363 27.8 21.0 0 230.8 12.9% Year 2 385 43.6 27.8 0 177.8 7.3% Year 3 411 45.8 22.6 0 132.0 5.6% 30 Analysis by gender as shown in Table 8 below indicates that girls outperformed boys by correctly sounding an average of 7 additional letters within 1 minute. Data also shows that similar percentages of girls and boys had zero scores for this subtest. Table 8: Sub-test 3 Letter Sounds: Results by Gender Letter Sounds N Mean SD Min Max % zero scores Overall 1,159 39.4 25.3 0 230.8 8.5% Girls 568 43.0 24.7 0 230.8 8.3% Boys 591 36.0 25.4 0 177.8 8.6% Presented in Figure 3 below are the results by region, which clearly shows that there is a significant variation in scores between the regions. Guadalcanal students recorded the highest average score of 45 clspm. Choiseul and Isabel students had the lowest average scores of 27 clspm. An interesting finding for the Western region was that Year 2 students correctly sounded an average of 51 letters per minute compared to an average of 42 clspm for the Year 3 students in the same area. Figure 3: Letter Sound Results by Region and Year Letter Sounds average results by Region and Year 60 sounds per minute (clspm) Number of correct letters 50 40 30 20 10 0 Choiseul Guadalcanal Honiara Isabel Malaita Western Yr1 19.1 37.6 34.1 22.2 26.2 29.0 Yr2 30.1 49.7 38.9 25.4 41.5 50.8 Yr3 34.6 49.1 51.5 33.6 48.6 42.4 Overall 26.9 45.2 41.3 27.3 39.4 41.1 4.1.4 Sub-test 4 – Familiar Words The familiar word sub-test measures students’ ability to read familiar words with fluency and accuracy, both of which are necessary to become fluent readers. In administering this reading test, students were given a list of 50 familiar words (selected from high frequency words in Years 1 – 3 readers available in the classrooms) with instructions to read as many 31 as they could in one minute. Familiar-word reading is a timed test scored by the number of correct familiar words read per minute (cfwpm). Table 9 shows the results of the familiar word reading sub-test by year and includes mean scores and zero scores. The overall mean score was 20 cfwpm. Results show that the test was most difficult for Year 1 in which 51% of students could not identify one familiar word. The percentage of non-readers drops to 23% in Year 2 and 18% in Year 3, clearly showing an increase in knowledge. Overall, students read 20 cfwpm. Performance increased steadily each year from 10 cfwpm in Year 1 to 18 in Year 2 (an increase of 8 cfwpm) to 32 in Year 3 (an increase of 14 words). Table 9: Sub-test 4 Familiar Words: Results by Year Familiar Words N Mean SD Min Max % zero scores Overall 1,159 20.1 23.0 0 150.0 30.0% Year 1 363 9.5 17.9 0 150.0 51.1% Year 2 385 18.3 18.9 0 136.4 23.3% Year 3 411 31.7 25.6 0 120.0 17.5% Boys recorded a mean score of 18 cfwpm compared to 23 cfwpm for the girls, a difference of 5 words. Out of 591 boys that participated, 189 (32%) of them scored zero in this subtest and 159 (28%) of 568 girls. Table 10: Sub-test 4 Familiar Words: Results by Gender Familiar Words N Mean SD Min Max % zero scores Overall 1,159 20.1 23.0 0 150.0 30.0% Girls 568 22.6 22.6 0 136.4 28.3% Boys 591 17.7 25.0 0 150.0 31.6% Based on Figure 4 below, overall average scores across the regions ranged from 10 to 24 cfwpm. Honiara recorded the highest average score of 24 cfwpm and Western had the second highest average of 19 cfwpm followed by Guadalcanal with 17 cfwpm. The rest of the regions had averages that were below the overall mean score of 17: Malaita with 16 cfwpm, Isabel with 12 cfwpm and Choiseul with the lowest of 10 cfwpm. 32 Figure 4: Familiar Words Results by Region and Year Familiar words average results by Region and Year 40 Number of correct words 35 per minute (cwpm) 30 25 20 15 10 5 0 Choiseul Guadalcanal Honiara Isabel Malaita Western Yr1 4.1 8.7 14.2 4.0 4.4 8.9 Yr2 10.9 18.7 22.0 10.8 13.9 22.3 Yr3 17.9 23.9 36.1 20.6 27.8 25.0 Overall 10.1 16.8 23.9 12.2 15.9 19.3 4.1.5 Sub-test 5 – Non-words Non-word or invented words fluency measures a student’s ability to decode individual phonemes (use of the alphabetical principle) and then blend the sounds together to read words. It is a measure of students’ automaticity and accuracy in matching the letters to sounds in order to read the non-words. Unlike familiar word reading in which students can read from memory or sight recognition, on the non-word reading sub-test students must sound out the words and so it is a purer measure of decoding ability. Students were provided with a table of 50 made-up words and instructed to read as many as they could within one-minute. The test was timed and measured by the number of correct non-words read per minute (cnwpm). Table 11 below shows there was a significant proportion of students who scored zero - over half in Year 1 (64%), nearly half in Year 2 (43%) and about a third (30) in Year 3 with an overall average of 13 cnwpm. Year 1 students correctly decoded 7 cnwpm while Year 2 students read 13 cnwpm and Year 3 students read 18 cnwpm. Thus, average scores increased by 6 cnwpm from Year 1 to Year 2, and by 5 cnwpm from Year 2 to Year 3. These results do indicate progress from Years 1-3 however, the fluency of decoding ability appears too low to support comprehension. Table 11: Sub-test 5 Non-words: Results by Year Non-words N Mean SD Min Max % zero scores Overall 1,159 12.9 17.4 0 176.5 45.1% Year 1 363 7.1 5.3 0 85.7 63.5% Year 2 385 13.2 10.9 0 176.5 43.4% 33 Year 3 411 17.8 11.6 0 100.0 30.4% On average, girls correctly read an average of two more non-words than boys (14 and 11.9, respectively). For zero scores, half of the total number of boys (50%) could not read a single non-word correctly compared to 41% of girls. Table 12: Sub-test 5 Non-words Results by Gender Non-words N Mean SD Min Max % zero scores Overall 1,159 12.9 17.4 0 176.5 45.1% Girls 568 14.0 15.3 0 113.1 40.5% Boys 591 11.9 19.2 0 176.5 49.5% Based on Figure 5, results by region show that students from the Honiara area read an overall average of 16 cnwpm, which was the highest score for the 6 participating regions. Choiseul and Isabel had the lowest mean scores, each with an average of 6 correct non- words per minute. Figure 5: Non-words Results by Region and Year Non-words average results by Region and Year Number of correct non-words 30 per minute (cnwpm) 25 20 15 10 5 0 Choiseul Guadalcanal Honiara Isabel Malaita Western Yr1 3.2 7.3 9.8 2.5 2.9 5.3 Yr2 6.0 12.6 14.3 5.5 8.4 17.0 Yr3 8.4 15.5 25.6 9.1 18.8 11.2 Overall 5.6 11.6 16.4 5.9 10.4 11.3 4.1.6 Sub-test 6 – Oral Passage Reading 34 Oral reading fluency assessments have become a common methodology for measuring reading proficiency and growth13. Speed and accuracy are measurable elements of Oral Reading Fluency (ORF). It encompasses all of the previous reading skills plus the skills needed for comprehension - the ability to translate letters into sounds, unify sounds into words, process connections, relate text to meaning, and make inferences.14 ORF has been shown to be a powerful predictor of overall reading competence and comprehension15. This is the actual timed correct rate at which an individual reads and is measured as a raw score of total words read in a minute, minus the number of errors. The oral passage reading sub-test is timed. In order for students to understand a simple passage, they must be able to read it fast enough to retain the words in short-term memory. Research16 suggests a minimum fluency rate of 45-60 words per minute, depending on the complexity of the language. In this sub-test, students were asked to read a very short story comprised of 53 words. After one minute, the assessor stopped students and recorded the number of words read correctly. If the child could not read any words correctly in the first line, the assessor stopped the test early and the child received a score of zero. The results in Table 13 shows that over half of Year 1 students, 37% of students in Year 2 and 22% in Year 3 could not read a single word correctly of a grade-level story passage. The overall mean score for the ORF sub-test was 21 correct words per minute (cwpm), well below the 45-60 cwpm oral reading fluency standard. The bulk of the low scores were in Year 1, where students read an average of only 7 cwpm. Year 2 students improved by 12 cwpm with an overall average of 19 cwpm and an increase of 17 cwpm for Year 3, where students read an average of 36 cwpm. Despite impressive progress from Years 1-3, students in Year 3 scored below the international fluency standard. It is important to note that the SIEGRA tool was administered in English, which is the students’ second language, and this may have contributed to the low fluency scores. Table 13: Sub-test 6 Oral Passage Reading Fluency by Year Oral Reading Fluency N Mean SD Min Max % zero scores Overall 1,159 21.2 28.3 0 212.0 39.8% Year 1 363 7.0 14.4 0 212.0 62.4% Year 2 385 19.0 22.6 0 159.0 37.1% Year 3 411 35.8 34.7 0 156.0 22.4% Girls correctly read an average of 7 more cwpm than boys (24.8 compared to 17.7). Almost half of the boys (44%) scored zero on this subtest. 13 Wolf & Katzir-Cohen, 2001 14Hasbrouck, J., & Tindal, G. A. (2006). “Oral reading fluency norms : A valuable assessment tool for reading teachers.� The Reading Teacher, 59(7), 636–644. 15 Ibid 16Abadzi, H. (2011). Reading Fluency Measurements in EFA FTI Countries: Outcomes and Improvement Prospects. Education for all Fast Track Initiative 35 Table 14: Sub-test 6 Oral Passage Reading Fluency by Gender Oral Reading Fluency N Mean SD Min Max % zero scores Overall 1,159 21.2 28.3 0 212.0 39.9% Girls 568 24.8 30.9 0 167.4 35.6% Boys 591 17.7 25.0 0 212.0 43.9% Analysis by region shows that students from Honiara had the highest overall average fluency rate of 26 cwpm while Choiseul ranked the lowest with a mean score of 11 cwpm. Year 3 students in Honiara scored considerably higher than other regions with a mean score of 39 cwpm compared to 20-30 cwpm for the other regions; and they are not far from achieving the 45-60 cwpm fluency benchmark. Figure 6: Oral Passage Reading Results by Region and Year Oral Reading Fluency average results by Region and Year Oral reading fluency (orf) 45.0 40.0 35.0 30.0 25.0 20.0 15.0 10.0 5.0 0.0 Choiseul Guadalcanal Honiara Isabel Malaita Western Yr1 2.9 5.6 15.6 1.9 4.5 6.9 Yr2 12.1 18.8 23 9.7 14.0 22 Yr3 22.2 26.3 39.3 23.0 30.1 20.0 Overall 11.2 16.5 25.8 12.0 16.8 16.7 4.1.7 Sub-test 7 – Reading Comprehension The reading comprehension subtask measures the ability to answer comprehension questions based on the passage read. A total of five questions were provided for this subtest, consisting of direct, fact-based questions and at least one question requiring inference from the passage read. The questions were developed by the writers during the instrument development workshop. Students were asked questions only up to the point where they had stopped reading. For instance, if the child read the first sentence (10 words), s/he was asked one question. If s/he read half of the text (27 words), s/he was asked three questions; and, if s/he read all seven sentences (53 words), s/he was asked all five comprehension questions. Similarly, if learners did not read any of the text, they were not asked any questions and received a score 36 of zero. This sub-test is scored by the number of questions answered correctly. The scores are also calculated to determine the percentage of questions answered correctly out of a total possible (five). Table 15 shows that students achieved an overall average score of 1 correct response (22% correct). Out of 1,159 students, 59% of them could not respond to one question accurately. The highest percentage of students with zero scores was in Year 1 (84%), followed by Year 2 (53%) and Year 3 (41%). Students in Year 1 answered an average of 0.3 questions correctly, which means they scored 7% correct. Year 2 students scored an average of 23% correct or 1 correct response and those in Year 3 had an average percentage of 35% correct or 2 correct responses. Students showed a slight progression in learning from Years 1-3. Overall, students are not meeting the international reading comprehension benchmark of 80%. Since fluency determines the number of questions asked of students, low achievements means that students are not reading with fluency to have sufficient time to process the meaning of text. Table 15: Sub-test 7 Reading Comprehension by Year Reading N Mean % Correct SD Min Max % zero Comprehension scores Overall 1,159 1.1 21.7 1.6 0 5 58.7% Year 1 363 0.3 5.6 0.3 0 5 84.3% Year 2 385 1.1 22.8 1.5 0 5 53.0% Year 3 411 1.7 34.7 1.8 0 5 41.3% Figures in Table 16 show that there is minimal difference in reading comprehension abilities for girls and boys. Girls scored an average score of 25% correct and boys achieved an average of 18% correct. This means that both responded to average of 1 comprehension question correctly. Over 50% of girls and boys scored zero. Table 16: Sub-test 7 Reading Comprehension by Gender Reading N Mean % Correct SD Min Max % zero Comprehension scores Overall 1,159 1.1 21.7 1.6 0 5 58.7% Girls 568 1.3 25.4 1.7 0 5 54.4% Boys 591 0.9 18.1 1.4 0 5 62.8% Figure 7 below shows the average score for each of the six regions. With the exception of Isabel, all regions had an average percentage of more than 10% which is equivalent to 1 correct response. 37 Figure 7: Reading Comprehension Results by Region and Year Reading Comprehension average results by Region and Year comprehension (%) Correct reading 50.0 45.0 40.0 35.0 30.0 25.0 20.0 15.0 10.0 5.0 0.0 Choiseul Guadalcanal Honiara Isabel Malaita Western Yr1 6.0 7.3 13.2 0.7 4.0 6.9 Yr2 13.7 18.2 21.8 8.0 20.0 32.7 Yr3 18.6 33.7 47.2 16.0 38.1 31.9 Overall 11.9 19.1 27.1 8.5 21.5 24.7 4.1.8 Sub-test 8 – Listening Comprehension The purpose of the listening comprehension assessment is to measure whether the student can listen to a short passage being read aloud and then answer several questions correctly with a word or a simple statement. Poor performance on a listening comprehension tool would suggest that students simply do not have the basic vocabulary that the reading materials expect, or that they have difficulty processing what they hear. Testing of listening comprehension separately from reading comprehension is therefore important due to the different ways in which learners approach, process, and respond to text. The assessor read a short story to students and then asked five comprehension questions. Students had 15 seconds to respond to each question. As this was an untimed test, all students heard the entire story and responded to all five questions. For this reason, scores are based on the percentage of questions answered correctly. The overall mean score for listening comprehension was 45% correct, which is equivalent to 2 correct responses out of five questions asked. The results by year show progression between Years 1-3. By the end of Year 1, students could comprehend about 31% of the passage while Year 2 students understood 50% and Year 3 students comprehended 54% of the passage. Of the five questions asked, Year 1 students correctly responded to 1.5 questions, Year 2 students answered 2.5 questions correctly and Year 3 students correctly responded to 2.7 questions; thus there was a greater increase in comprehension between Years 1 and 2 than between Years 2 and 3. 38 Table 17: Sub-test 8 Listening Comprehension Results by Year Listening N Mean Mean SD Min Max % zero Comprehension % (raw) scores Overall 1,159 45.3 2.3 32.6 0 100 20.1% Year 1 363 30.5 1.5 29.4 0 100 34.4% Year 2 385 49.6 2.5 32.8 0 100 15.6% Year 3 411 54.4 2.7 30.7 0 100 11.7% Analysis by gender shows that girls were slightly better in listening comprehension than boys (47.1% (2 or 3 correct responses) and 43.5% (2 correct responses) respectively). Girls and boys had the same percentage of those with zero scores. Table 18: Sub-test 8 Listening Comprehension Results by Gender Listening N Mean Mean SD Min Max % zero Comprehension (%) (raw) scores Overall 1,159 45.3 2.3 1.6 0 5 20.1% Girls 568 47.1 2.4 1.6 0 5 20.0% Boys 591 43.5 2.2 1.5 0 5 20.1% Figure 8 presents performance by region for the 3 year levels. The Western region recorded the highest mean score of 52.2% which translates to 2.6 correct responses. Students in Malaita could comprehend 45.6% of the passage (2.3 correct answers) and those in Honiara 40.1% and Guadalcanal 39.3% which is 2.0 questions correctly answered. Figure 8: Listening Comprehension Results by Region and Year Listening Comprehension average results by Region and Year comprehension (%) Correct listening 70 60 50 40 30 20 10 0 Choiseul Guadalcanal Honiara Isabel Malaita Western Yr1 26.5 31.9 28.9 11.7 29.6 34.5 Yr2 32.2 37.1 40.9 13.4 49.0 62.2 Yr3 39.0 50.0 50.7 27.9 55.9 57.6 Overall 31.8 39.3 40.1 18.1 45.6 52.2 39 4.1.9 Sub-tests 9 - Dictation The dictation sub-test measures students’ alphabet knowledge and ability to hear and distinguish individual letter sounds in words and to spell words correctly. The sub-test assessed four basic writing skills: spelling, spacing, capitalization and punctuation (use of full stop). A compound sentence of 11 words was used for the dictation including a compound noun comprised of three morphemes. In pilot testing, the sentence was found to provide a good discrimination of scores between grades. The scores were calculated using weights to arrive at a maximum score of 100%. The spelling items received a weighting of 60% while other components (spacing, capitalization, comma and full stop) received a weighting of 40%. Given the focus of the EGRA is to assess reading ability, the spelling component received more weighting. The analysis for dictation therefore focused on two areas – orthography and convention. For this untimed sub-test, the assessor read aloud a short sentence of 11 words (The black dog was sleeping on Tom's mat under the tree) and asked students to write down what they had heard. The assessor read the sentence three times, once before students began writing and twice while they were writing. Students were given 15 seconds to complete writing after the third reading. Sub-test 9a – Dictation Orthography To assess for orthography, students were scored by the number of words spelt correctly out of a total of 34 points (3 points each for the 11 words and 1 point for the apostrophe to show possession in the phrase “Tom’s mat�). The average score for the entire sample in orthography was 13.2. Analysis of zero scores show that 72% of students in Year 1, 39% for Year 2 and 29% for Year 3 could not spell one word correctly. Performance increased by 10.3 points from Year 1 to Year 2 and by 4.3 points from Year 2 to Year 3. Both Year 2 and Year 3 students scored higher than the overall average of 13.2 (14.8 and 19.1 respectively). Table 19: Dictation Orthography Results by Year Dictation: N Mean SD Min Max % zero scores Orthography Overall 1,137 13.2 14.3 0 34 45.8% Year 1 344 4.5 9.6 0 34 71.9% Year 2 384 14.8 14.0 0 34 39.3% Year 3 409 19.1 14.3 0 34 28.8% Table 20 presents the overall mean scores for girls and boys. Girls scored an average of 14.9 points, which is higher than the boys average score of 11.7. Almost half of the boys (49.3%) and girls (42%) could not spell a single word correctly. 40 Table 20: Dictation Orthography Results by Gender Dictation: N Mean SD Min Max % zero Orthography scores Overall 1,137 13.2 14.3 0 34 45.8% Girls 557 14.9 14.5 0 34 42.2% Boys 580 11.7 13.9 0 34 49.3% The performance by region is shown in Figure 9. The highest mean score of 16.8 in dictation orthography was from the Honiara region and Guadalcanal region ranked second highest with a mean score of 16.2 points. There is a notable difference of 9.3 points between the highest and lowest mean scores for this subtest. Figure 9: Dictation Orthography Results by Region and Year Dictation - orthography average results by Province and Year 30 Correct orthography 25 20 15 10 5 0 Choiseul Guadalcanal Honiara Isabel Malaita Western Yr1 2.8 9.4 11.1 3.2 2.3 5.7 Yr2 7.0 18.1 15.2 9.7 12.8 19.4 Yr3 10.4 22.1 24.4 13.8 21.7 15.1 Overall 7.5 16.2 16.8 9.1 12.8 13.7 Subtest 9b. Dictation Convention Analysis for writing convention is presented in Table 21. These results were based on the same sentence that the students had written down from what they heard orally from the assessor. Overall, a mean score of 1.2 out of a possible score of 4 points is recorded for the correct use of a fullstop and capital letters for names and start of sentence. Year 1 recorded a mean score of 0.4, Year 2 had 1.4 and Year 3 had 1.8. In summary, Year 1 and Year 2 scored below half of the total possible score of 4. For the zero scores, 80% of students in Year 1, 45% of Year 2 students and 32% of those in Year 3 could not correctly apply any of the writing conventions tested to the sentence. Results therefore indicate that students are not consistently using full stops and capital letters in their writings or that they are still unsure of how to mark the beginning and ending of a sentence or use capital letters for proper nouns. 41 Table 21: Dictation Convention Results by Year Dictation: N Mean SD Min Max % zero scores Convention Overall 1,141 1.2 1.4 0 4 51.6% Year 1 361 0.4 0.8 0 4 80.4% Year 2 380 1.4 1.4 0 4 45.2% Year 3 401 1.8 1.5 0 4 32.1% There was minimal difference between boys and girls performance in this component of dictation. More than half the boys (55.3%) could not apply any of the writing conventions to the dictation sentence. Table 22: Dictation Convention Results by Gender Dictation: N Mean SD Min Max % zero Convention scores Overall 1,141 1.2 1.4 0 4 51.6% Girls 559 1.4 1.5 0 4 47.7% Boys 582 1.0 1.3 0 4 55.3% According to Figure 10 below, Honiara region had the highest ranking of 1.6 points out of a possible score of 4. Students from Choiseul seemed to have great difficulties with writing conventions with an average score of 0.2. Figure 10: Dictation Convention Results by Region and Year Dictation - convention average results by Region and Correct convention Year 2.5 2 1.5 1 0.5 0 Choiseul Guadalcanal Honiara Isabel Malaita Western Yr1 0.1 0.8 1.1 0.4 0.1 0.5 Yr2 0.14 1.6 1.5 1 1.1 2.0 Yr3 0.4 1.9 2.3 1.3 2.0 1.7 Overall 0.2 1.4 1.6 0.9 1.1 1.4 Results for the dictation subtest therefore indicate that students lack writing skills. They are struggling with spelling, use of apostrophe to show possession and are not consistently 42 using writing conventions like full stops and capital letters to mark the beginning and ending of a sentence or to indicate proper nouns. 4.1.10 Sub-test 10 - Vocabulary Vocabulary refers to the ability to understand the meaning of words when we hear or read them (receptive). Reading experts suggest that vocabulary knowledge of between 90 and 95% of the words in a text is required for comprehension17. Since none of the original EGRA subtasks measure vocabulary directly, this untimed vocabulary subtask was added to measure receptive language skills of individual words and phrases related to common objects and spatial relationships for both Pijin and English languages. Therefore for part 1 of the subtest, students were provided with 5 pictures of different objects and asked to name them in English and Pijin. Part 2 required the students to demonstrate their understanding of 5 spatial words in English (on, next, behind, under, to the right) using a piece of paper and a pencil. The purpose of the subtest was to compare students’ different levels of vocabulary in each language. This is especially important for English language learners, as they try to navigate their way through a non-native vocabulary. Young children naturally learn to communicate through listening and speaking. In order to make the transition to communicating through reading and writing, they need a large meaning vocabulary and effective decoding skills. The score for this subtest was based on the number of words the student is able to correctly identify. Table 23 shows students average scores for naming the given objects in English and in Pijin as well as their understanding of spatial words. The total possible score was 15 points – 5 points for correctly naming all of the objects in English, 5 points for correctly naming all of them in Pijin and another 5 points for correctly placing the pencil using the spatial words provided. Results show that students in Year 1 had a mean score of 10 points and those in Years 2 and 3 scored an average of 11 points. None of the students scored zero in this subtest. In summary, there was similar performance between the years and overall, students performed well in this subtest. Table 23: Vocabulary Results by Year Vocabulary N Mean SD Min Max % zero scores Overall 1,159 10.5 2.9 0 15 0.1% Year 1 363 9.5 3.0 0 15 0.3% Year 2 385 10.7 2.9 0 15 0.0% Year 3 411 11.2 2.6 0 15 0.0% 17 Nagy, W. E., & Scott, J. (2000). Vocabulary processes. In M. E. A. Kamil, P. B. Mosenthal, P. D. Pearson, & R. Barr, (Eds.), Handbook of reading research (Vol. III, pp. 269-284). Mahwah, NJ: Erlbaum. 43 Figures in Table 24 show very little difference between girls and boys performance. Table 24: Vocabulary Results by Gender Vocabulary N Mean SD Min Max % zero scores Overall 1,159 10.5 2.9 0 15 0.2% Girls 568 10.6 3.0 0 15 0.4% Boys 591 10.4 2.9 0 15 0.0% When comparing students’ performance based on the two languages used, students recognized 87% of the items in Pijin compared to only 52% of the English items, which indicates they have more knowledge of Pijin words than English. This is an important finding to inform developments in the schools language policy as well as classroom instruction in terms of more emphasis on building students English language vocabulary. Table 25: Descriptive Statistics N Minimum Maximum Mean STD Vocabulary Part 1 score (%) Pijin 1159 .00 100.00 87.3817 20.91388 Vocabulary Part 1 score (%) 1159 .00 100.00 52.0205 31.84940 English Analysis by region shows that Choiseul had the lowest mean score of 6.4 out of 15. The other five regions had mean scores that ranged from 9-12 which shows that most students are making good progress towards mastering the fluency words for their grades that are likely to appear in their daily reading and yearly testing. Figure 11: Vocabulary Results by Region and Year Vocabulary average results by Region and Year 14 Correct vocabulary 12 10 8 6 4 2 0 Choiseul Guadalcanal Honiara Isabel Malaita Western Yr1 5.9 8.7 9.2 7.9 9.4 11.1 Yr2 6.0 9.3 10.0 8.7 11.0 11.9 Yr3 7.6 9.4 10.7 9.2 11.2 12.5 Overall 6.4 9.1 10.0 8.6 10.6 11.9 44 4.2 Summary of findings Figure 12 presents the percentage of students with zero scores by subtest and year level. Overall, Year 1 recorded the highest percentage of students with zero scores for all subtests particularly for reading comprehension (84%) and dictation convention (80%) and dictation orthography (72%). More than half of the students had zero scores for six out of the ten subtests. The sub-test with the highest percentage of zero scores was reading comprehension. Over 80% of students in Year 1 scored zero, 53% in Year 2 and 41% in Year 3 could not respond to one reading comprehension question and scored zero. The sub-test with the least percentage of zero scores was vocabulary, in which no students in Year 2 and Year 3 scored zero. Although Year 3 students had the lowest percentage of those with zero scores overall, significant proportions of them scored zero in non-word reading (30%), in dictation orthography (29%) and in dictation convention (32%). Figure 12: Percentage of Students Scoring Zero by Subtest and Year 90 80 Percent of Students Scoring Zero by Year 70 60 Percent 50 40 30 20 10 0 Letter Letter Listenin Initial Familiar Nonwor Oral Reading Dictatio Dictatio Vocabul Name Sound g sounds Word d Reading compreh n_orthog n_conve ary Knowled Identific compreh of words Reading Reading Fluency ension raphy ntion ge ation ension Year1 0.3 1.7 15.9 8.5 51.1 63.5 62.4 84.3 34.4 71.9 80.4 Year2 0.0 4.2 7.5 8.3 23.3 43.4 37.1 53.0 15.6 39.3 45.2 Year3 0.0 0.5 5.9 8.6 17.5 30.4 22.4 41.3 11.7 28.8 32.1 Sub-tests Year1 Year2 Year3 On the timed reading fluency subtests, students scored highest in the letter name knowledge subtest with a mean score of 61 clpm. For the letter sounds subtest, students were able to correctly identify an average of 39 clspm. As well, students’ achieved an overall average score of 20 cwpm for the familiar word subtest. In the oral reading fluency sub-test, students’ mean scores significantly improved between the years from 7 cwpm in Year 1 to 19 cwpm in Year 2 and Year 3 students could correctly read an average of 36 cwpm. Despite this progression in learning for the years, the majority 45 of students across the years are reading well below international fluency standards18 of 45- 60 cwpm. The mean score for Year 3 students however is not too far behind this international standard. Students scored lowest in the non-word subtest with an overall average of 13 cnwpm. This indicates that students struggled most with decoding familiar and unfamiliar words, which is reflected in low reading fluency and reading comprehension scores. Figure 13: Summary of Results: Number of correct answers for timed subtests by Year Timed Sub-test Average Results by Year 80 72.6 70 62.5 Correct Answers 60 45.2 50 45.0 42.3 40 30 26.2 26.4 26.6 16.7 15.8 16.8 20 11.2 10 6.4 4.3 5.6 0 Letter Name Letter Sound Familiar Word Nonword Oral Passage Knowledge Identification Reading Reading Reading Sub-tests Year1 Year2 Year3 For the untimed subtests in Figure 14, listening comprehension skills were higher than reading comprehension for all year levels. Vocabulary had the highest overall mean score of 11 out of a possible score of 15 however, students performed better in Pijin than in English. Students performed well in the phonemic awareness subtest correctly identifying an average of 8 out of 10 initial sounds of words. For dictation orthography, students scored an average of 13.2 out of a possible score of 34 points. Analysis of zero scores showed that 72% of students in Year 1, 39% for Year 2 and 29% for Year 3 could not spell one word correctly in the dictation sentence and/or use an apostrophe to show possession. Analysis for the application of writing conventions to the same sentence showed an overall mean score of 1.2 out of a possible score of 4 points. Results therefore suggest that students are struggling with spelling, use of apostrophe to 18 Abadzi, H. (2011). Reading Fluency Measurements in EFA FTI Countries: Outcomes and Improvement Prospects. Education for all Fast Track Initiative 46 show possession and are not consistently using full stops and capital letters to indicate the start and ending of a sentence or for proper nouns. On average, students by the end of Year 3 could accurately respond to 2 comprehension questions (35% correct). A 35% score for reading comprehension is well below the international benchmark of 80%. Figure 14: Summary of Results: Number of correct answers for untimed sub-tests by Year Level Untimed Sub-test Average Results by Year 25 19.1 20 Correct Answers 14.8 15 11.2 10.7 9.5 8.4 10 7.3 8.2 4.5 5 2.5 2.7 1.1 1.7 1.5 1.4 1.8 0.3 0.4 0 Sub-tests Year1 Year2 Year3 For all of the timed subtests, girls outperformed their male counterparts. The subtest with the biggest difference in performance was letter name knowledge where boys scored an average of 8 clpm less than the girls. For the rest of the timed subtests, the difference between boys and girls mean scores were in the range of 2-7 points. Figure 15: Summary of Results: Percentage of Correct Responses for Timed sub-tests by Gender Timed Reading Results by Gender 64.7 56.7 Correct Answers 60 42.4 34.3 40 14.7 19.2 9.6 11.8 14.7 18.9 20 0 Letter Name Letter Sound Familiar Word Nonword Reading Oral Passage Knowledge Identification Reading Reading Sub-tests Boys Girls 47 For the untimed subtest, there were more female readers than male, and girls read better than boys in all reading skills tested. Figure 16: Summary of Results: Number of correct answers for untimed sub-tests by Gender Untimed Results by Gender 16 14.9 Correct Answers 14 11.7 12 10.4 10.6 10 7.8 8.2 8 6 4 2.2 2.4 2 0.9 1.3 1.0 1.4 0 sub-tests Boys Girls For comparisons between the regions, Choiseul scored the lowest for 8 subtests. The students with the lowest average scores for the remaining two subtests (Reading and Listening Comprehensions) were from the Isabel region. The difference between the highest and the lowest scores across the regions for all subtests ranged from 0.6 to 21.2 points. For the timed subtests, students from Honiara consistently scored the highest across the subtests except for letter sounds where those from Guadalcanal had a mean score of 45 clspm compared to 41 clspm for Honiara. However, Year 3 students from Honiara had the highest mean score of 52 clspm compared to all other regions with mean scores of less than 50 clspm. Choisel and Isabel had the lowest scores for all of the timed subtests. Students in Year 3 from Honiara had the highest mean scores for all of the timed subtests especially for the non-words subtest (26 cnwpm), oral reading fluency (39 cwpm) and familiar words (36 cfwpm). For the untimed subtests, all regions had similar results for initial sounds with mean scores of 8. With the exception of Isabel, all other regions had a mean score of more than 10 percent (1 correct response) for reading comprehension. The Western region had the highest mean score for listening comprehension with 52% (3 correct responses), Malaita, Honiara, Guadalcanal and Choiseul students could correctly respond to 2 questions and Isabel had the lowest score of just 1 correct respond (18%). For the Vocabulary subtest, Western, Malaita and Honiara had the top three mean scores of 12, 11 and 10 respectively out of a possible score of 15. Guadalcanal and Isabel had mean scores of 9 and the lowest 48 was Choiseul with a mean score of 6. For the dictation subtest, Honiara, Guadalcanal and Western regions had the 3 highest mean scores for dictation orthography and dictation convention and Choiseul had the lowest mean scores for both. 49 Chapter 5 / Performance in reading fluency and comprehension According to the second edition of the EGRA toolkit19, there are two steps to identifying a reading fluency benchmark. The first is to identify the level of reading comprehension that is expected for the grade level. In most countries, this level is set at 80% or higher (4 correct answers out of 5 questions) for grades 1-3 which is also the international standard for reading comprehension. This benchmark shows countries commitment to the importance of “readers� being those who can comprehend most of the text. The 80% threshold for reading comprehension will be used for the purposes of this EGRA study. Once the reading comprehension benchmark is set, the second step is to use EGRA data to show the range of ORF scores obtained by students against the desired level of comprehension. Given that students may be able to achieve the 80% comprehension benchmark at different fluency rates, and there could be a wide range of scores, a third step is added to the analysis. The third step is to identify the range of scores with the highest proportion of students meeting the comprehension benchmark. With this information, stakeholders may decide on the value within the fluency range that should be put forward as the reading fluency benchmark. Table 26 shows the actual distribution of correct scores in percentages for reading comprehension. Overall, 7% of students scored 80% correct and 5% achieved 100%. Thus, the total percentage of students achieving 80% or more reading comprehension is 12%. Table 26: Distribution of correct scores in percentages for reading comprehension by Year Level Comp. Score Year 1 Year 2 Year 3 Overall Mean 0% 85% 53% 41% 59% 20% 8% 15% 10% 11% 40% 4% 11% 13% 10% 60% 1% 11% 12% 8% 80% 0.3% 4% 14% 7% 100% 1% 5% 9% 5% Total 100% 100% 100% 100% As mentioned above, it was decided to set the level of comprehension at 80% as an indicator that students demonstrate full understanding of the text. Table 27 shows the 19 RTI International. (2016). Early Grade Reading Assessment (EGRA) Toolkit, Second Edition. Washington, DC: United States Agency for International Development. 50 percentage of students by year and gender that comprehended 80% or more of the text read. The distribution of scores by Year is: 1.3% of Year 1 students, 9% of Year 2 and 23% of Year 3 students who understood 80% or more of the text given. In terms of gender, 8% of female students and 4% of male students reached the 80% benchmark in reading comprehension, which is 4 correct responses out of 5 comprehension questions. Table 27: Percentage of students reading with 80% or more comprehension by Year Level and Gender 80% or more reading comprehension Mean (%) SD (%) N Overall 12% 24.3% 1159 Year 1 1.3% 7.4% 363 Year 2 9% 20.9% 385 Year 3 23% 32.2% 411 Female 8% 27.4% 568 Male 4% 20.3% 591 The second step looks at the distribution of fluency scores of students who reached the threshold level in reading comprehension. Figure 17 shows the distribution of ORF scores for each level of reading comprehension. Globally, an increase in the level of reading comprehension is associated with an increase in ORF scores. Table 28 confirms that mean scores in ORF were higher when students had higher levels of reading comprehension. It also shows the average mean score for students comprehending 80% of the text. The average mean score of students comprehending 80% of the text is 76.3 cwpm. Meanwhile, some students who were unable to answer a single reading comprehension question correctly could read up to 21220 correct words per minute. 20 Note that ORF scores may exceed the total number of words in a passage since it is a timed test. If students could read the entire passage within the 60 seconds allowed, the fluency score would take into account the time remaining from the 60 seconds. This explains why some students present such high value as 136 correct words per minute although the passage had only 60 words. 51 Figure 17: Summary of Results: ORF Distribution by Reading Comprehension Level Table 28: ORF Score by Reading Comprehension Score Reading Comp. Mean Fluency SD Min Max Sample (n) Score Score 0% 6.9 16.2 0 212 670 20% 27.3 19.7 0 120 167 40% 38.3 23.7 0 118 106 60% 62.1 31.6 14 159 88 80% 76.3 27.1 37 142 59 100% 82.6 25.2 50 145 69 Distribution of ORF scores for students who reach 80% in reading comprehension Figure 18 shows the distribution of ORF scores for the 139 students who achieved the level of 80% correct in reading comprehension. The majority of the students who met the benchmark read 75 or more correct words per minute. Scores above 53 indicate that students finished the paragraph in less than one minute while scores below 53 imply that students who did not finish the paragraph were able to infer enough to correctly respond to the questions. 52 Figure 18: Distribution of ORF scores for students reading with at least 80% Comprehension Table 29 shows that the fluency rate of 75 and more cwpm has more students meeting rather than not meeting the benchmark. Policymakers may decide on an acceptable range (e.g., 75-80 cwpm) for the fluency benchmark. If the benchmark for reading comprehension was lowered to 60%, (from the international benchmark of 80%) then the fluency scores would also be lowered to 62 cwpm as shown in Table 28 above. Table 29: Distribution of ORF Scores for students meeting and not meeting the 80% benchmark Correct number of words per % meeting 80% % not meeting 80% RC minute (ORF) RC benchmark benchmark 45-49 4 96 50-54 46 54 55-59 43 57 60-64 35 65 65-69 53 47 70-74 50 50 75 and more 61 39 In conclusion, greater ORF fluency is associated with higher levels of reading comprehension. Only 12% of students were identified as being able to comprehend 80% of what they read. More females than males achieved the benchmark level of comprehension. ORF scores for students achieving the benchmark shows great variability. The fluency rate with the highest proportion of students meeting the comprehension benchmark was 75 cwpm. Stakeholders should discuss and decide on the number of correct words per minute that could define a student as a fluent reader. The decision should be based on the distribution of scores for students who reached the reading comprehension benchmark. 53 Chapter 6 / SSME Findings and Analysis of Factors Associated with Reading Fluency This section reports the results of regression analyses conducted to explore the effect of student and teacher factors (independent variables) on oral reading fluency scores (dependent variable). The factors are organized into eight themes related to the research questions: 1) teacher training; 2) instructional materials and professional development; 3) print rich environment; 4) time on task; 5) teacher tongue; 6) student tongue; 7) reading assessment; 8) home environment. The results of the analysis directly respond to research questions categorized under each theme. Although the main focus is on the correlation between the above factors and ORF, Tables 50 – 55 in Annex C detail the factors associated with statistically significant increases in reading comprehension performance. Teacher Training This theme investigates teachers’ level of training on specific skills for teaching reading. Specific information requested within this theme as indicators of teacher quality include practices around the percentage of teachers who received preservice training in early grade literacy and teaching English as a second language as well as the number of teachers who received training within the last two years of teaching. Table 30 shows the results for the key indicators of the theme of teacher training. These results are based on what the teachers reported. Figures indicate that the percentage of teachers who received training was well below 50%. Only 11% of the teachers said they received training in the past year and 14% within the last two years and 5% of them said they received training in both early grade literacy and teaching English as a second language. Table 30: Theme 1 - Teacher training Key Indicators Indicator Percent During pre-service - received training in early grade literacy 25% During pre-service - received training on teaching English as a Second Language 23% During pre-service - received trainings in EG literacy and on teaching ESL 5% Received training within the past year 11% Received training two years ago 14% Instructional Materials 54 It is also fundamental that there is sufficient support materials for the teachers and students to use in the classroom for teaching and learning of reading. Based on the teacher and head teacher questionnaires, Table 31 shows the percentage of teachers with the relevant resources as well as ongoing professional development support which are the key indicators for the theme of instructional materials and professional development. Overall, less than 50% of teachers are meeting the majority of the indicators. The teachers therefore are not well supported in terms of the availability of instructional materials and the opportunity to attend the required professional development programs. Data shows that less than half of the teachers had teaching guides (32%) which is one of the key documents for any teacher to have. One interesting finding is that according to the head teachers, 50% of them provide teaching and learning materials and only 33% of teachers have sufficient materials for teaching reading. Only 28% of head teachers said they equip their schools with reading corners and libraries. In terms of professional development, 27% of the teachers reported they attended training on early grade reading. For head teachers, only 13% said they had been trained on teaching English as a second language, 33% said they received training on early grade literacy and 18% reported been trained on both. Less than 50% of teachers (33%) said they received classroom observation and feedback to inform the implementation of their reading activities. Based on the teachers’ responses, support from the head teachers (21%) and the Ministry of Education (18%) in terms of classroom observation and coaching seems insufficient. Table 31: Theme 2 - Instructional materials and Professional Development Key Indicators Indicator Percent Teachers with teaching guide shown (Yrs 1-3) 32% Teachers with curriculum statement shown 72% Teachers with sufficient teaching materials for teaching reading 33% Have student English textbooks and resources correspond with teacher’s manual 60% Teachers attending in-service training for early grade reading 27% Teachers receiving classroom observation and feedback on implementing reading activities 33% Teachers receiving classroom observation and coaching from Head teacher 21% Teachers receiving classroom observation and coaching from MEHRD 18% Head teachers trained in EG literacy 33% Head teachers trained on teaching ESL 13% Head teachers trained on both EG literacy and ESL 18% Head teachers who provide feedback and coaching in classroom 63% Head teachers who assist with lesson planning 63% Head teachers who organise training workshops 48% Head teachers who send teachers to in-service trainings 35% Head teachers who provide teaching and learning materials 50% Head teachers who equip schools with libraries/reading corners 28% Head teachers who manage school funds to support literacy strategy 33% Print Rich Environment 55 This theme offers a sense of what physical learning materials and supplies are available within classrooms for instruction, including reading books, textbooks, charts, journals, student work displays, student profiles, posters and other relevant print resources. The key indicators for this theme therefore include displays of recent students’ work, posters and charts, student profiles, sufficient space for organizing reading activities, reading corners/libraries, textbooks per student, instructional readers and materials and big books. Table 32 presents the percentage of teachers meeting the key indicators of a print rich classroom environment that enhances reading. Statistics indicate that more than half of the classrooms observed had a variety of teaching aids including readers, big books, games etc. A high percentage of classrooms had readers and big books (85% and 83% respectively), 73% had recent students work displayed and 54% had charts with phonics. However, less than 50% of classrooms had posters with songs, and spelling/vocab and only 14% kept student profiles. Overall, the majority of classrooms observed had sufficient space for reading activities but not fully utilized. Table 32: Theme 3 - Print Rich Environment Key Indicators Indicator Percent Teaching aids/readers 85% Teaching aids/big books 83% Teaching aids/posters/charts with poems 53% Teaching aids/posters/charts with songs 36% Teaching aids/posters/charts with phonics 54% Classrooms with recent students’ work displayed 73% Classrooms with spelling/vocabs written on posters/charts 46% Classrooms with student profiles 14% Classrooms with sufficient space for organizing reading activities 62% To answer the question of whether there are sufficient reading and teaching materials to support learning, the data in Table 33 shows that the classrooms are not adequately resourced. Out of all the schools visited, 51% had libraries and 41% had reading corners. Only 8% of the classrooms observed had more than 20 student textbooks, 1% had 11-20 textbooks and 37% had 1-5 copies of student textbooks. Only 2% of the classrooms had 11 or more instructional readers, 17% had 20 or more instructional materials and 15% had none at all. Classrooms also lacked big books, posters and charts. It is interesting to note that only 41% of classrooms had reading corners and yet, 62% of classrooms had adequate space for reading activities (refer to Table 28). Table 33: Theme 3 - Print Rich Environment Key Indicators Indicator Percent Number of student textbooks 1-5 37% Number of student textbooks 11-20 1% Number of student textbooks more than 20 8% Classrooms with reading corner 41% Schools with libraries 51% 56 Classrooms with 11 or more instructional readers 2% Classrooms with 20 or more instructional materials 17% Classrooms with no instructional materials 15% Classrooms with 11 or more Big Books, posters or charts 6% Classrooms with 20 or more Big Books, posters or charts 2% Classrooms with no Big Books 17% Time On Task Even when good teaching techniques are applied, students cannot succeed if they are not given sufficient learning time at school. Time-on-task is therefore an important indicator in determining school effectiveness. Time-on-task in the classroom includes such teacher activities as verbal instruction and leading a discussion or group activity. Students are spending time on-task when they are reading aloud or silently, engaging in a discussion or debate, practicing a skill, and doing work in their books. The key indicator for this theme is the amount of time spent on reading instruction and reading. In response to the research question of whether teachers allocate enough time on reading activities, half of the teachers reported that they spend at least 26-35 minutes on reading instruction and reading, which is the recommended timeframe. For the rest of the teachers surveyed, 18% said they spend 16-25 minutes on reading instruction and reading, 19% spend 11-15 minutes and 10% do it for 5-10 minutes. Table 34: Theme 4 - Time on Task Indicator Percent Classroom with 5-10 minutes of reading instruction and reading 10% Classroom with 11-15 minutes of reading instruction and reading 19% Classroom with 16-25 minutes of reading instruction and reading 18% Classroom with 26-35 minutes of reading instruction and reading 50% Language and Instruction for Teachers This theme explores the level of basic reading and comprehension skills of teachers in the language of instruction. The measures for this theme included the number of teachers who were able to read grade level passages in English fluently and with comprehension, able to write stories in English, teach in English and those that switch between English and Pijin while teaching. Teachers were asked to rate their abilities in English if they were to carry out a number of reading tasks. Data showed that overall, the majority of teachers said they were good in speaking English (74%) which is the language of instruction but only 4% said they were fluent speakers and 8% reported they were able to read and write in English fluently (refer to Table 35). The majority of teachers (86%) stated that they teach in Pijin, 78% reported they teach in English and quite a significant number (88%) said they switched between Pijin and English in class. Code switching is not always detrimental to proficiency in the 57 learning of a language, but may be considered as a useful strategy in classroom interaction, if the aim is to make meaning clear. However, this practice should be minimal to ensure that teaching and learning of the target language English is given the prominence it requires. Alarmingly, only 11% of the teachers said they were able to read grade level English passages fluently and with comprehension and only 5% of them said they could write stories in perfect English. Table 35: Theme 5 - Language and Instruction: English Reading Instruction for Teachers Indicator Percent Teachers fluent in English 4% Teachers good in English 74% Teachers able to read and write in English fluently 8% Teachers able to read and write in good English 78% Teachers able to read grade-level English passages fluently and with comprehension 11% Teachers able to write stories in excellent English 5% Teachers teaching in English 78% Teachers teaching in Pijin 86% Teachers who switch between Pijin and English in the class 88% Language and Instruction for Students Teachers were also asked to rate their students abilities when completing different tasks per year level in English. Based on the overall teacher assessments of their students’ abilities from their classroom activities, only 1% of the students could understand English fluently but the majority (42%) had a good level of understanding. According to the teachers’ ratings of their students’ abilities to read a grade level English passage, only 5% could read fluently and 15% were very good. Teachers reported that 20% of their students are able to recognise and name English letters correctly. Table 36: Theme 6 - Language and Instruction: English Reading Instruction for Students Indicator Percent Students understanding in English (fluent) 1% Students understanding in English (good) 42% Read grade-level English passages (excellent) 5% Read grade-level English passages (very good) 15% Recognize and say English letter names (excellent) 20% Classroom Reading Assessment 58 This theme focuses on information about the conduct of formative reading assessments in the classrooms and whether assessment information is used to inform changes to teaching practice. The key indicators for this theme is the percentage of teachers who conduct formative reading assessment, modify instruction based on assessment information and those who have received training or tools for reading assessments. More than half of the teachers (62%) reported they conduct formative reading assessments and 67% said they use the findings to inform teaching methodology. The percentage of teachers who received training on reading assessments is 13% but only 7% also received the relevant tools for conducting assessment. Table 37: Theme 7 - Reading Assessment Indicator Percent Teachers who conduct formative reading assessment 62% Teachers who modify instruction 67% Teachers who have received training only 13% Teachers who have received training and tools 7% To help students develop reading skills is a responsibility shared by the family and the school. Students' exposure to various reading materials at home and family support for students' literacy efforts plays a critical role in students' growth as readers. Given the importance of the home for literacy development, students were asked a number of key questions about their home environment. Student Home Environment Statistics show that a significant percentage of students do not have a supportive home environment. Majority of the students (66%) read independently at home and 67% are read to. Less than 50% receive help with their homework from either parent and only 25% have books or other reading materials to read at home. A small percentage of students (13%) have access to computer and mobile phones. Table 38: Theme 8 - Home Environment Indicator Percent Students who receive help with homework from mother 33% Students who receive help with homework from father 34% Students who read to someone at home 42% Students who read independently at home 66% Students who are read to at home 67% Students who have books or other reading materials to read at home 25% Students who have access to a computer or mobile device 13% 59 Numerous studies have demonstrated the benefits of increasing students' exposure to literacy materials in their homes, especially for lower-achieving students21. Students were asked about the presence of reading books, computer and mobile devices in their homes that they can access and use for reading. As shown in Table 32, only 25.3% of students have books or other reading materials at home and 10.4% have access to a computer or mobile device. 6.1 Association of student characteristics to student reading performance General background information and reading activities were collected from the student questionnaire. The factors as shown in Table 38 include information on whether the student attended preschool, ate breakfast before arriving at the school, language spoken at home, family literacy, whether students receive help with homework, availability of reading materials and whether students read or are read to at home. Of the students surveyed, 60% attended preschool before Year 1. Most of the students (85.2%) speak vernacular at home, 32.9% speak Pijin and 3.5% use English. A lot of the students (75.8%) have someone who can read in their homes but only 25.2% have books, newspapers or other materials to read at home in Pijin and 23.7% have these available in English in their homes. Data also shows that a high percentage of students (88.5%) eat breakfast in the morning before going to school. Results also show that 69.6% of students receive help with their homework, 67.2% have someone to read to them at home and 65.8% read to themselves. Overall, 84.9% of students enjoy reading. Table 39: Student background characteristics Student Characteristics % of cases SE* N Student attends preschool before Year 1 60.0% 0.3 1159 Student speaks English at home 3.5% 0.01 1159 Student speaks Pijin at home 32.9% 0.01 1159 Student speaks vernacular at home 85.2% 0.01 1159 Student eats before arriving to school 88.5% 0.3 1159 Student has someone who can read at home 75.8% 0.2 1159 Student receives help with homework 69.6% 6.2 1159 Someone asks student about what he/she did in school 52.9% 0.4 1159 Student tells someone at home when he/she gets good marks 65.8% 0.4 1159 Student has books, newspapers or other things to read at home 25.2% - 1159 Student has books, newspapers or other things to read at home in English 23.7% 0.01 1159 Someone reads to student at home 67.2% 0.01 1159 Student reads aloud to someone at home 41.8% 5.2 1159 21 Goldenbery et al. 1992; Koskinen et al. 1995 60 Student reads to himself/herself at home 65.8% 6.1 1159 Student reads on a computer or mobile device at home 6.0% 9.4 1159 Student likes to read 84.9% 0.8 1159 A number of factors are associated with high levels of ORF as the following findings in Table 40 indicate. The factors that were statistically significant were whether the students had books, newspapers or other reading materials in Pijin to read at home (an increase of an average of 15 cwpm), whether students read aloud to someone at home (an increase of an average of 12 cwpm), and whether they read to themselves at home (an increase of an average of 11 cwpm). Other statistically significant factors that were associated with improved ORF scores by an average of 6 to 9 cwpm included students that speak Pijin or vernacular at home, those with literate brothers, students who receive help with their homework from their fathers, and students with books, newspapers or other English reading materials at home. The factor that had a negative association with ORF mean scores (11 less words) was whether someone reads to students at home. This however was not statistically significant. An interesting finding is that there was no correlation between students who read on a computer or mobile device at home and ORF scores. Table 40: Association of student characteristics to Oral Reading Fluency (ORF) scores Student Characteristics Change in ORF SE Score (+/-) Student attends preschool before Year 1 0.06 0.09 Student eats breakfast before arriving to school 0.08 0.08 Student speaks English at home 2.57 6.68 Student speaks Pijin at home 9.59* 2.00 Student speaks vernacular at home 9.39* 2.93 Student has a literate mother 3.21 2.25 Student has a literate father 1.62 2.27 Student has a literate sister -4.29 2.01 Student has a literate brother 6.25* 1.96 Student receives help with homework from the mother -0.30 2.04 Student receives help with homework from father 7.20* 1.62 Student receives help with homework from sister 3.57 1.78 Student receives help with homework from brother -0.26 2.19 Someone asks student about what he/she did in school -0.05 0.06 Student tells someone at home when he/she gets good marks 0.01 0.06 Student has books, newspapers or other things to read at home 0.01 0.07 Student has books, newspapers or other things to read at home in English 6.19* 2.10 Student has books, newspapers or other things to read at home in Pijin 15.34* 2.48 Student has books, newspapers or other things to read at home in English 1.46 2.64 Student reads aloud to someone at home 12.10* 1.82 Student reads to himself/herself at home 10.99* 1.96 Someone reads to student at home -11.61 2.06 Student reads on a computer or mobile device at home -0.001 0.003 Student likes to read -0.10 0.06 61 The statistically significant factors associated with high levels of ORF are the ones in bold and marked with an asterisk. 6.2 Association of teacher characteristics to student reading performance Research shows that effective teachers are the most important factor contributing to student achievement. Although curricula, reduced class size, district funding, family and community involvement all contribute to school improvement and student achievement, the most influential factor is the teacher. The association between teachers’ characteristics and student reading performance was explored using data collected in the teacher interview questionnaire. The profiles of teachers surveyed are presented in Table 41. Teachers have an average of 13.6 years of experience and average age is 38. Figures show that most of their experience is on the island they are currently teaching. Of the 114 participating teachers, 83% are female and 40% have a primary teaching certificate. Almost half (46%) of them have a reading corner in their classroom and 61% have met with their students’ parents. The majority of the teachers were absent from the classrooms in the last term but most of them live close by the school. Table 41: Profiles of Teachers in SIEGRA Teacher characteristics % of cases N Female 83% 114 Has a primary teaching certificate 40% 114 Has a reading corner in the classroom 46% 114 Has not been absent from school in the last term 25% 114 Has met with parents of his/her students 61% 114 Teacher demographics Mean (sd) N Age of the teacher 38.0 (9.1) 114 Number of years of teaching experience 13.6 (9.7) 114 Number of years spent teaching on the island 8.6 (8.0) 114 Number of minutes from home to school 15.5 (16.4) 114 In order to identify the teacher characteristics associated with better student reading outcomes, a separate regression analysis was conducted for each teacher characteristic (see Table 42). The dependent variable was the average number of words read per minute in the oral reading passage (scores from subtest 6) and the independent variables were the teacher characteristics described above (Table 41). The results in Table 42 indicate that most factors did not have statistically significant associations with ORF scores except for whether the teacher has met with the student’s parents which showed an increase of an average of 11 cwpm. Having a reading corner in the classroom and not being absent from school in the last term seemed to have a positive association with ORF scores but it was not statistically significant. 62 Table 42: Association of teachers' characteristics to Oral Reading Fluency Score (ORF) Teacher Characteristics Change in ORF Score SE (+/-) Has a primary teaching certificate -4.01 4.57 Has a reading corner in the classroom 5.56 4.40 Has not been absent from school in the last term 5.63 4.90 Has met with the parents of his/her students 11.35* 4.72 Age of the teacher 0.47 0.51 Number of year of experience in teaching -0.69 0.49 Number of minute from home to school 0.03 0.13 The statistically significant factors associated with high levels of ORF are the ones in bold and marked with an asterisk. 6.3 Association of teacher training and teaching guides to student reading performance Information was also collected to determine the relationship between teaching resources and student reading performance. Teachers were asked if they had a teaching syllabus and also whether they had received any training on how to teach reading in the last three years. As shown in Table 42, 79% of teachers had a syllabus for teaching reading and only 35% had a teacher guide. Less than 50% of teachers have received training on reading instruction in the last three years. Table 43: Teacher Training and Guides Teacher Training and Guides % of cases N Teacher has a syllabus 79% 114 Teacher has a teacher guide 35% 114 Teacher have receive training on how to teach reading in the last 38% 114 three years Results of the regression analysis for teaching resources and its association with student reading performance, measured by a change in ORF score are presented in Table 44 Overall, statistics show positive impacts but insignificant. Teachers that have indicated having specific training for teaching reading in the past three years seemed to have a positive effect on student’s reading fluency performance but its statistically insignificant. Table 44: Association of training and guides to student oral reading fluency (ORF) scores Change in ORF score SE Teacher has Curriculum Statement for teaching English 4.00 4.59 Teacher have receive training on how to teach reading in the last 5.53 4.63 three years 63 6.4 Association of Classroom Environment to student reading performance Information on classroom environment especially the types of reading resources available was collected through classroom observations. Assessors recorded whether they observed the following classroom displays: spelling/vocabulary, charts/posters, songs/hymns on blackboard, charts and posters, students work, space for reading activities, reading corner, student profiles. Classrooms observed had an average of 2.13 displays. The observation also covered the types of printed materials used in instruction, such as newspapers, magazines, flashcards, food wrappers and packaging, prepaid cards, objects in treasure boxes and any other materials. There was an average of 1.3 printed materials used in the classroom. Table 45: Average Number of Classroom Displays and Materials Observed Classroom environment Mean (sd) SE N Classroom displays 2.13 (1.02) 0.10 95 Print materials used in instruction 1.3 (1.49) 0.15 95 As shown in Table 45, 46% of classrooms had spelling/vocabulary words displayed, only 6% had songs/hymns/stories written on the blackboard, 38% had them written on charts or posters, and 73% had student work displayed. Additional factors related to the classroom environment that were observed included information on whether there was space for organizing group activities, whether there was a reading corner in the classroom and whether teachers maintained folders with students’ work (student profiles). The results show that less than 50% of classrooms had a reading corner (41%), 62% had sufficient space for organizing group work, 14% kept student profiles and 73% displayed students work. Table 46: Frequency and Type of Classroom Displays/Resources Available Types of classroom displays/resources available % of classes SE N Spelling/vocabulary displayed on charts/posters 46% 0.05 95 Song/hymns displayed on blackboard 6% 0.02 95 Song/hymns displayed on charts/posters 38% 0.05 95 Student work displayed 73% 0.05 95 Sufficient classroom space for organized group activities 62% 0.05 95 Reading corner in the classroom 41% 0.05 95 Student profiles (folder with student work and student info) 14% 0.03 95 The classroom environment variable that had the most positive relation with ORF scores was the reading corner in the classroom. The students which have access to reading corners in their classrooms read an average of 11 more words per minute. Another factor with a positive relation with ORF scores is student profiles with an average of 9 more words per minute but it is statistically insignificant. All other variables also had no statistically significant association with students reading fluency. 64 Table 47: Association of Classroom Environment to Student ORF Scores Classroom Environment Change in ORF Score SE (+/-) Classroom displays 1.04 2.71 Spelling/vocabulary displayed -3.74 3.67 Song/hymns/stories displayed on blackboard -1.59 10.22 Song/hymns/stories displayed on charts/posters 4.15 5.67 Student work displayed 3.31 6.03 Print materials used in instruction 0.22 1.88 Sufficient classroom space for organized group activities -6.84 5.16 Reading corner in the classroom 11.42* 4.79 Student profiles (folder with student work and student info) 9.44 7.19 The statistically significant factors associated with high levels of ORF are the ones in bold and marked with an asterisk. 6.5 Association of Teacher Instructional and Assessment Methods to Student Performance The final set of regression analysis examined the relationship between instructional and assessment methods and student performance in ORF. Table 47 shows the results of regression analysis of the frequency in which students and teachers performed sixteen instructional and assessment methods within the course of the week. The dependent variable is the mean ORF score of the students and the independent variables are the instructional methods. The mean ORF score of students who were never exposed to an instructional or assessment method was compared to the mean ORF score of students who were exposed daily, 3-4 days, 1-2 days, not in the last 5 days and never. Results from those regression analyses were examined to ascertain if being exposed to a specific method has a positive or negative relation with ORF scores and if so, whether the effect is the same for all rates of exposure (e.g., 1-2 days, daily, etc.). All regression models include gender and year for controlling for those two characteristics. Statistics presented below also show the teaching methods used during reading instruction and the frequency in which each strategy was conducted. The most frequently used instructional method was reading comprehension. The most common classroom activities were shared reading, students spelling words in their exercise books and students writing sentences. 65 Table 48: Frequency of methods used during reading instruction Never Rarely Sometimes Often Daily N (Not in the 1 or 2 days 3-4 last 5 days) days Asking students to practice Letter 2% 6% 51% 8% 33% 113 Name Asking students to orally retell a 1% 3% 64% 12% 20% 113 story that they have read Asking students to practice letter 3% 3% 50% 16% 29% 113 sounds Asking students to sound out 11% 4% 54% 13% 18% 113 unfamiliar words using knowledge of letter sounds Asking students to learn meanings 0% 1% 50% 14% 35% 113 of new words/vocabulary Shared reading 3% 2% 22% 11% 62% 113 Group-guided reading 4% 4% 40% 17% 35% 113 Listening to a child read aloud one 10% 9% 45% 11% 25% 113 on one Students reading on their own 3% 3% 52% 10% 32% 113 silently Reading comprehension activities 3% 3% 45% 9% 40% 113 Children take books home to read 35% 4% 43% 6% 12% 113 with their parents Evaluating student’s oral reading 9% 10% 56% 8% 17% 113 Teachers work on word building 7% 6% 58% 8% 20% 113 with students Asking students to read and draw 4% 3% 56% 7% 30% 113 Students working on spelling words 1% 2% 42% 8% 46% 113 in exercise books Students writing sentences 3% 3% 37% 12% 45% 113 Results of the regression analysis presented in Table 49 below show that some instructional methods used do positively correlate with students’ ORF scores. Students taking books home to read with their parents on a daily basis seemed to have the most positive relationship with ORF scores with an average of 18.66 more words per minute. However, several teaching methods showed a negative association with students’ ORF scores. Dailyshared reading recorded an average of 13.94 less words and when this activity is done regularly (3-4) days, students scored an average of 21.36 less words. As well, when teachers work regularly on word building with students, results show that students had lower ORF scores of 18.57 words less. 66 Table 49: Association of teacher instructional and assessment methods to student performance Teaching instructional and assessment methods Change in ORF score SE 65.Teaching Letter Names Never 12.86 16.69 Rarely 7.76 9.28 Often -3.12 8.31 Daily -1.88 4.92 66.Asking children to orally retell a story that they have read Never 29.18 23.12 Rarely 2.88 13.53 Often 1.16 6.71 Daily 9.03 5.51 67.Teaching new letter sounds Never 4.46 13.82 Rarely 1.43 13.82 Often -3.10 6.32 Daily -3.41 5.17 68.Asking children to sound out unfamiliar words using knowledge of letter sounds Never 0.41 7.30 Rarely 10.04 10.75 Often -6.00 6.66 Daily -5.14 6.07 69.Teaching meaning of new vocabulary words Rarely -24.22 23.34 Often -398 6.55 Daily -0.53 4.85 70.Shared reading Never 22.42 13.24 Rarely -12.39 15.92 Often -21.36* 7.41 Daily -13.94* 5.06 71.Group Guided reading Never -2.09 10.89 Rarely 4.68 12.06 Often -9.28 6.32 Daily -4.07 5.06 Q72. Listening to a child read aloud Never 2.84 7.96 Rarely -8.55 7.96 Often -7.34 7.15 Daily 3.66 5.41 Q73.Students readings on their own silently 67 Never -19.21 13.53 Rarely -22.71 13.53 Often -4.87 7.25 Daily 0.49 4.85 Q74.Reading comprehension activities Never -0.85 13.86 Rarely 0.25 13.86 Often 1.78 8.07 Daily -3.00 4.77 Q75.Children take books home to read with their parents Never -1.41 4.80 Rarely -3.80 11.64 Often 10.96 9.05 Daily 18.66* 6.98 Q76.Evaluating student’s oral reading Never -12.78 7.72 Rarely 1.11 7.15 Often -3.02 8.08 Daily 10.07 5.94 77.Teacher works on word building with students Never -13.80 8.47 Rarely -5.63 8.99 Often -18.57* 8.04 Daily -0.31 5.48 78.Students read and draw Never 0.89 10.75 Rarely 5.32 13.67 Often -12.74 8.69 Daily -2.20 4.94 79.Students work on spelling words in exercise books Never -20.85 23.44 Rarely -0.25 16.75 Often -8.15 8.07 Daily -2.80 4.67 80.Students writing sentences Never -7.58 12.02 Rarely -24.66 13.73 Often -5.62 7.29 Daily -0.66 4.81 The statistically significant factors associated with high levels of ORF are the ones in bold and marked with an asterisk. 68 Chapter 7 / Conclusions and Next Steps SIEGRA was administered in the Solomon Islands to provide evidence to the MEHRD on the current situation of early grade reading abilities to elicit critical dialogue among stakeholders that would effectively inform the direction and next steps to improve reading in the early grades. Results indicate that while students have good automatic letter recognition, initial sounds, letter sounds and vocabulary skills in Pijin, they struggle with decoding familiar and unfamiliar words and identifying vocabulary words in English. As a result, the majority of students are unable to read fluently with comprehension in English. Data showed an overall fluency rate of 21 cwpm which is well below the international standard of 45-60 words per minute although Year 3 students seem to be making good progress to becoming fluent readers with a mean score of 36 cwpm. This reading passage for this subtest had vocabulary taken from their classroom textbooks and so the words should have been familiar to them. These results indicate that students are generally not learning to read in their early years of schooling. Almost all of students in Year 1 and a little over half of Year 2 have zero reading comprehension skills. In Year 3, comprehension improves where less than half of the students scored zero. However, the majority of students in Years 1-3 are reading below the 80% comprehension benchmark. Overall, 12% met the benchmark. In Year 3, about 76 % of students did not meet the benchmark. Across all sub-tests, girls performed better than boys. More females than males met the 80% reading comprehension benchmark. Students in Honiara and Western regions consistently performed better than students from the other 4 areas. In terms of results by year, there is evidence of learning progression especially between Years 1 and 2. Despite this progress in learning between the years, it is not at the rate the students need to enable them to become fluent readers or read with comprehension. The analysis identified a number of student characteristics, instructional methods and resources that demonstrated significant positive correlations with reading performance. Students who had books, newspapers and other reading materials to read at home in Pijin, communicate in Pijin or vernacular at home, receive assistance with homework, read to someone at home or by themselves scored an average of 6-15 more words. As well, students who take books home on a daily basis to read with their parents was found to be a critical activity for increasing students’ fluency rates. As expected, the availability of reading materials in the classroom especially through reading corners or the use of these materials in teaching instruction was positively related to ORF scores. These findings 69 suggest that certain aspects of school management, instruction and support from the community are important factors to ensuring early reading achievements amongst students. Further research could explore the types of materials that contribute most to improved reading outcomes. Shared reading and teachers working with the students on word building exercises were negatively related to student ORF scores. One reason could be that teachers are not closely following the teaching guide and results show that only 35% of the teachers surveyed had teacher guides. Overemphasizing decoding skills or assessment and not utilizing results to inform instruction can affect student performance. Although the language of instruction is English, 86% of teachers were teaching in Pijin and 88% switch between the two languages. What is even more alarming is that only 11% of the teachers were able to read grade level English passages fluently and with comprehension and 5% could write stories well in English. According to the teachers’ ratings of their students’ abilities to read a grade level English passage, only 5% could read fluently and 15% were very good. It is also interesting to note that for the vocabulary subtask, students recognised items better in Pijin (87%) than in English (54%). It is therefore not surprising that the students are not fluent in English since the teachers are in the same situation. These results suggest that adequate language support in the English language is lacking for students as well as teachers and could inform developments in the language of instruction policy as well as efforts to develop students’ vocabulary skills and teachers reading and writing abilities in English. Overall, these findings show that students’ limited mastery of decoding skills must be addressed to improve their oral reading fluency and reading comprehension. It is important to note that learning is in English which is the students’ second language and therefore delays in mastering reading skills are expected. However, there are also other contributing factors to this delay in learning such as the quality of reading instruction and the availability and appropriateness of resources used. Whatever the reasons are for the delay in learning, one of the main goals of any education system is for students to be reading with comprehension in the early years of schooling. Recommendations Based on the findings, the following recommendations are presented for consideration by MEHRD, as means to improve the quality of early grade reading instruction for Solomon Islands students: 1. Teaching and Learning 1a. Teachers should provide explicit instruction in reading comprehension skills for students in early grades. Given that only 12% of students were able to comprehend 80% or more of grade level text, teachers should focus on improving reading comprehension levels for all years. Research 70 shows that teachers are often the ones to ask questions orally and that students are expected to respond, but rarely are students asked to develop their own questions as they read (think aloud) or to make predictions and then check their predictions. Students need to be empowered to make meaning from text and to use an inquiry-based approach to reading. Teachers could ask students to make predictions before, during and after reading, to retell stories, or to identify problems within the story and provide possible solutions. 1b. Provide remedial instruction for non-readers. An analysis of zero scores revealed that 62% of students in Year 1, 37% in Year 2 and 22% in Year 3 could not correctly read a single word in the reading passage. Given this significant number of zero scores, it is important for the MEHRD to conduct classroom level assessments to identify non-readers in Years 1-3, diagnose the causes, and design targeted intervention strategies to address deficiencies. For instance, teachers may group students according to ability and provide remedial activities and appropriately levelled text. This “catch-up approach� is being used by UNICEF in Zambia using J-Pal’s research in India, which demonstrated that students grouped by ability is more effective than mixed- ability grouping. UNICEF will assess students in all three grades and group them according to reading abilities (non-readers, those who can read letter sounds, syllables, words, passages, etc.). Grouping students by reading ability rather than grades has produced dramatic results in India, Kenya and Ghana. Teachers and school administrators should further determine whether non-readers have learning disabilities (e.g., dyslexia) and design relevant intervention strategies for special needs students. 1c. Develop and implement activities that specifically focus on raising boys’ performance and interest in reading. The results illustrate that boys consistently performed lower than their female counterparts across the subtests. There may be cultural or gender barriers that affect boys’ interest and engagement in reading activities. In addition to designing strategies to address low reading abilities of boys and girls, stakeholders at all levels should discuss the potential challenges specific to boys and design strategies to improve boys’ reading achievement. Successful strategies that have worked in other countries include developing gender-sensitive materials that attract boys’ attention (such as sports, science fiction, fantasy, comic books, digital text, and stories that are humorous) and integrating reading into extracurricular activities (e.g., sports, health clubs). 2. Teacher Training 2a. Train Years 1-3 teachers in reading instruction and assessment with a focus on decoding skills and reading comprehension. Only 5% of teachers surveyed received pre-service training on early grade literacy and teaching English as a second language. Furthermore, only 11% of the teachers were trained on this in the last year and 14% in the last two years. With regards to reading assessment, only 13% of the teachers received training and 7% of this number received the relevant tools for conducting assessment. SIEGRA results indicate that students are particularly 71 weak in decoding unfamiliar words, and have low reading comprehension skills. Students also have very low writing skills as shown in the dictation orthography and dictation convention subtest results. Therefore, teachers need to be trained on early grade reading instruction and assessment. Training should concentrate on providing teachers with instructional strategies for building foundational reading skills such as phonics, decoding and comprehension. Additionally, training should incorporate curriculum expectations, highlight writing and its relationship with reading, and provide teachers with the specific approaches, classroom activities and assessment methods to achieve results. 2b. Ensure that pre-service course content provides new teachers with essential knowledge and skills related to improving reading and literacy outcomes. One of the main objectives of primary teacher education courses should be to equip pre- service teachers with the skills to teach and support emergent skills of students. Pre-service training programs therefore need to assist teacher trainees to understand and be able to use various strategies to develop foundational literacy skills for early grade students. Emphasis in pre-service should include training future teachers on how to deliver explicit teaching about phonemic awareness, alphabetic principle/phonics, vocabulary, reading fluency and comprehension and writing. There should also be opportunities for teacher trainees to observe and engage with effective reading teaching practices before they are sent out into the field of teaching. 3. Reading Materials 3a. Ensure that teachers have adequate and appropriate instructional materials for quality instruction. The development of reading skills in any language depends on many factors and among them is the availability and use of appropriate instructional materials. The findings show that only 32% of teachers had teaching guides and 28% did not have curriculum statements. As well, only 33% reported to have sufficient materials for teaching reading. The curriculum statement is a must-have tool in the classroom for every teacher as it outlines the philosophy, goals, subject content, learning standards, teaching strategies and teaching resources for any area of learning. An equally important tool is the teaching guide which serves as an important source of support in focusing early grades reading instruction on what students need to know and be able to do based on the literacy goals and other learning outcomes in the prescribed curriculum. These important teaching and learning tools together with other reading instructional materials provide strong support for the effective teaching and learning of reading. In light of the above findings, it is recommended that the teachers are provided with adequate and varied instructional materials for teaching and learning of English reading skills in primary schools. 3b. Ensure that classrooms have libraries/reading corners with a variety of reading materials and that these are used during reading instruction. 72 Reading corners in the classroom was a statistically significant factor associated with improved ORF average scores of 11 cwpm. Results show that 41% of classrooms had reading corners and only 51% of the schools had libraries. The majority of the classrooms (37%) had 1-5 copies of student textbooks, only 2% had 11 or more instructional readers and 15% had none at all. Classrooms are therefore not well equipped with the necessary reading resources to support reading instruction. Thus, it is recommended to develop reading corners in the classrooms with a variety of books (fiction and non-fiction) including graded readers and challenging materials for the experienced readers. Teacher guides for reading should be available to assist teachers in effectively using these books for reading instruction. The International Reading Association (IRA) recommends that classroom libraries start with at least seven books per child and purchase two additional new books per year. The optimal number of books in a classroom library is 300-600, depending on the grade level and number of copies22. The number of books teachers should expect children to read during the school year is 100-125 picture books by the end of Grade 1 and 50-75 chapter books by the end of Grade 2. Ideally, reading materials should include a variety of narrative and expository books at different levels of difficulty and reflect the interests of the pupils. A low-cost option is to provide e-readers if the schools can make available the required equipment to support student access to such resources. E-readers allow students and teachers to choose from a variety of genres, it is portable so students can read from home or school, and it’s read aloud features provide additional support for emergent readers2324. In addition to provision of an increased number of hard and soft copy books, teachers should be trained on how to better integrate materials into their instruction and on how to develop attractive reading corners25. 3c. Encourage family involvement as an integral part of early grades reading. Analysis of the contextual information collected show that students’ characteristics had the greatest association with improved ORF scores. Students that have access to reading materials in Pijin at home is critical to increasing oral reading fluency by 15 cwpm and 6 cwpm if the materials are in English. Results also show that students who read aloud to himself/herself at home had an additional 11 cwpm. As well, students taking books home on a daily basis to read with their parents was statistically significant and associated with higher ORF scores of 19 more cwpm. It is therefore important to engage parents in reading activities to ensure that students and parents are spending the recommended time (20-40 minutes per day) reading at home. Parents should also be encouraged to have reading materials at home to support students in practicing reading outside of school. Establishing 22 Neuman, S. (undated). The importance of the classroom library. Available at: http://teacher.scholastic.com/products/paperbacks/downloads/library.pdf 23 Adams, A. & van der Gaag, J. (2011). First Step to Literacy: Getting Books in the Hands of Children Available at: https://www.brookings.edu/research/first-step-to-literacy-getting-books-in-the-hands-of-children/ 24 UNESCO (2014). Reading in the mobile era: A study of mobile reading in developing countries. http://unesdoc.unesco.org/images/0022/002274/227436E.pdf 25 Neuman, S. (undated). The importance of the classroom library. 73 partnerships with parents is a huge challenge and one that is often avoided in literacy initiatives, but it is critical. 4. Language 4a. Language of Instruction Policy. As previously mentioned, students who had access to reading materials in Pijin at home scored 15 cwpm more than their peers. As well, students speaking Pijin or vernacular at home was associated with higher ORF scores of 10 more cwpm and 9 more cwpm respectively and both factors were statistically significant. Findings also showed that while the language of instruction is English, students’ vocabulary skills are stronger in Pijin than English. Many researches argue that children most easily acquire reading skills in their mother tongue and that, with appropriate instruction, materials, and other supportive resources and effective guidance, they can successfully transfer those skills to a second language, and in this case English, resulting eventually in better achievement in both languages. The SIEGRA results is critical evidence that students are learning better in their mother tongue hence MEHRD might need to reconsider its language of instruction policy for primary schools. 4b. Support students’ language skills in their mother tongue or dominant language. As previously mentioned, results have shown that students have better vocabulary skills in Pijin (an average of 87% correct responses) than English (an average of 52% correct responses). The vocabulary subtest measures receptive language skills to better gauge students’ expressive oral language skills. The results suggest that the students need to develop vocabulary skills in English to support comprehension. It is therefore recommended that teachers place more emphasis on building students English language vocabulary. 4c. Training of teachers to improve their skills in reading and writing in English and on the principles of code switching. An alarming finding of this study was that only 11% of the teachers said they were able to read grade level English passages fluently with comprehension and only 5% reported they could write stories in perfect English. The study also found that the majority of teachers were in fact teaching in Pijin (87%) and 88% said they switch between English and Pijin in class. Given the above results, it is therefore recommended that teachers be trained on reading and writing skills in the English language if it is retained as the language of instruction. Teachers also need training on the principles of code switching to ensure that teaching and learning of the target language (English) is given the prominence it requires. 5. National Benchmarks for Reading 5a. Establish national benchmarks for reading skills in early grades 74 A benchmarking system provides critical evidence to redirect the education system as a whole to get as many students as possible to achieve the approved standards. This requires improving classroom instruction, ensuring a culture of shared accountability over learning at the school level as well as strengthening practices and support for reading at home. The findings of this study can guide the MEHRD and its stakeholders in developing specific benchmarks for reading in Years 1 to 3. Data shows that only 12% of students assessed met the international reading benchmark of 80% and for ORF, students scored 21 correct words per minute which is also below the international standard of 45-60 correct words per minute. The Ministry of Education together with its stakeholders should therefore decide on the level of comprehension that is acceptable as demonstrating full understanding of a given text. Most countries have settled on 80% or higher (4 or more correct responses out of 5 questions) as the desirable level of comprehension. With a reading comprehension benchmark confirmed, SIEGRA data can then be used to show the range of oral reading fluency (ORF) scores, measured in correct words per minute (cwpm), obtained by students to be able to achieve the desired level of comprehension. Discussion is needed to determine the value within that range that is put forward as the benchmark for ORF. SIEGRA results show that for those students who met the 80% benchmark, their reading fluency rate was 76 cwpm. This can be the starting point for MEHRD in determining the national benchmark. Alternatively, a range can indicate the levels of skill development that are acceptable as “proficient� or meeting a grade level standard (for example, 40 to 50 cwpm). The process then proceeds in the same manner for each skill area assessed in the SIEGRA subtests. Once the benchmark and targets are set, the MEHRD can then conduct a national reading campaign to inform all stakeholders of the new benchmarks, train teachers and then regularly monitor and report progress towards achieving the targets at all levels (national, regional and schools). Equally important is for MEHRD to ensure that if a benchmark system is introduced, it should include adequate mechanisms to identify struggling and non-readers in order for them to receive the necessary support to reach the grade standards before completion of a school year. 6. Additional research on findings not well understood 6a. Conduct a follow-up study on teaching practices and assessment methods to identify strengths and weaknesses. Several teaching methods showed a negative association with students’ ORF scores. Daily shared reading recorded an average of 14 less words and, when this activity is done regularly (3-4) days, students scored an average of 21 less words. As well, when teachers work regularly on word building with students, results showed that students had lower ORF scores of 19 less words. It is also interesting to note that results indicate that the most frequently used instructional method was reading comprehension and yet, students’ scores in this subtest were very low. It is therefore important to understand why these approaches have an adverse relationship. It could be because teachers were not adequately supported through pre-service and in-service training to master the new teaching methodologies or that training was not aligned with best practices in reading instruction. Additionally, teachers may not understand how to accurately assess students and utilize the results for reflection and lesson planning. A further investigation into the teaching practices and 75 assessment methods used in the classrooms will better inform professional development intervention strategies and other ongoing support in these areas for effective reading instruction. 6b. Identify the causes for the differences in performance across regions and develop context-relevant interventions. The Honiara region scored the highest across the majority of subtasks while the Choiseul region scored the lowest for 8 subtests. Analysis of the findings by region also showed that students from the Isabel area were the weakest in reading comprehension, listening comprehension and shared the bottom ranking with Choiseul for the initial sounds subtest. The difference between the highest and the lowest scores across the regions for all subtests is significant and warrants an investigation into the causes. Once the causes are identified, interventions specific to the regions’ contexts can then be developed and implemented to address the gaps. It is also recommended that an investigation be conducted to determine the factors contributing to high reading achievements in Honiara and the Western regions and to identify lessons learned and best practices that may be replicated to improve scores in other regions. 76 Annex 1 / Tables Annex 1.A: Correlations between tasks Annex 1.B: 95% Confidence interval for EGRA Task Annex 1.C: Solomon Islands Regression with Reading Comprehension as Outcome Annex A: Correlations between tasks Vocabula Letter Initial Sound Letter Famili Non- Oral Reading Listening Dictation Dictation ry Name Identification Sound ar words Readin Comprehensi Comprehensi Orthography Convention Knowled Knowled (Task 2) Knowled Words g on on (Task 9) (Task 10) ge ge (Task init_sound_p ge (Task (Task (Task Fluenc (Task 7) (Task 8) orthography_p convention_p 1) cnt 3) 4) 5) y (Task readcomp_pc listcomp_pcn cnt cnt Voc_pcnt clpm clspm cwpm cnwp 6) nt t m orf Voc 1 task Task 1 .384** 1 Task 2 .312** .403** 1 Task 3 .416** .671** .538** 1 Task 4 .406** .619** .290** .586** 1 Task 5 .409** .517** .298** .571** .751** 1 77 Task 6 .319** .603** .280** .496** .789** .679** 1 Task 7 .399** .573** .292** .509** .702** .666** .844** 1 Task 8 .517** .423** .307** .410** .393** .420** .400** .531** 1 Task 9 .268** .564** .323** .557** .620** .570** .676** .709** .499** 1 Task 1 .290** .531** .296** .517** .612** .556** .652** .694** .484** 0.896** 10 **. Correlation is significant at the 0.01 level (2-tailed). 78 Annex B: 95% Confidence interval for EGRA Task 95% Confidence Interval for mean on Vocabulary Task by Year and Gender 95% Confidence Interval Task: Vocabulary Lower bound Upper Bound (voc_pcnt) Overall 69.8 72.0 Year 1 62.7 66.8 Year 2 69.7 73.7 Year 3 73.6 77.0 Girls 69.8 73.1 Boys 68.8 71.9 95% Confidence Interval for mean at Task 1 by Year and Gender 95% Confidence Interval Task 1: Letter Name Lower bound Upper Bound (clpm) Overall 59.3 62.7 Year 1 43.1 48.3 Year 2 59.6 65.1 Year 3 70.0 75.7 Girls 62.8 67.6 Boys 54.7 59.4 95% Confidence Interval for mean at Task 2 by Year and Gender 95% Confidence Interval Task 2: Initial Sound Lower bound Upper Bound (init_sound_pcnt) Overall 78.1 81.9 Year 1 69.4 77.5 Year 2 78.6 84.8 Year 3 81.3 86.6 Girls 79.3 84.6 Boys 75.5 80.8 79 95% Confidence Interval for mean at Task 3 by Year and Gender 95% Confidence Interval Task 3: Letter Sounds Lower bound Upper Bound (clspm) Overall 38.2 41.2 Year 1 25.9 30.4 Year 2 40.80 46.4 Year 3 43.9 48.1 Girls 41.2 45.4 Boys 34.1 38.3 95% Confidence Interval for mean at Task 4 by Year and Gender 95% Confidence Interval Task 4: Familiar Word Lower bound Upper Bound (cwpm) Overall 19.0 21.9 Year 1 7.8 11.7 Year 2 16.2 20.0 Year 3 29.0 34.5 Girls 20.8 24.7 Boys 16.1 20.3 95% Confidence Interval for mean at Task 5 by Year and Gender 95% Confidence Interval Task 5: Non-word Lower bound Upper Bound (cnwpm) Overall 12.0 14.1 Year 1 6.0 8.6 Year 2 11.0 15.3 Year 3 16.3 19.4 Girls 12.8 15.4 Boys 10.5 13.6 95% Confidence Interval for mean at Task 6 by Year and Gender 95% Confidence Interval Task 6: Oral Passage Lower bound Upper Bound (orf) 80 Overall 19.6 22.9 Year 1 5.6 8.8 Year 2 16.3 20.9 Year 3 32.3 39.1 Girls 22.2 27.4 Boys 15.7 19.9 95% Confidence Interval for mean at Task 7 by Year and Gender 95% Confidence Interval Task 6: Reading Lower bound Upper Bound Comprehension (readcom_pcnt) Overall 20.0 23.6 Year 1 3.9 7.3 Year 2 19.5 25.6 Year 3 31.4 38.4 Girls 22.8 28.4 Boys 15.8 20.5 95% Confidence Interval for mean at Task 8 by Year and Gender 95% Confidence Interval Task 8: Listening Lower bound Upper Bound Comprehension (listcom_pcnt Overall 43.7 47.6 Year 1 27.4 33.7 Year 2 46.3 53.0 Year 3 51.7 57.8 Girls 44.6 50.3 Boys 41.3 46.5 81 95% Confidence Interval for mean at Task 9a by Year and Gender 95% Confidence Interval Task 9a: Dictation – Lower bound Upper Bound Orthography_pcnt Overall 34.4 39.1 Year 1 9.6 15.3 Year 2 37.2 45.1 Year 3 49.4 57.3 Girls 37.8 44.6 Boys 29.3 35.7 95% Confidence Interval for mean at Task 9b by Year and Gender 95% Confidence Interval Task 9b: Dictation – Lower bound Upper Bound Convention_pcnt Overall 28.4 32.7 Year 1 6.9 11.5 Year 2 30.2 37.7 Year 3 41.9 49.3 Girls 32.3 38.8 Boys 23.1 28.5 82 Annex C: Regression Analysis with Reading Comprehension as Outcome Table 50: Impact of student characteristics on Reading Comprehension percent score Change in Student Characteristics ORF Score (+/-) Student attends preschool before Year 1 0.09 Student eats breakfast before arriving to school 0.03 Student speaks English at home 20.06* Student has a literate mother 3.87 Student has a literate father 2.78 Student has a literate sister -1.95 Student has a literate brother 4.22 Student receives help with homework from the mother 1.43 Student receives help with homework from father 10.31* Student receives help with homework from sister 4.96* Student receives help with homework from brother 0.35 Someone asks student about what he/she did in school -0.03 Student tells someone at home when he/she gets good marks 0.03 Student has books, newspapers or other things to read at home 0.01 Student has books, newspapers or other things to read at home in English 3.16 Student reads aloud to someone at home 13.41* Student reads to himself/herself at home 15.69* Someone reads to student at home -6.51 Student reads on a computer or mobile device at home 0.01 Student likes to read -0.12 Table 51: Effect of teachers' characteristics on Reading Comprehension percent score Teacher Characteristics Change in RC Score (+/-) Has a primary teaching certificate -1.69 Has a reading corner in the classroom 1.72 Has not been absent from school in the last term 4.98 Has met with the parents of his/her students 10.05* Age of the teacher 0.55 Number of years teaching -0.66 Number of minute from home to school -0.03 Table 52: Effect of training and guides on Reading Comprehension percent score Change in RC score Teacher have a syllabus/curriculum in English -0.02 Teacher have a manual for teaching English 0.12 Teacher have receive training on how to teach reading in the last three 0.31 years 83 Table 53: Effect of Classroom Environment on Reading Comprehension percent score Classroom Environment Change in RC Score (+/-) Classroom displays 0.12 Spelling/vocabulary displayed -3.38 Song/hymns/stories displayed on blackboard -12.19 Song/hymns/stories displayed on charts/posters 4.66 Student work displayed 5.19 Print materials used in instruction 1.30 Sufficient classroom space for organized group activities -9.02* Reading corner in the classroom 5.62 Student profiles (folder with student work and student info) 0.91 Table 54: Effect of Reading Instructional Resources on Reading Comprehension percent score Language use in classroom Change in RC Score (+/-) Reading instructional materials in classroom 1.53 Table 55: Effect of teacher instructional and assessment methods on Reading Comprehension raw scores (not in percent scores) Teaching instructional and assessment methods Change in reading comp raw score 65.Teaching Letter Names Never 0.31 Rarely 0.66 Often 0.24 Daily -0.01 66.Asking children to orally retell a story that they have read Never 1.19 Rarely 0.89 Often 0.33 Daily 0.34 67.Teaching new letter sounds Never -0.06 Rarely 0.50 Often 0.12 Daily -0.02 68.Asking children to sound out unfamiliar words using knowledge of letter sounds Never -0.10 Rarely 0.67 Often 0.09 84 Daily -0.02 69.Teaching meaning of new vocabulary words Rarely -1.07 Often 0.05 Daily -0.02 70.Shared reading Never 0.19 Rarely -0.01 Often -0.47 Daily -0.33 71.Group Guided reading Never -0.26 Rarely 0.24 Often -0.16 Daily -0.15 Q72. Listening to a child read aloud Never 0.33 Rarely -0.09 Often -0.23 Daily 0.10 Q73.Students readings on their own silently Rarely -0.92 Sometimes -0.99 Often -0.21 Daily -0.07 Q74.Reading comprehension activities Never 0.84 Rarely -0.10 Often 0.48 Daily 0.08 Q75.Children take books home to read with their parents Never -0.01 Rarely 0.19 Often 0.81* Daily 0.83* Q76.Evaluating student’s oral reading with running records or any other method Never -0.67* Rarely 0.22 Often -0.17 Daily 0.28 77.Teacher works on word building with students 85 Never -0.43 Rarely -0.34 Often -0.68 Daily -0.09 78.Students read and draw Never -0.01 Rarely 0.41 Often -0.51 Daily -0.03 79.Students work on spelling words in exercise books Never -1.00 Rarely -0.35 Often -0.17 Daily -0.03 80.Students writing sentences Never -0.10 Rarely -0.75 Often -0.06 Daily 0.14 86 Annex 2 / Instruments Annex 2.A: EGRA Instrument Annex 2.B: Student Questionnaire Annex 2.C: Head Teacher Questionnaire Annex 2.D: Teacher Questionnaire Annex 2.E: Classroom Observation 87 Annex 2.A: EGRA Instrument 88 89 90 91 92 93 94 Annex 2.B: Student Questionnaire 95 96 Annex 2.C: Head Teacher Questionnaire 97 98 99 Annex 2.D: Teacher Questionnaire 100 101 102 103 104 Annex 2.E: Classroom Observation 105 106 107 Annex 3 / Test Reliability Measures Annexe Table 1 presents indicators of test reliability for the Solomon Islands EGRA. The first indicator of reliability is the “item-test� correlation which is the correlation between each sub-domain and a composite measure. The composite measure is the sum of the standardized scores of each sub-domain following the Cronbach’s Alpha methodology. The second indicator is the “item-rest� correlation which is the correlation of each sub- domain with a composite measure excluding the sub-domain. The composite measure for this second indicator is the sum of the standardized scores for all sub-domains excluding the sub-domain in question. These two indicators help identify sub-domains that are less correlated with the EGRA test as a whole in order to identify potential outlier sub-domains. Finally, Cronbach’s Alpha is calculated for the sub-domains that are not timed; these are scored as percent correct in EGRA. RTI (2009:82) does not recommend the Cronbach’s Alpha test for timed sub-domains as this may inflate the measure of test reliability26. This indicator provides an overall measure of the correlation between the non-timed sub- domains; a typical benchmark in research studies is 0.7. For the Solomon Islands EGRA, correct invented words per minute and listening comprehension stand out as being less correlated with the other domains. Their item-test correlations are 0.49 and 0.51, respectively, compared to a range of 0.65 to 0.88 for the other domains, and their item-rest correlation is 0.36 and 0.38, respectively, compared to a range 0.54 to 0.84 for the other domains. However, Cronbach’s Alpha for the non -timed sub-domains, including listening comprehension, is 0.77 and exceeds the typical reliability benchmark of 0.7. 26 RTI (2009). Early Grade Reading Assessment Toolkit. Research Triangle Park, N.C.: RTI International 108 Annex Table 1. Reliability measures Item-test Item-rest correlation correlation Correct letters per minute 0.80 0.74 Correct letter sounds per minute 0.77 0.70 Correct words per minute 0.89 0.85 Correct invented words per minute 0.83 0.78 Oral reading fluency 0.84 0.78 Initial sounds (percent correct) 0.54 0.42 Reading comprehension (percent correct) 0.84 0.78 Listening comprehension (percent correct) 0.61 0.51 Dictation (percent correct) 0.79 0.73 Cronbach's alpha (percent correct items only) 0.74 109