Report No: AUS0000657 TEGRA LEARNING GAINS REPORT FROM MIDLINE TO ENDLINE MARCH 2018 EDU 1 © 2017 The World Bank 1818 H Street NW, Washington DC 20433 Telephone: 202-473-1000; Internet: www.worldbank.org Some rights reserved This work is a product of the staff of The World Bank. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of the Executive Directors of The World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this work is subject to copyright. Because The World Bank encourages dissemination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. Attribution—Please cite the work as follows: “World Bank. 2018. PEARL TEGRA Learning Gains Report from Midline to Endline. © World Bank.� All queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@worldbank.org. 2 Table of Contents Acronyms ......................................................................................................................................... 4 Acknowledgement ........................................................................................................................... 5 Executive Summary ......................................................................................................................... 6 Summary of key findings ............................................................................................................. 6 Student Contextual Factors ....................................................................................................... 10 Conclusions and Recommendations ......................................................................................... 10 Introduction .................................................................................................................................... 12 Background ............................................................................................................................... 12 Objectives .................................................................................................................................. 13 Structure of Report .................................................................................................................... 13 Chapter 2 - TEGRA Design and Implementation .......................................................................... 14 Sample size and sampling methodology................................................................................... 14 Instrument Design ..................................................................................................................... 15 Fieldwork and Challenges Encountered ................................................................................... 15 Chapter 3- LEARNING GAINS ...................................................................................................... 16 EGRA Summary Results – Reductions in zero-scoring students ............................................. 16 EGRA Summary Results – Overall Learning Gains .................................................................. 20 Learning Gains by Subtask ....................................................................................................... 24 Chapter 4 – Student Contextual Factors ....................................................................................... 49 Household Assets ..................................................................................................................... 49 Student Characteristics ............................................................................................................. 50 Student-Teacher Interaction ...................................................................................................... 50 Home Support ........................................................................................................................... 51 Time Spent Reading.................................................................................................................. 52 Reading Materials at Home and School.................................................................................... 52 Chapter 5 - Conclusions ................................................................................................................ 54 Summary of Key Findings ......................................................................................................... 54 Conclusions and Recommendations ......................................................................................... 56 Annex: Test reliability measures .................................................................................................... 57 3 Acronyms CLRW Come Lets Read and Write clpm Correct letters per minute clspm Correct letter sounds per minute cfwpm Correct familiar words per minute cnwpm Correct nonwords per minute cwpm Correct words per minute CPBA Community Play-based Activities ET4D Education Technology for Development MET Ministry of Education and Training PEARL Pacific Early Age Readiness and Learning TEGRA Tonga Early Grade Reading Assessment WB World Bank 4 Acknowledgement This report was prepared by: Dr. Brenda Sinclair, Global Education, Gender & Evaluation Consultant, Global Education Advisors, San Francisco, CA USA Dr. Michel Rousseau, Assessment and Measurement Specialist, Université du Quebec, Montreal, Canada Isabelle Duston, Chief Executive Officer, Education Technology for Development The authors would like to express their gratitude to the Tonga Ministry of Education and Training, the Tonga PEARL team, TEGRA enumerators, and all teachers and students that participated in the survey. To all, Malo aupito! 5 Executive Summary This report presents the learning gains achieved by students participating in two rounds of the Tonga Early Grade Reading Assessment (TEGRA)1, corresponding to the 2017 school year. The assessments are part of a data collection strategy to evaluate the impact of the “Come, Let’s Read and Write Program� (CLRW) which operated in Tongan primary classrooms – Classes 1 and 2 – in the 2015, 2016 and 2017 school years. This report provides an initial exploration of the learning gains between the midline TEGRA that took place in March 2017 and an endline TEGRA conducted seven months later in October 2017. It was administered to a sample of 75 schools in Tonga that were supported (treatment) or monitored (control) by the “Come, Let’s Read and Write Program� between 2014 and 2017, under the Pacific Early Age Readiness and Learning (PEARL) program. The assessments were conducted through a collaborative partnership with the World Bank, Education Technology for Development (Et4d) and the Tonga Ministry of Education and Training (MET) with funding provided by the Global Partnership for Education (GPE). Each TEGRA included a nationally randomized sample of over 2,000 students enrolled in 74 schools across the islands of Tongatapu, Ha’apai, Vava’u, Niuafo’ou, and Niuatoputapu. Learning gains from midline to endline were analyzed for 2,274 students who participated in both assessments. This sample comprised 562 girls and 629 boys in Class 1 and 539 girls and 544 boys in Class 2. A similar yet equated version of the TEGRA instrument was used for both assessments and consisted of the following nine subtasks that measure foundational reading skills: Subtask 1: Letter Name Identification; Subtask 2: Initial Sound Identification; Subtask 3: Letter Sound Identification; Subtask 4: Familiar Word reading; Subtask 5: Nonword reading; Subtask 6: Oral Reading Fluency; Subtask 7: Reading Comprehension; Subtask 8: Listening Comprehension; and Subtask 9: Dictation. Summary of key findings Student results from the TEGRAs provided a measure of learning gains in key reading skills during the 2017 school year in PEARL-supported schools (treatment and control groups). To analyze these learning gains, we examined the difference in mean scores between the March 2017 midline (start of the school year) and October 2017 endline (end of the year) assessments as well as the reductions in the proportion of students scoring zero. Results from the TEGRAs were compared for each Class (Class 1 and Class 2) and each sub-population (gender, island group and school type – treatment and control). Overall mean scores provide a measure of average reading abilities in basic reading skills. Zero scores represent the share of students unable to answer at least one correct item in the test- thus being almost excluded from any learning experience during instruction. Both measures are important and complementary; together, they provide insight into how well students are acquiring basic reading skills in the early grades, the tasks they struggle with the most that require immediate attention, and show overall progress achieved from the beginning to the end 1This activity is part of the Pacific Early Age Readiness and Learning Program (PEARL), which was established to improve the school readiness and literacy outcomes of children throughout the Pacific region. 6 of the school year. A summary of the differences in zero scores and gains in mean scores from midline to endline for each subgroup (class, gender, island group and school type - treatment or control group) are summarized below. Class 1 Results. Students in Class 1 showed dramatic reductions in zero scores and significant increases in mean scores from midline to endline. Reductions in zero scores ranged from 40 percentage points on the Letter Name subtask to 70 percentage points on the Overall Dictation subtask. On the word reading fluency subtasks (familiar word, nonword, and oral reading fluency subtasks), the percentage of zero scores decreased by 60 percentage points. For the comprehension subtasks (Listening Comprehension and Reading Comprehension), zero scores decreased by 40-44 percentage points, respectively. The greatest decline was on the Dictation subtask, with a reduction of 70.6 percentage points from midline to endline. In terms of mean scores, Class 1 showed gains across all subtasks. On the timed fluency subtasks,2 gains ranged from an increase of 6.4 correct nonwords per minute (cnwpm) on the Nonword subtask to 30 correct letters per minute (clpm) on the Letter Name subtask. For the untimed subtasks, gains in Class 1 ranged from 14.8 percentage points in Listening Comprehension to 27.9 percentage points on Initial Sound. The most impressive increase was in Dictation mean scores, which jumped from 6.3% at midline to 55.6% at endline – a difference of 49.3 percentage points. Overall, Class 1 students improved in all reading skills during the 2017 school year, showing considerable increases in the dictation, letter name, initial sound, and letter sound subtasks. Across all skills, the lowest gains were observed in decoding (reading of nonwords) and listening comprehension subtasks. These results suggest that Class 1 is a determinant grade to help students achieve considerable learning gains in letter fluency and basic writing skills (dictation, spelling). Class 2 Results. Class 2 also showed reductions in zero scores across all subtasks. Reductions in zero scores ranged from -1.7% on the Letter Name subtask to -30.8 percentage points in Reading Comprehension. On the word reading fluency subtasks, zero scores decreased by 15-18 percentage points with the greatest improvement in Oral Reading Fluency (-18). The percentage of students who could not read one word of the reading passage (Oral Reading Fluency subtask) decreased from 22% at midline to 4% at endline. For the comprehension subtasks (Listening Comprehension and Reading Comprehension), zero scores decreased by 21-31 percentage points, respectively. The greatest decline was in Reading Comprehension in which zero scores dropped from 42% at midline to 11% at endline. Mean scores for Class 2 increased from midline to endline across all subtasks, with the most significant gains in the word fluency and reading comprehension subtasks. For the timed word reading fluency subtasks, gains ranged from 9 more correct nonwords per minute on the Nonword subtask to 20 more correct words per minute for the Oral Reading Fluency subtask. For the untimed subtasks, gains in Class 2 ranged from 16.3 percentage points on the Initial Sound subtask to 25 percentage points on the Reading Comprehension subtask. Overall, Class 2 students improved in all reading skills during the 2017 school year, showing considerable gains in word reading fluency and comprehension. Across all skills, the lowest gains were observed in decoding (reading of nonwords) and listening comprehension subtasks. These results suggest that Class 2 is a determinant grade for gaining word knowledge, fluency and reading comprehension after students have mastered the lower foundational reading skills in Class 1. 2 Note that subtasks designed to measure reading fluency (e.g., Letter Name, Letter Sound, Familiar Word, Nonword, Oral Reading Fluency), are timed to one minute and the scores are reported as the number of correct items (e.g., letters, words) read per minute. 7 Gains by Gender. The results by gender were mixed. In Class 1, both boys and girls showed similar rates of reduced zero scores, ranging from 37 to 70 percentage points. Girls showed greater decreases among Familiar Word, Nonword, ORF and Reading Comprehension subtasks. The greatest difference was on the ORF subtask in which girls’ zero scores decreased 14 more percentage points than boys (by 71% compared to a decrease of 57% for boys). When comparing mean scores by gender in Class 1, girls showed higher gains across all subtasks, with increases ranging from 9.7 cnwpm on the Nonword subtask to 54.2 percentage points in Dictation while boys’ gains ranged from 3.6 cnwpm on the Nonword subtask to 45.3 percentage points in Dictation. Class 1 girls achieved notably higher gains on the Dictation (+9 percentage points), Letter Name (+7 clpm) and Initial Sound subtasks (+6 percentage points) compared to gains in mean scores for boys. In Class 2, girls showed a greater reduction of zero scores in four of the nine subtasks while boys showed a greater reduction in four subtasks (Familiar Word, Nonword, Oral Reading Fluency and Reading Comprehension). Both scored similarly on the Dictation subtask. The most striking difference in performance was in the Oral Reading Fluency and Reading Comprehension subtasks. On the Oral Reading Fluency subtask, boys’ zero scores dropped by 20% compared to 14% for girls, a difference of 6 percentage points. In Reading Comprehension, the percentage of zero scores declined by 34% for boys and 26% for girls, a difference of 8 percentage points. In terms of mean scores by gender, girls expressed slightly higher gains across eight of the nine subtasks, with the exception of Dictation. Girls’ gains ranged from an increase of 10.7 cnwpm on the Nonword subtask to 25.7 percentage points on the Reading Comprehension subtask while boys gains ranged from 6.7 percentage points in Listening Comprehension to 24.2 percentage points on Reading Comprehension. Girls exhibited notable gains in ORF and Reading Comprehension subtasks compared to boys. Class 2 girls read 6.5 more cwpm than boys on the ORF subtask and gained 7 more percentage points than boys in Reading Comprehension. Overall, the results indicate that both boys and girls improved; however, girls showed higher gains across all subtasks in Class 1 and across eight of the nine subtasks in Class 2 (with the exception of Dictation). These differences in performance should not be discounted because they are consistent and can be taken as a sign of systematic, yet not well understood, differences in the learning opportunities and experiences offered to boys and girls. There may be cultural or gender barriers that affect boys’ interest and engagement in reading activities. These results are reflective of international trends, which show that girls consistently outperform boys on standardized tests in the early primary grades in almost every country in the world. 3 Gains by Island Group. The gains for Class 1 by Island Group were mixed. Niuatoputapu achieved the highest gains in five subtasks while Vava’u ranked highest on two subtasks (Letter Name and Dictation) and Hapai ranked highest on one subtask (Letter sound). Three island groups scored similarly for Letter Sound: Hapai (23.8), Niuatoputapu (23.4) and Vava’u (23.2). Finally, Niuafo’ou had the highest gains on the Listening Comprehension subtask. In Class 2, the difference in gains by island group was more distinct. Niuatoputapu had the highest gains on seven subtasks while Hapai gained the most on one subtask (Letter Name) and Niuafo’ou ranked highest on one subtask (Dictation). The island groups that demonstrated the lowest learning gains were Niuafo’ou and Tongatapu. Niuafo’ou students achieved the lowest gains on three subtasks in Class 1 (Initial Sound, Letter Sound and Reading Comprehension) and four subtasks in Class 2 (Letter Name, Familiar Word, 3 Loveless, T. (2015). Brown Center Report on American Education: How well are American students learning? Washington, DC: Brookings Institute. Available at: https://www.brookings.edu/wp-content/uploads/2016/06/2015-Brown-Center- Report_FINAL-3.pdf 8 Nonword, and ORF), but they showed the highest gains in Listening Comprehension in Class 1 and substantial gains in Class 2. Tongatapu had the lowest gains in three subtasks in Class 1 (Familiar Word, Nonword and ORF), but showed the highest gains in Reading Comprehension in Class 2. Performance on the Oral Reading Fluency (ORF) and Reading Comprehension subtasks showed vast differences by island group. Endline scores on the ORF subtask ranged from 10.9 cwpm (Tongatapu) to 23 cwpm (Niuatoputapu) in Class 1, and from 19.6 cwpm (Tongatapu) to 22.6 cwpm (Niuatoputapu) in Class 2. Reading comprehension endline scores ranged from 12.5% in Niuafo’ou to 19.1% in Niuatoputapo in Class 1, and from 48% in Tongatapu to 79% in Niuatoputapu. While Tongatapu island had the lowest endline mean scores for reading comprehension in Class 2, it had the highest gains (26 percentage points) among all island groups. Based on these results, students on the island of Niuatoputapu tend to perform better in reading than students on the other islands. These students had the highest reading fluency and comprehension scores, while students on the island of Niuafo’ou tended to perform the lowest across some subtasks, although they showed the highest gains on listening comprehension in Class 1. These students, therefore, may have had initial difficulties understanding the language of instruction, which may have hindered their reading skills development. Meanwhile, Tongatapu island had the lowest gains in three subtasks in Class 1, but showed the highest gains in Reading Comprehension in Class 2. Therefore, the reading instruction in this island was highly effective. Gains by PEARL School Type4. The comparison of learning gains by PEARL school type shows the treatment group achieved higher gains in the majority of subtasks compared to the control group. In Class 1, zero scores decreased by 40 to 68 percentage points for the treatment group and by 36 to 74 percentage points for the control group. The treatment group demonstrated a greater reduction in zero scores among four subtasks: Letter Name, Initial Sound, Letter Sound and Listening Comprehension. The control group demonstrated larger decreases in three subtasks: Familiar Word, ORF, and Dictation. There was an equivalent decline in zero scores on the Nonword and Reading Comprehension subtasks for both groups. Compared to the control group, the treatment group achieved higher gains in all subtasks, except Dictation. Between midline and endline, the Class 1 treatment group gained between 7.9 cnwpm on the Nonword subtask to 47.2 percentage points on the Dictation subtask, while the control group gained between 4.6 cnwpm on the nonword subtask and 51.6 percentage points on the Dictation subtask. The most noteworthy and statistically significant gains were on the Letter Sound and the Initial Sound Identification subtasks. On the Letter Sound subtask, the treatment group gained 28 more clspm while the control group gained 12.8 more clspm by endline, resulting in a true gain of 15 more clspm for the treatment group. On the Initial Sound subtask, the treatment group gained 9 more percentage points than the control group, gaining 32.1 percentage points at endline while the control group gained 22.9 percentage points. These differences in gains constitute an effect size 5 of the difference in learning of [0.10] or [0.15] of learning in the school year. The lowest differences in learning between the two groups were observed in Listening Comprehension with an effect size of 0.04. In Class 2, zero scores decreased by 1.6 to 34.7 percentage points for the treatment group and by 0.8 to 25.2 percentage points for the control group. The control group showed a greater decline in six of nine subtasks, while the treatment group had a greater reduction in zero scores on two subtasks (Letter Name and Reading Comprehension). 4 PEARL schools refer to the 75 randomly selected schools from the government and non-government education systems that were either supported (treatment group) or monitored (control group) by the “Come, Let’s Read and Write Program� in the 2015, 2016 and 2017 school years. 5 Effect size is a statistic that presents differences in standardized value. The conventional values are: small=0.2, medium=0.5 and large=0.8. 9 In terms of differences in mean scores, the Class 2 treatment group achieved higher gains than the control group in seven of the nine TEGRA subtasks from midline to endline. Gains for the Class 2 treatment group ranged from 7.2 percentage points on the Listening Comprehension subtask to 27.9 on the Reading Comprehension subtask. Meanwhile, the control group gained between 8.1 cnwpn on the Nonword subtask to 20.6 percentage points on the Reading Comprehension subtask. Gains on the fluency subtasks averaged 3 more correct letters/words per minute compared to the control group. On the reading comprehension subtask, the treatment group scored 7 percentage points higher than the control group (28 versus 21, respectively). The control group scored 4 percentage points higher on the Initial Sound subtask and 6 percentage points higher in Listening Comprehension. The results indicate that both school types showed a reduction of zero scores and increased learning from midline to endline; however, the treatment group showed slightly higher increases on the majority of subtasks. Student Contextual Factors The student contextual factors indicate that on average students do have a supportive literacy environment at school and at home. The majority of students have time and materials to read at school. At home, most students receive help with homework from a literate family member and are supported to read books that they bring from school or have at home. However, the majority of students (63%) do not have a school textbook. As part of the ongoing World Bank impact evaluation of the CLRW program, classroom observation data from monitoring and coaching may be used to further assess the quality of reading instruction, availability of teaching and learning materials, and assessment methods. Conclusions and Recommendations Learning gains in Class 1 and Class 2 students during the 2017 school year are evident in the decreasing proportion of zero scores and increasing mean scores from midline to endline. As students progress through the grades, they are acquiring more knowledge of letters, words and decoding skills resulting in increased fluency and comprehension. The progress achieved in Class 1 (e.g., learning letter names, initial sounds and letter sounds) serves as a stepping-stone for students to become more fluent readers in Class 2. The fact that fluency in letter sound knowledge manifests into more fluent readers with higher comprehension rates in Class 2 confirms lower-level skills are predictors of reading fluency. The difference in treatment and control groups gains demonstrates that improvements can be made to strengthen reading instruction and outcomes in both grades. It is important to note that both the treatment group and control group improved from Class 1 to Class 2, and Class 2 gains were small, however notable in letter sounds and initial sounds indicating some effect from the intervention. What is most impressive is the rate at which students are advancing in reading comprehension. If students continue to advance at the same rate, they will be well within international standards for fluency (60 cwpm) and comprehension (80% correct). Nonetheless, weaknesses in decoding and listening comprehension could affect comprehension of more difficult content and thus need to be strengthened before the end of Class 3. Additionally, differences between gender, island groups and school types will need to be addressed in order to achieve equitable outcomes in Oral Reading Fluency and Comprehension. 10 Finally, it is recommended that the World Bank Impact Evaluation team complement these findings with classroom observation monitoring data to explore the quality of instruction, availability of textbooks and reading materials, and reading assessment methods, in order to identify more concrete ways to improve reading instruction and overall learning outcomes. 11 Introduction Background The Kingdom of Tonga carried out its first assessment of early grade reading in November 2009 by staff and consultants of the then Ministry of Education, Women’s Affairs and Culture (MEWAC) in collaboration with the World Bank and with support from the Australian and New Zealand governments and the Global Partnership for Education -formerly Education for All Fast Track initiative (EFA-FTI). The 2009 TEGRA showed that only 3 out of 10 students completing Class 3 were able to read and understand a short story. The results from the study prompted Tongan educators to introduce changes in the official curriculum to improve the scope and focus of reading instruction in the first grades of primary education. In 2014, the Ministry of Education and Training (MET--formerly MEWAC) agreed to pilot a structured approach to reading instruction to improve the effectiveness of teaching and learning of reading skills in the early grades. The structured approach translated into the “Come, Let’s Read and Write (CLRW)� reading program and was developed to improve reading instruction in Classes 1 and 2 and monitor student progress in the classroom. The CLRW program was implemented in selected pilot schools and communities in Tonga using an experimental design involving treatment and control schools. Treatment and control schools were identified in Tongatapu, Ha’apai, Va’vau and Niuatoputapu Island groups. Treatment schools were supported with instructional materials aligned with the official curriculum outcomes, intensive teacher training and regular coaching visits to support teachers as they implemented the program. Control schools continued to implement the official curriculum and were only monitored throughout the duration of the program. As part of the evaluation design for the reading program, several rounds of TEGRA assessments were planned to assess student progress in reading across the three cohorts of students that would be served by the program. A baseline took place in 2014 and planned midline assessments were schedule at the beginning and end of the 2015, 2016 and 2017 school years. This report presents the learning gains achieved by students participating in the second to last (midline) and last (endline) rounds of Early Grade Reading Assessments, both administered in the 2017 school year. The midline Tonga Early Grade Reading Assessment (TEGRA) of the PEARL program took place in March 2017 with a nationally representative sample of 2,610 students (Classes 1-2) in 73 primary schools (37 treatment and 36 control schools) covered by the “Come, Let’s Read and Write� reading intervention. The endline TEGRA was carried out in October 2017 with a total of 3,979 students (Classes 1-3) in the same schools (386 treatment schools and 36 control schools) across the islands of Tongatapu, Ha’apai, Vava’u, Niuafo’ou, and Niuatoputapu. Each TEGRA was carried out at the request of the Ministry of Education and Training (MET) under the PEARL program and funded by the Global Partnership for Education. The data collection was conducted by a group of enumerators supervised and managed by the Pacific Early Age Readiness and Learning (PEARL) team, with technical assistance by Education Technology for Development (Et4d). The midline and endline TEGRAs built on the experience of previous studies by utilizing instruments, school sampling lists, resources, and lessons learned to produce high quality data and reliable results. 6 There were 38 treatment schools assessed at endline versus 37 treatment schools at midline. The additional school captured in the endline TEGRA was included in the sample at midline, but could not be reached due to the entire school attending a funeral. 12 Objectives The purpose of this report is to present an initial exploration of learning gains achieved by a sample of students tested both at the start (midline) and end (endline) of the 2017 school year, who are enrolled in schools participating in the PEARL-supported CLRW program. This report is not meant to evaluate the CLRW program, as a World Bank team is currently conducting a rigorous evaluation of the CLRW program based on an evaluation design developed at the beginning of the program. This report aims to complement that evaluation by providing an initial exploration of the effects observed during the 2017 school year. This analysis focuses on presenting learning gains following two key indicators: the proportion of students with zero scores and the mean scores for each TEGRA subtask. These two indicators provide a measure of the share students who are unable to demonstrate the minimum competence in each task (i.e. zero scores) and the overall competence, in terms of fluency or comprehension, of the average Class 1 or Class 2 classroom in each subtask. Together, they inform educators of the skills and share of students that are in greatest need of support in a given Class, and the overall reading performance of students in these Classes. In the TEGRA results sections (Chapter 3), the differences in percentage of zero scoring students and in mean scores are shown for Classes 1 and 2, and for each sub-population: Gender, Island group, and Type of school (Treatment or Control). The learning gains represent the difference in mean scores from the midterm TEGRA to the endline TEGRA. Structure of Report The report is divided into five chapters beginning with this introductory section in Chapter 1. Chapter 2 describes the EGRA design and implementation process. Learning gains achieved for the overall population and each sub-group are presented in Chapter 3 followed by an analysis of student contextual factors in Chapter 4. Finally, the conclusions are reported in Chapter 5. 13 Chapter 2 - TEGRA Design and Implementation Sample size and sampling methodology Analysis for the learning gains report was conducted on 2,274 students that were assessed at both the March 2017 midline TEGRA and the October 2017 endline assessment. Table 1 shows the total sample by class for the midline and endline TEGRAs as well as the shared sample of students who were tested during both assessments. At the midline TEGRA, a total of 2,614 students were assessed. The endline sample size was 2,754 due to one additional treatment school that was reached at endline, but unavailable at midline 7 . However, of the students who were tested at midline, 13% did not participate in the endline and thus could not be used in the learning gains analysis. The total number of students we were able to identify at both assessments was 2,274. The loss of students at endline could potentially bias the results of the learning gains analysis but since it was a rather small proportion of students, any effect is insignificant. Table 1: EGRA midline, endline and learning gains sample by grade Midline EGRA Endline EGRA Shared Sample Grade 1 1,402 1,430 1,191 Grade 2 1,212 1,324 1,083 Total 2,614 2,754 2,274 Table 2 shows the entire sample assessed at midline and endline by gender and class, as well as those who participated in both assessments (shared sample). Learning gains were analysed for 629 boys and 562 girls in Class 1, and for 544 boys and 539 girls in Class 2. Table 2: EGRA midline, endline and learning gains sample by class and gender Midline EGRA Endline EGRA Shared Sample Boys Girls Boys Girls Boys Girls Class 1 733 669 749 681 629 562 Class 2 616 596 663 661 544 539 Total 1,349 1,265 1,412 1,342 1,173 1,101 7 There were 38 treatment schools assessed at endline versus 37 treatment schools at midline. The additional school captured in the endline TEGRA was included in the sample at midline, but could not be reached due to the entire school attending a funeral. 14 Instrument Design Building on existing TEGRA instruments from previous assessments provided by the PEARL team, the TEGRA midline consisted of nine subtasks: Subtask 1: Letter Name recognition; Subtask 2: Initial Sound recognition; Subtask 3: Letter Sound recognition; Subtask 4: Familiar Word reading; Subtask 5: Non-word reading; Subtask 6: Oral Reading Fluency; Subtask 7: Reading Comprehension; Subtask 8: Listening Comprehension; Subtask 9: Overall Dictation. The October 2017 endline TEGRA utilized a similiar equated version of the TEGRA instrument administered in the March midline TEGRA with minor adjustments to avoid memory recall (e.g., shuffling the items, using new reading passages and dictation sentences). The fluency subtasks (Letter Name, Letter Sound, Familiar Word, Nonword, Oral Reading Fluency) were timed to one minute and scored as the correct number of items read within the minute. The Initial Sound, Reading Comprehension, Listening Comprehension and Dictation subtasks were untimed and scored as a percentage of items correct from the total possible score. Student context questionnaires consisted of 34 questions related to student characteristics, teacher-student interaction, home literacy support, time spent reading and reading materials at home and school. The questionnaire was administered to all students assessed in Class 1 and 2. There were no adaptations to the original student questionnaires provided by the PEARL team and used in previous TEGRAs. The instruments were all developed in the Tongan language and were conducted using Tangerine software on tablets provided by the PEARL team. Fieldwork and Challenges Encountered For the midline TEGRA, data collection was conducted in the Kingdom of Tonga between March 7 and March 27, 2017. Data collection for the endline assessment took place from October 2 to November 1, 2017. Lessons learned during the midline were applied to the endline resulting in very few challenges encountered by endline. For instance, the endline benefited from the verified list of students Et4d received from the PEARL team, the simplified process (major paper reduction) resulting from addressing challenges encountered during the midterm assessment, and the conscientious work of the logistics and assessment teams. The only complication experienced by the team was exam schedules and World Teachers’ Day interfering with the first week of data collection. Due to these events, data collection could not take place for three of the five days planned and data collection had to be extended. A recommendation, in the case of future data collections, is for data collection not to be planned on dates the MET has planned standardized tests, and for the team to be given more than two weeks notice to adjust schedules, if needed (as last minute changes can pose financial implications). 15 Chapter 3- Learning Gains For the purpose of this report, learning gains are defined as any increase in mean scores between midterm and final performance evaluations. Results from the midline and endline TEGRAs were compared to determine differences in zero scores and gains in mean scores for the overall sample, and for each sub-population. The learning gains are reported by subtask, gender, island group, and school type (treatment or control group). The summary section reports the overall mean gains and reduction of zero scores for Class 1 and Class 2, as well as the learning gains of girls and boys, and of the treatment and control groups. The subsequent section provides an in-depth analysis of learning gains for each specific subtask. The results by subtask section presents gains for each Class, gender, school type (Treatment and Control) and island group. EGRA Summary Results – Reductions in zero-scoring students ZERO SCORES BY CLASS Table 3 illustrates the difference in midline and endline zero scores for Class 1 and Class 2. Class 1 students showed a reduction in zero scores by 40 to 70 percentage points. The most significant reduction was in the Dictation subtask, with a decrease of 70.6 percentage points from midline to endline. The percentage of zero scores on word reading fluency subtasks (Familiar Word, Nonword and Reading Fluency) decreased by about 60% each while the percentage of zero scores for Reading and Listening Comprehension reduced by about 40%. It is notable to add that at midline (i.e. start of the year) over 90% of Class 1 students could not read one word correctly on the Reading Fluency subtask, but by endline the proportion of non-readers dropped to 34%. Similarly, students with zero reading comprehension skills also dramatically reduced from 99% at midline to 54% at endline. These results show how Class 1 is a determinant year for a large share of students to make initial steps into recognizing letter sounds, reading words and comprehending sentences. For Class 2, the percentage of zero scores was much smaller at midline partly because Class 2 students have been already benefitted from one full-year of instruction. Starting from a lower base, reductions in overall zero scores among Class 2 are more modest than those observed in Class 1. By endline, zero scores for Class 2 reduced in range between 2 and 31 percentage points overall. The most substantial decrease in zero scores was in Reading Comprehension (-30.8), where the percentage of zero scores dropped from 42% at midline to 11% at endline. Furthermore, the percentage of Class 2 students who could not read one word correctly on the ORF subtask dropped from 22% at midline to 4% at endline. These results suggest that the curriculum expectation that no student should remain unable to read after two years of instruction is not only possible, but well within the reach of Tongan educators. 16 Table 3: Class 1 and Class 2 Midline and Endline Zero Scores Class 1 Class 2 Midline Endline Midline Endline Sub-Test Difference Difference (%) (%) (%) (%) Letter Name 43.1 3 -40.1 2 0.3 -1.7 Initial Sound 84.9 34.4 -50.5 24.9 11.7 -13.2 Letter Sound 71.9 18.1 -53.8 8.6 3.4 -5.2 Familiar Word 97.1 36.4 -60.7 22.4 5.9 -16.5 Nonword 97.2 38 -59.2 23.2 7.9 -15.3 Oral Reading Fluency 96.8 33.6 -63.2 21.6 4 -17.6 Reading Comprehension 98.9 54.2 -44.7 42 11.2 -30.8 Listening 68.8 30.6 -38.2 37.3 16.1 -21.2 Comprehension Overall Dictation 83.1 12.5 -70.6 4 1.9 -2.1 ZERO SCORES BY GENDER The difference in zero scores for Class 1 boys and girls is illustrated in Table 4. Based on Class 1 results, both boys and girls showed similar rates of reduced zero scores, ranging from 37 to 70 percentage points. However, girls showed greater decreases among Familiar Word, Nonword, ORF and Reading Comprehension subtasks. The greatest difference was on the ORF subtask in which girls’ zero scores decreased 14 more percentage points than boys (by 71% compared to a decrease of 57% for boys). Table 4: Class 1 Midline and Endline Zero Scores by Gender Class 1 Boys Girls Sub-Test Midline Endline Midline Endline Difference Difference (%) (%) (%) (%) Letter Name 45.1 3.5 -41.6 40.7 2.4 -38.3 Initial Sound 85.7 38.9 -46.8 84 28.9 -55.1 Letter Sound 70.6 20 -50.6 73.5 15.8 -57.7 Familiar Word 96.9 42.2 -54.7 97.2 29.4 -67.8 Nonword 97.2 44.8 -52.4 97.2 29.8 -67.4 Oral Reading Fluency 96.5 39.8 -56.7 97.2 26.2 -71 Reading 98.9 61.3 -37.6 98.8 45.5 -53.3 Comprehension Listening 67.9 30.9 -37 69.8 30.3 -39.5 Comprehension Overall Dictation 84.7 14.4 -70.3 81.1 10.1 -71 17 The difference in zero scores from midline to endline by gender for Class 2 is illustrated in Table 5. Class 2 boys performed better than girls as demonstrated by a greater reduction of zero scores in Familiar Word, Nonword, ORF, and Reading Comprehension subtasks. The most notable difference was in ORF and Reading Comprehension subtasks. On the ORF subtask, boys’ zero scores dropped by 20% compared to 14% for girls, a difference of 6 percentage points. In Reading Comprehension, the percentage of zero scores declined by 34% for boys and 26% for girls, a difference of 8 percentage points. Table 5: Class 2 Midline and Endline Zero Scores by Gender Class 2 Boys Girls Sub-Test Midline Endline Midline Endline Difference Difference (%) (%) (%) (%) Letter Name 2.1 0.5 -1.6 1.9 0 -1.9 Initial Sound 25.5 14.4 -11.1 24.2 8 -16.2 Letter Sound 8.6 4.2 -4.4 8.6 2.2 -6.4 Familiar Word 26.5 8.1 -18.4 16.6 3 -13.6 Nonword 25.7 9.5 -16.2 19.6 5.7 -13.9 Oral Reading Fluency 25.6 5.5 -20.1 15.9 2 -13.9 Reading 46.9 12.9 -34 35 8.8 -26.2 Comprehension Listening 34.5 15.1 -19.4 41.1 17.6 -23.5 Comprehension Overall Dictation 4.3 2.3 -2 3.4 1.4 -2 ZERO SCORES BY SCHOOL TYPE Table 6 presents the difference in zero scores for Class 1 treatment and control groups. Both groups demonstrated a reduction in zero scores from midline to endline, suggesting Class 1 teachers in both groups can facilitate instructional activities that help most students go from zero to initial levels in reading. Zero scores decreased by 40 to 68 percentage points for the treatment group and by 36 to 74 percentage points for the control group. The treatment group showed a greater reduction among four subtasks: Letter Name, Initial Sound, Letter Sound and Listening Comprehension. The most notable difference was on the Letter Sound subtask with a decrease of 57.5 percentage points compared to 49.6 percentage points for the control group. The control group demonstrated larger decreases in three subtasks: Familiar Word, ORF, and Dictation. There was an equivalent decline in zero scores on the Nonword and Reading Comprehension subtasks for both groups. 18 Table 6: Class 1 Midline and Endline Zero Scores by School Type 8 Class 1 Treatment Control Sub-Test Midline Endline Midline Endline Difference Difference (%) (%) (%) (%) Letter Name 46.3 3 -43.3 39.4 3 -36.4 Initial Sound 83.7 30.3 -53.4 86.3 39.2 -47.1 Letter Sound 69.8 12.3 -57.5 74.4 24.8 -49.6 Familiar Word 97.4 37.9 -59.5 96.7 34.7 -62 Nonword 97 37.8 -59.2 97.4 38.2 -59.2 Oral Reading Fluency 96.7 34.3 -62.4 96.9 32.9 -64 Reading 98.6 54 -44.6 99.3 54.3 -45 Comprehension Listening 67.9 28.3 -39.6 69.7 33.2 -36.5 Comprehension Overall Dictation 82.9 15 -67.9 83.2 9.5 -73.7 The differences in zero scores for Class 2 treatment and control groups are presented in Table 7. Both groups demonstrated a reduction in zero scores from midline to endline. The control group showed a greater decline in six of nine subtasks, while the treatment group had a greater decrease in zero scores on two subtasks (Letter Name and Reading Comprehension). It is important to note that the control group had a higher percentage of zero scores at midline on seven subtasks compared to the treatment group. For the treatment group, the most notable difference was on the Reading Comprehension subtask with a decrease of 35 percentage points compared to 25 percentage points for the control group. On the Nonword subtask, zero scores reduced about 15 percentage points for both school types. Table 7: Class 2 Midline and Endline Zero Scores by School Type Class 2 Treatment Control Sub-Test Midline Endline Midline Endline Difference Difference (%) (%) (%) (%) Letter Name 2.6 0.2 -2.4 1.2 0.4 -0.8 Initial Sound* 17.6 8.6 -9 35.3 16.2 -19.1 Letter Sound* 3.5 0.7 -2.8 15.8 7.3 -8.5 Familiar Word 21.2 3.4 -17.8 24.1 0.9 -23.2 Nonword 21.2 5.7 -15.5 25.9 11 -14.9 Oral Reading Fluency 18.9 2.8 -16.1 25.4 5.8 -19.6 Reading 42.1 7.4 -34.7 41.8 16.6 -25.2 Comprehension 8 Note that for Table 5, none of the differences observed in the treatment and control groups were statistically significant for Class 1. 19 Listening 35 16.4 -18.6 40.4 15.8 -24.6 Comprehension Overall Dictation 3 1.4 -1.6 5.2 2.8 -2.4 *Difference in gain score between School type were significant at p<.05 EGRA Summary Results – Overall Learning Gains The learning gains for Class 1 and Class 2 are presented in Table 8 and represent the differences in mean scores from midline to endline. As illustrated, both classes showed positive gains across all subtasks. On the timed fluency subtasks which measure automaticity in letter recognition and word reading, gains in Class 1 ranged from an increase of 6.4 correct nonwords per minute (cnwpm) on the Nonword subtask to 30 more correct letters per minute (clpm) on the Letter Name subtask. For the untimed subtasks measuring percentage of items correct, gains in Class 1 ranged from 14.8 percentage points in Listening Comprehension to 27.9 percentage points on Initial Sound. The greatest increase was in Dictation mean scores, which jumped from 6.3% correct at midline to 55.6% correct at endline – a difference of 49.3 percentage points. In Class 2, gains ranged from 9.7 more correct nonwords per minute to 20.2 more correct words per minute on the Oral Reading Fluency (ORF) subtask. Fluency scores on the ORF subtask improved from 19 cwpm at midline to 39 cwpm at endline. On the untimed subtasks, gains ranged 9.6 percentage points on the Listening Comprehension subtask to 25 percentage points in Reading Comprehension. Reading Comprehension scores improved from 25% correct at midline to 50% at endline. While these results are impressive, examining scores by sub-group (gender, school type, and island group) is important for understanding the actual gain of each type of population. Table 8: Class 1 and Class 2 midline and endline mean scores Class 1 Class 2 Sub-Test Midline Endline Gains Midline Endline Gains Letter Name (clpm)* 13.5 43.8 30.3 50.1 62.9 12.8 Initial Sound (%)* 6.7 34.6 27.9 48.6 64.9 16.3 Letter Sound (clspm)* 6.2 27.1 20.9 35.5 46.9 11.4 Familiar Word (cwpm)* 1.2 13.4 12.2 19.1 36.8 17.7 Nonword (cnwpm)* 1.8 8.2 6.4 11.6 21.3 9.7 Oral Reading Fluency 1.4 13.1 11.7 19.1 39.3 20.2 (cwpm)* Reading Comprehension 0.4 16.9 16.5 25.4 50.2 24.8 (%) Listening Comprehension 9.8 24.6 14.8 26.2 35.8 9.6 (%) Overall Dictation (%)* 6.3 55.6 49.3 69 82.5 13.5 * Difference in gain score between Class were significant at p<.05 20 LEARNING GAINS BY GENDER The Class 1 gains of boys and girls are presented in Table 9. Girls showed higher gains across all subtasks, with increases ranging from 9.7 cnwpm on the Nonword subtask to 54.2 percentage points in Dictation while boys’ gains ranged from 3.6 cnwpm on the Nonword subtask to 45.3 percentage points in Dictation. The highest gains for girls were on the Dictation, Letter Name and Initial Sound Subtasks. On the Letter Name subtask, girls gained 34 clpm by endline compared to 27 clpm for boys. On the Initial Sound subtask, girls identified 31% of Initial Sounds at endline compared to 25% for boys. Table 9: Class 1 Midline and Endline Mean Scores by Gender Class 1 Boys Girls Sub-Test Midline Endline Gains Midline Endline Gains Letter Name (clpm)* 13.1 40.3 27.2 13.9 47.9 34 Initial Sound (%)* 6.6 31.7 25.1 6.9 38.1 31.2 Letter Sound (clspm)* 7.9 24.8 16.9 4.1 30 25.9 Familiar Word 1.8 11.1 9.3 0.4 16.2 15.8 (cwpm)* Nonword (cnwpm)* 3.1 6.7 3.6 0.3 10 9.7 Oral Reading Fluency 2.2 10.9 8.7 0.4 15.6 15.2 (cwpm)* Reading 0.3 13.6 13.3 0.5 21.1 20.6 Comprehension (%) Listening 10.4 24.2 13.8 9.1 25.1 16 Comprehension (%) Overall Dictation (%)* 5.8 51.1 45.3 6.9 61.1 54.2 * Difference in gain score between Class were significant at p<.05 Gains for boys and girls in Class 2 are illustrated in Table 10. Girls expressed slightly higher gains across eight of the nine subtasks, with the exception of Dictation. Girls gains ranged from an increase of 10.7 cnwpm on the Nonword subtask to 25.7 percentage points on the Reading Comprehension subtask while boys gains ranged from 6.7 percentage points in Listening Comprehension to 24.2 percentage points on Reading Comprehension. Girls exhibited notable gains in ORF and Reading Comprehension subtasks compared to boys. Class 2 girls read 6.5 more cwpm than boys on the ORF subtask and gained 7 more percentage points than boys in Reading Comprehension. 21 Table 10: Class 2 Midline and Endline Mean Scores by Gender Class 2 Boys Girls Midline Endline Midline Endline Sub-Test Gains Gains (%) (%) (%) (%) Letter Name (clpm)* 47.8 59.5 11.7 53.5 67.8 14.3 Initial Sound (%)* 46.8 61.9 15.1 51.3 69.1 17.8 Letter Sound (clspm)* 34.6 44.9 10.3 36.7 49.9 13.2 Familiar Word 16.6 33.2 16.6 22.5 41.9 19.4 (cwpm)* Nonword (cnwpm)* 10.2 19.4 9.2 13.4 24.1 10.7 Oral Reading Fluency 16.5 35.3 18.8 22.9 44.8 21.9 (cwpm)* Reading 22.1 46.3 24.2 30 55.7 25.7 Comprehension (%) Listening 28.8 35.5 6.7 22.6 36.2 13.6 Comprehension (%) Overall Dictation (%)* 66.5 80.9 14.4 72.5 84.9 12.4 * Difference in gain score between Class were significant at p<.05 LEARNING GAINS BY SCHOOL TYPE The Class 1 gains for the treatment and control groups are presented in Table 11. When compared to the control group, the treatment group achieved higher gains in all subtasks, except Dictation. Between midline and endline, the Class 1 treatment group gained between 7.9 cnwpm on the Nonword subtask to 47.2 percentage points on the Dictation subtask, while the control group gained between 4.6 on the nonword subtask and 51.6 on the Dictation subtask. The most noteworthy and statistically significant gains were on the Letter Sound (an average of 15 letter sounds more than in those in the control group) and Initial Sound subtasks (an average of 9 percentage points or almost one more question in this subtask than the control group). On the Letter Sound subtask, the treatment group gained 28 more clspm while the control group gained 12.8 more clspm by endline, resulting in a true gain of 15 more clspm for the treatment group. On the Initial Sound subtask, the treatment group gained 32.1 percentage points at endline while the control group gained 22.9 percentage points (a difference of 9 percentage points). These differences in gains constitute an effect size 9 of the difference in learning of [0.10] or [0.15] of learning in the school year. The lowest differences in learning between the two groups were observed in Listening Comprehension with an effect size of 0.04. 9 Effect size is a statistic that present differences in standardized value. The conventional values are: small=0.2, medium=0.5 and large=0.8. 22 Table 11: Class 1 Midline and Endline Mean Scores by School Type Class 1 Treatment Control Sub-Test Midline Endline Gains Midline Endline Gains Letter Name (clpm) 10.6 42.2 31.6 16.8 45.6 28.8 Initial Sound (%)* 7.6 39.7 32.1 5.7 28.6 22.9 Letter Sound (clspm)* 5.4 33.4 28 7.1 19.9 12.8 Familiar Word (cwpm) 0.5 13.9 13.4 2 12.8 10.8 Nonword (cnwpm) 0.3 8.2 7.9 3.5 8.1 4.6 Oral Reading Fluency 0.7 13.9 13.2 2.2 12.1 9.9 (cwpm)* Reading 0.5 17.4 16.9 0.3 16.5 16.2 Comprehension (%) Listening 10.4 26.1 15.7 9.1 22.8 13.7 Comprehension (%) Overall Dictation (%) 6.1 53.3 47.2 6.7 58.3 51.6 * Difference in gain score between School type were significant at p<.05 Table 12 displays Class 2 gains for the treatment and control groups. The treatment group achieved higher gains than the control group in seven of the nine TEGRA subtasks. Gains on the fluency subtasks averaged 3 more correct letters/words per minute compared to the control group. On the reading comprehension subtask, the treatment group scored 7 percentage points higher than the control group (28 versus 21, respectively). The control group scored 4 percentage points higher on the Initial Sound subtask and 6 percentage points higher in Listening Comprehension. Table 12: Class 2 Midline and Endline Mean Scores by School Type Class 2 Treatment Control Sub-Test Midline Endline Gains Midline Endline Gains Letter Name (clpm) 47.9 62.5 14.6 53.2 63.7 10.5 Initial Sound (%) 55.6 70.1 14.5 38.7 57.6 18.9 Letter Sound (clspm) 43.8 56.1 12.3 23.6 34 10.4 Familiar Word (cwpm) 20.1 39.1 19 17.6 33.6 16 Nonword (cnwpm) 11.7 22.6 10.9 11.4 19.5 8.1 Oral Reading Fluency 19.6 41.3 21.7 18.5 36.3 17.8 (cwpm) Reading 25.8 53.7 27.9 24.7 45.3 20.6 Comprehension (%)* Listening 28.2 35.4 7.2 23.4 36.4 13 Comprehension (%)* Overall Dictation (%) 68.9 83.2 14.3 69.1 81.6 12.5 * Difference in gain score between School type were significant at p<.05 23 Learning Gains by Subtask LETTER NAME SUBTASK The Letter Name knowledge subtask is the most basic EGRA subtask. It measures students’ ability to read the names of the letters of the alphabet with fluency. During the assessment, students were given a sheet of 100 randomly distributed upper- and lowercase letters of the Tongan alphabet and asked to read as many letters as possible within one minute. The mean score is expressed as the number of correct letters per minute (clpm). Learning Gains by Class Figure 1 shows the overall gains in Class 1 and Class 2 for the Letter Name subtask. Class 1 students identified 13.5 clpm at midline and 43.8 clpm at endline, resulting in a gain of 30 clpm. Class 2 students identified 50 clpm at midline and 63 clpm at endline, an increase of 13 clpm. Therefore, Class 1 expressed higher gains than Class 2. Figure 1: Letter Name Learning Gains Class 1 Class 1 43.8 30.3 13.5 Midline Endline Gains Figure 2: Letter Name Learning Gains Class 2 Class 2 62.9 50.1 12.8 Midline Endline Gains 24 Learning Gains by Gender Table 13 shows the learning gains disaggregated by gender for the Letter Name subtask. This tasks measures automaticity in reading letter names. Girls showed higher gains in automatically identifying letter names across both classes. In Class 1, girls gained 7 more letters identified per minute than boys (gain of 34 at endline compared to 27 for boys). Class 2 girls learned 2 more letters per minute than boys (gain of 14 at endline compared to 12 for boys). Table 13: Letter Name Learning Gains by Gender Letter Boys Girls Name Midline Endline Gains Midline Endline Gains Class 1 13.1 40.3 27.2 13.9 47.9 34 Class 2 47.8 59.5 11.7 53.5 67.8 14.3 Learning Gains by Island Group The Letter Name results by Island Group are presented in Table 14. Students in schools in Vava’u and Ha’apai demonstrated the highest gains in this subtask across all group islands and across both classes. Vava’u achieved the highest gains (33.9 clpm) in Class 1 and the second highest in Class 2. Meanwhile Ha’apai demonstrated the highest gains in Class 2 (15.3 clpm) and ranked second in Class 1 (14.3 clpm). Niuatoputapu demonstrated the lowest gains in Class 1(21.5 clpm) while Niuafo’ou achieved the lowest gains in Class 2, and both results were statistically significant (meaning the results are likely true for the general population of Class 2 students on those island groups). Table 14: Letter Name Learning Gains by Island Class 1 Class 2 Subtask Midline Endline Gains Midline Endline Gains Ha’apai 14.8 46.5 31.7 57.4 72.7 15.3 Niuafo’ou* 22.3 51.9 29.6 60.8 61.4* 0.6 Niuatoputapu* 13.4 34.9* 21.5 50.8 57.3 6.5 Tongatapu 13.8 43.6 29.8 49.9 62.5 12.6 Vava’u 10.4 44.3 33.9 46.9 61.2 14.3 *Island group is significantly different from the others (p<.05) Learning Gains by School Type According to the Letter Name subtask results for Class 1 in Figure 3, the treatment group scored lower than the control group at endline, but showed slightly higher gains. Students in the treatment group identified 10.6 clpm at midline and 42.2 clpm at endline, resulting in a total gain of 31.6 letters. Meanwhile, learners in the control group read 16.8 clpm at midline and 45.6 clpm at endline, for a total gain of 28.8 letters. Thus, the treatment group identified 2.8 more letter names than the control group. 25 Figure 3: Letter Name Learning Gains by School Type (Class 1) 45.6 42.2 31.6 28.8 16.8 Treatment 10.6 Control Midline Endline Gains Class 1 The Letter Name subtask results by school type for Class 2 are represented in Figure 4. Similar to Class 1, the treatment group scored slightly lower than the control group, but achieved higher gains. At midline, students in the treatment group identified 47.9 clpm versus 53.2 clpm for the control group. By endline, the treatment group gained 14.6 more letters per minute with a mean score of 62.5 clpm while the control group gained 10.5 clpm with a mean score of 63.7 clpm. Therefore, the treatment group identified 4.1 more letter names than the control group. Figure 4: Letter Name Learning Gains by School Type (Class 2) 62.5 63.7 53.2 47.9 Treatment 14.6 10.5 Control Midline Endline Gains Class 2 INITIAL SOUND SUBTASK The Initial Sound subtask is a measure of phonological awareness - awareness of speech sounds and how sounds make up words. In this subtask, students were presented with a word orally and asked to isolate and pronounce only the first sound of the word. It is an untimed subtask and it does not intend to measure automaticity in the skill but overall accuracy. The task consists of ten words and the students need to identify the initial sound in each word for a maximum score of 10. The mean score is reported as a percentage of the total items. 26 Learning Gains by Class Figure 5 shows the overall gains in Class 1 and Class 2 for the Initial Sound subtask. In Class 1, students identified 6.7% of initial sounds presented at midline and 34.6% at endline, resulting in an increase of 27.9 percentage points. Class 2 students correctly identified 48.6% of initial sounds presented at midline and 64.9% at endline, an increase of 16.3 percentage points. Figure 5: Initial Sound Subtask Learning Gains Class 1 34.6 27.9 Class 1 6.7 Midline Endline Gains Figure 6: Initial Sound Subtask Learning Gains Class 2 Class 2 64.9 48.6 16.3 Midline Endline Gains Learning Gains by Gender Table 15 shows the learning gains disaggregated by gender for the Initial Sounds subtask. Girls demonstrated higher gains across both classes. Class 1 girls scored 6.9% correct at midline and 38% correct at endline, an increase of 31.2 percentage points. Meanwhile, boys’ mean scores increased from 6.6% correct at midline to 31.7% correct at endline, an increase of 25.1 percentage points. In Class 2, girls demonstrated gains of 17.8 percentage points compared to 15 for boys. Girls in Class 2 identified 51.3% of initial sounds correctly at midline and 69.1% at endline. Meanwhile, boys’ mean scores increased from 46.8% at midline to 61.9% by endline. 27 Table 15: Initial Sound Learning Gains by Gender Boys Girls Midline Endline Gains Midline Endline Gains Class 1 6.6 31.7 25.1 6.9 38.1 31.2 Class 2 46.8 61.9 15.1 51.3 69.1 17.8 Learning Gains by Island Group The learning gains by island group for the Initial Sounds subtask is shown in Table 16. In Class 1, Niuatoputapu island group demonstrated the highest gains with mean scores increasing from 11.3% correct at midline to 45.5% correct at endline, and the difference was statistically significant. In Class 2, Tongatapu achieved the highest gains of 19.9 percentage points from 44.5% correct at midline to 64.5% correct at endline. In terms of the lowest gains, Niuafo’ou exhibited the lowest gains in Class 1 (2.5 percentage points) and Niutoputapu demonstrated the lowest gains in Class 2 (9.8 percentage points). These results were statistically significant. Table 16: Initial Sound Learning Gains by Island Group Class 1 Class 2 Subtask Midline Endline Gains Midline Endline Gains Ha’apai* 11.9 31.9 20 64.6 64.9* 0.3 Niuafo’ou* 5.8 8.3* 2.5 22.2 38.2 16 Niuatoputapu* 11.3 45.5* 34.2 85.9 95.7 9.8 Tongatapu 5.1 34.8 29.7 44.5 64.4 19.9 Vava’u 9.8 33.9 24.1 58.7 63.9 5.2 * Island group is significantly different from the others (p<.05) Learning Gains by School Type Learning gains on the Initial Sounds subtask by school type for Class 1 are displayed in Figure 7. Mean scores for the treatment group increased from 7.6% correct at midline to 39.7% correct at endline, an increase of 32.1 percentage points. The control group also improved from a mean score of 5.7% correct at midline to 28.6% correct at endline, an increase of 22.9 percentage points. Overall, the treatment group outperformed the control group by 9 percentage points. Figure 7: Initial Sounds Learning Gains by School Type (Class 1) 39.7 32.1 28.6 22.9 Treatment 7.6 5.7 Control Midline (%) Endline (%) Gains Class 1 28 Figure 8 shows the gains on the Initial Sounds subtask by school type for Class 2. The treatment group correctly identified 55.6% of initial sounds during the midline and 70.1% at endline, which is an overall gain of 14.5 percentage points. Class 2 students in the control group scored lower at midline and endline (38.7% and 57.6% correct, respectively), but experienced higher gains (18.9 percentage points). Figure 8: Initial Sounds Learning Gains by School Type (Class 2) 70.1 55.6 57.6 38.7 18.9 Treatment 14.5 Control Midline (%) Endline (%) Gains Class 2 LETTER SOUND SUBTASK The alphabetic principle is the understanding that words are made up of sounds and that letters are symbols that represent those sounds 10 . The Letter Sound knowledge subtask assesses students’ ability to identify the correct sound associated with each letter of the alphabet. It includes letters presented as stand-alone letters or letters that appear in the beginning, middle or end of a word. Learners were provided with a sheet of 100 stand-alone letters and asked to identify as many letter sounds as they could within one minute. The mean score is expressed as the number of correct letter sounds per minute (clspm). Learning Gains by Class Gains in the Letter Sound subtask by Class are displayed in Figure 9. Class 1 demonstrated lower mean scores than Class 2, but had higher gains. Students in Class 1 identified 6.2 clspm during midline and 27 clspm at endline, demonstrating increased knowledge of 21 letter sounds. Class 2 students identified 35.5 clspm at midline and 46.9 clspm, a gain of 11.5 correct letter sounds. 10 RTI International. 2015. Early Grade Reading Assessment (EGRA) Toolkit, Second Edition. Washington, DC: United States Agency for International Development. 29 Figure 9: Letter Sound Learning Gains Class 1 Class 1 27.1 20.9 6.2 Midline Endline Gains Figure 10: Letter Sound Learning Gains Class 2 Class 2 46.9 35.5 11.4 Midline Endline Gains Learning Gains by Gender Based on the gains by gender in Table 17, girls demonstrated higher gains than boys across both classes. Girls in Class 1 identified 9 more letter sounds than boys between midine and endline (gain of 25.9 letter sounds versus 16.9 for boys). In Class 2, girls gained nearly 3 more letter sounds than boys (13.2 letter sounds gained compared to a gain of 10.3 for boys). Therefore, although girls had lower scores at midline, they learned at a faster rate and achieved higher mean scores at endline. Table 17: Letter Sound Learning Gains by Gender Letter Sound Boys Girls Midline Endline Gains Midline Endline Gains Class 1 7.9 24.8 16.9 4.1 30 25.9 Class 2 34.6 44.9 10.3 36.7 49.9 13.2 Learning Gains by Island Group The learning gains by island group for the Letter Sounds subtask is illustrated in Table 18. Three islands showed gains of 23-24 clspm in Class 1: Ha’apai (23.9 clspm), Niuatoputapu (23.4 clspm), and Vava’u (23.2 clspm). Tongatapu gained 20 clspm between midline and endline. However, Niuafo’ou mean scores declined by 1.7 clspm, from 8.9 clspm at midline to 7.2 clspm at endline. In 30 Class 2, Tongatapu achieved the highest gains of 12.7, while the other island group gained between 6 and 8.9 more letter sounds. Table 18: Letter Sounds Learning Gains by Island Group Class 1 Class 2 Subtask Midline Endline Gains Midline Endline Gains Ha’apai 6.9 30.8 23.9 46.5 52.5 6 Niuafo’ou* 8.9 7.2 -1.7 11.4 18.5 7.1 Niuatoputapu 16.3 39.7 23.4 64.8 73.4 8.6 Tongatapu 5.6 25.7 20.1 32.1 44.8 12.7 Vava’u 6.2 29.4 23.2 44.9 53.8 8.9 *Island group is significantly different from the others (p<.05) Learning Gains by School Type Figure 11 shows the Letter Sounds learning gains by school type for Class 1. The treatment group gained over twice as many letter sounds than the control group (28 versus 12.8). Knowledge of letter sounds improved from 5.4 clspm at midline to 33.4 clspm at endline for the treatment group, and from 7.1 clspm to 19.9 clspm, respectively, for the control group. Figure 11: Letter Sound Learning Gains by School Type (Class 1) 33.4 28 19.9 12.8 Treatment 7.1 5.4 Control Midline Endline Gains Class 1 In Class 2, the treatment group scored higher than the control group in both assessments and achieved slightly higher gains (See Figure 10). The treatment group identified 43 clspm at midline and 56.1 clspm at endline, a gain of 12.3 letter sounds. Meanwhile, the control group mean scores increased from 23.6 clspm at midline to 34 clspm at endline, an increase of 10 letter sounds. Overall, the treatment group identified 1.9 more letter sounds than the control group. 31 Figure 12: Letter Sounds Learning Gains by School Type (Class 2) 56.1 43.8 34 23.6 Treatment 12.3 10.4 Control Midline Endline Gains Class 2 FAMILIAR WORD SUBTASK The familiar word subtask measures students’ ability to read familiar words with fluency and accuracy, both of which are necessary to become fluent readers. For this subtask, students were given a sheet of 50 familiar words from grade-level text with instructions to read as many as they could within one minute. The familiar word subtask is scored by the number of correct familiar words read per minute (cfwpm). Overall Learning Gains by Class Gains on the Familiar Word subtask by Class are illustrated in Figure 13. Class 1 students read 1.2 cfwpm at midline and 13.4 cfwpm at endline, resulting in a gain of 12.2 familiar words. Class 2 students read 19.1 cfwpm at midline and 36.8 cfwpm at endline, a total gain of 17.7 cfpm. Hence, Class 2 learned 5.5 more familiar words than Class 1. Figure 13: Familiar Word Learning Gains Class 1 13.4 12.2 Class 1 1.2 Midline Endline Gains 32 Figure 14: Familiar Words Learning Gains Class 2 Class 2 36.8 19.1 17.7 Midline Endline Gains Learning Gains by Gender Table 19 shows the Familiar Word gains disaggregated by gender. In both classes, girls scored lower than boys at midline, but higher at endline, resulting in higher overall gains. Girls gained 6.5 more familiar words than boys in Class 1 and 2.8 more familiar words than boys in Class 2. Table 19: Familiar Words Learning Gains by Gender Familiar Word Boys Girls Midline Endline Gains Midline Endline Gains Class 1 1.8 11.1 9.3 0.4 16.2 15.8 Class 2 16.6 33.2 16.6 22.5 41.9 19.4 Learning Gains by Island Group The Familiar Words learning gains by island group are shown in Table 20. Niuatoputapu island gained 21 familiar words in Class 1 and 22 familiar words in Class 2, ranking the highest of all island groups. The gains were statistically significant for Class 1. Meanwhile, Tongatapu and Niuafo’ou achieved the lowest gains for Class 1 and Class 2, respectively. Table 20: Familiar Words Learning Gains by Island Group Class 1 Class 2 Subtask Midline Endline Gains Midline Endline Gains Ha’apai 0.8 15.8 15 25.5 44.4 18.9 Niuafo’ou 0 11.6 11.6 16.9 34 17.1 Niuatoputapu* 0 21.6* 21.6 35.5 56.6 21.1 Tongatapu 1.5 12.8 11.3 17.2 35.2 18 Vava’u 0.4 12.6 12.2 23.8 39.2 15.4 *Island group is significantly different from the others (p<.05) 33 Learning Gains by School Type The number of familiar words gained by school type for Class 1 is shown in Figure 15. Both groups improved significantly between midline and endline, but the treatment group showed slightly higher gains of 2.6 more familiar words. The treatment group increased from 0.5 cfwpm at midline to 13.9 cfwpm at endline, an increase of 13.4 familiar words. Meanwhile, the control group improved from 2 cfwpm at midline to 12.8 cfwpm at endline, resulting in an increase of 10.8 familiar words. Figure 15: Familiar Words Learning Gains by School Type (Class 1) 13.9 13.4 12.8 10.8 Treatment 2 0.5 Control Midline Endline Gains Class 1 In Class 2, the treatment group read 3 more familiar words than the control group (See Figure 16). The mean scores for the treatment group improved from 20.1 cfwpm to 39.1 cfwpm. The control group scored 17.6 cfwpm at midline and 33.6 cfwpm at endline. The treatment group gained a total of 19 familiar words while the control group gained 16, a difference of 3 familiar words. Figure 16: Familiar Words Learning Gains by School Type (Class 2) 39.1 33.6 20.1 19 17.6 16 Treatment Control Midline Endline Gains Class 2 NONWORD SUBTASK In order to comprehend unfamiliar text, children must be able to decode unfamiliar words. Nonword reading measures students’ ability to decode made-up words that follow the linguistic rules of the Tongan language. Unlike familiar word reading in which students can read from memory or sight recognition, on the nonword reading sub-test students must sound out the words; thus, it is a purer measure of decoding ability. For the Nonword subtask assessment, students were provided with a table of 50 made-up words and instructed to read as many as they could within one-minute. The subtask is measured by the number of correct nonwords per minute (cnwpm). 34 Learning Gains by Class Learning gains on the Nonword subtask are depicted in Figure 17. Class 1 students read 1.8 cnwpm at midline and 8.2 cnwpm at endline, an increase of 6.4 nonwords. Class 2 gained 9.7 nonwords with mean scores increasing from 11.6 cnwpm at midline to 21.3 cnwpm by endline. Class 2 gained 3.3 more non-words than Class 1. Figure 17: Nonword Learning Gains Class 1 Class 1 8.2 6.4 1.8 Midline Endline Gains Figure 18: Nonword Learning Gains Class 2 Class 2 21.3 11.6 9.7 Midline Endline Gains Learning Gains by Gender As illustrated in Table 21, girls achieved higher gains in both classes. In Class 1, girls read 6.1 more nonwords than boys. At midline, girls read 0.3 cnwpm while boys read 3.1 cnwpm. By endline, girls’ scores jumped to 10 cnwpm (an increase of 9.7 nonwords) while boys scores increased to 6.7 nonwords (an increase of 3.6 nonwords). In Class 2, the gains for girls were much smaller (1.5 more nonwords than boys). Girls read 13.4 cnwpm at midline while boys read 10.2 cnwpm. At endline, girls read a total of 24 nonwords (an increase of 10.7) while boys’ scores improved to 19 cnwpm (an increase of 9.2). Table 21: Nonword Learning Gains by Gender Nonword Boys Girls Midline Endline Gains Midline Endline Gains Class 1 3.1 6.7 3.6 0.3 10 9.7 Class 2 10.2 19.4 9.2 13.4 24.1 10.7 35 Learning Gains by Island Group The gains by Island group for the Nonword subtask are presented in Table 22. Niuatoputapu island showed the highest gains in Class 1 (gain of 12 nonwords compared to 5-8 nonwords in other island groups) and the results were statistically significant. In Class 2, Tongatapu achieved the highest gains (10.6 compared to 5-8 nonwords for the other island groups). Meanwhile, Tongatapu showed the lowest gains in Class 1, and Niuafo’ou gained the lowest number of nonwords in Class 2. Table 22: Nonword Learning Gains by Island Group Class 1 Class 2 Subtask Midline Endline Gains Midline Endline Gains Ha’apai 0.6 8.8 8.2 14.9 22.8 7.9 Niuafo’ou 0 7.9 7.9 13.3 17.9 4.6 Niuatoputapu* 0 11.9* 11.9 22.3 30.1 7.8 Tongatapu 2.4 8 5.6 10.2 20.8 10.6 Vava’u 0.3 7.6 7.3 15.6 22.3 6.7 *Island group is significantly different from the others (p<.05) Learning Gains by School Type Figure 19 shows learning gains for the Nonword subtask by school type for Class 1. The treatment group mean scores were considerably lower than the control group at midline (0.3 cnwpm compared to 3.5 cnwpm, respectively). However, both groups scored similarly at endline with mean scores of 8 cnwpm. Overall, the treatment group gained 7.9 nonwords while the control gained 4.6. Thus, the treatment group learned 3.3 more nonwords than the control group. Figure 19: Learning Gains by School Type (Class 1) 8.2 8.1 7.9 4.6 3.5 Treatment 0.3 Control Midline Endline Gains Class 1 The learning gains for Class 2 on the Nonword subtask are shown in Figure 20. Both groups scored similarly at midline with mean scores of 11.7 cnwpm for the treatment group and 11.4 cnwpm for the control group. At endline, mean scores increased to 22.6 cnwpm for the treatment group (a gain of 10.9) and to 19.5 cnwpm for the control group (a gain of 8.1). Based on the gain scores, the treatment group read 2.8 more nonwords than the control group. 36 Figure 20: Learning Gains by School Type (Class 2) 22.6 19.5 11.7 11.4 10.9 8.1 Treatment Control Midline Endline Gains Class 2 ORAL READING FLUENCY SUBTASK The Oral Reading Fluency (ORF) subtask measures students’ ability to read a short story with sufficient speed and accuracy. It encompasses all of the previous reading skills plus the skills needed for comprehension -- the ability to translate letters into sounds, unify sounds into words, process connections, relate text to meaning, and make inferences. Similar to the previous fluency subtasks, the ORF subtask is a timed test to assess students’ fluency rates. Students were asked to read a short grade-level passage within one minute. If the child could not read any words correctly in the first line within the first 10 seconds, the assessor stopped the test early and the child received a score of zero. The mean score is expressed as the correct number of words per minute (cwpm). Learning Gains by Class Oral Reading Fluency learning gains by Class are shown in Figures 21 and 22. Class 1 students gained 17.7 words (from 1.4 at midline to 19.1 at endine). Class 2 students showed a noteworthy increase of 26.2 cwpm from 13.1 cwpm at midline to 39.3 cwpm at endline. Figure 21: Oral Reading Fluency Learning Gains Class 1 Class 1 19.1 17.7 1.4 Midline Endline Gains 37 Figure 22: Oral Reading Fluency Learning Gains Class 2 Class 2 39.3 26.2 13.1 Midline Endline Gains Learning Gains by Gender ORF gains by gender are shown in Table 23. Following the trend of previous subtasks, girls gained more words than boys across both classes. In Class 1, girls read 6.5 more words than boys (with a gain of 15.2 compared to 8.7 for boys). Class 2 girls read 3 more words of the oral passage than boys (with a gain of 21.9 cwpm compared to 18.8 cwpm for boys). Table 23: ORF Learning Gains by Gender ORF Boys Girls Midline Endline Gains Midline Endline Gains Class 1 2.2 10.9 8.7 0.4 15.6 15.2 Class 2 16.5 35.3 18.8 22.9 44.8 21.9 Learning Gains by Island Group ORF gains by Island group are presented in Table 24. In Class 1, ORF scores ranged from 10.9 cwpm in Tongatapu to 20.3 cwpm in Niuatoputapu. ORF scores in Class 2 ranged from 19.6 cwpm in Tongatapu to 22.6 cwpm in Niuatoputapu. Therefore, Niuatoputapu island showed the highest gains in Class 1 (20.3 cwpm compared to 10-13 words in other island groups) and in Class 2 (22.6 cwpm) while Tongatapu had the lowest gains in Class 1 and Niuafo’ou gained the least number of words in Class 2. Table 24: ORF Learning Gains by Island Group Class 1 Class 2 Subtask Midline Endline Gains Midline Endline Gains Ha’apai 1 14.4 13.4 23.2 45.1 21.9 Niuafo’ou 0 11.7 11.7 17.9 37 19.1 Niuatoputapu* 0 20.3* 20.3 37.1 59.7 22.6 Tongatapu 1.7 12.6 10.9 17 36.6 19.6 Vava’u 0.5 12.7 12.2 26.1 48.2 22.1 *Island group is significantly different from the others (p<.05) 38 Learning Gains by School Type ORF learning gains by school type in Figure 23 indicate significant gains for both the treatment and control groups in Class 1. The treatment group read less than half as many words as the control group (0.7 cwpm compared to 2.2 cwpm) at midline. At endline, the treatment group exceeded the control group reading a total of 13.9 cwpm compared to 12.1 for the control group. Overall, the treatment group gained 3.3 more words than the control group, gaining13.2 more words at endline compared to a gain of 9.9 for the control group. Figure 23: ORF Learning Gains by School Type (Class 1) 13.9 13.2 12.1 9.9 2.2 Treatment 0.7 Control Midline Endline Gains Class 1 Class 2 gains by school type are illustrated in Figure 24. In Class 2, the treatment group read 19.6 cwpm at midline versus 18.5 for the control group. At endline, mean scores jumped to 41.3 for the treatment group and to 36.3 for the control group. Overall, the treatment group read 3.9 more words at endline than the control with a gain of 21.7 versus 17.8 for the control group. Figure 24: ORF Learning Gains by School Type (Class 2) 41.3 36.3 19.6 21.7 18.5 17.8 Treatment Control Midline Endline Gains Class 2 READING COMPREHENSION SUBTASK This subtask measured students’ ability to understand the story they had read aloud in one minute. Students were asked from zero to five comprehension questions based on how much of the passage they had read. For instance, if they had not read any of the passage they were not asked any questions; if they had read the first 20 words they were asked the questions that corresponded to that part of the text; and if they read the entire passage, they were asked all five questions. The mean score is calculated as the percentage of correct answers out of a total possible of five questions. 39 Learning Gains by Class Reading Comprehension gains by Class are illustrated in Figures 25 and 26. Reading Comprehension mean scores in Class 1 increased from 0.4% at midline to 16.9% at endline, an increase of 16.5 percentage points. Class 2 students doubled their comprehension score from 25% at midline to 50% at endline. In terms of the number of questions answered correctly, Class 1 students were able to answer 0.8 questions correctly at endline, while Class 2 correctly responded to 2.5 of the 5 questions asked. Based on internal standards, by the end of Class 3, students should be able to respond to 4 out 5 questions (80%); thus if students continue to improve at the same rate, they should be on track. Figure 25: Reading Comprehension Learning Gains Class 1 Class 1 16.9 16.5 0.4 Midline Endline Gains Figure 26: Reading Comprehension Learning Gains Class 2 Class 2 50.2 25.4 24.8 Midline Endline Gains Learning Gains by Gender Table 25 presents the learning gains by gender. In Class 1, boys and girls scored similarly at baseline with mean scores of 0.3% correct for boys and 0.5% for girls. By endline, boys scores increased to 13.6% (a gain of 13 percentage points) while girls’ comprehended 21% (a gain of 21 percentage points). In Class 2, girls scored higher than boys at midline with a mean score of 30% compared to 22% for boys. At endline, mean scores increased to 46.3% for boys and 55.7% for girls. Thus, girls gained a total of 25.7 percentage points while boys gained 24.2 percentage points (a difference of 1.5%). 40 Table 25: Reading Comprehension Learning Gains by Gender Reading Boys Girls Comprehension Midline Endline Gains Midline Endline Gains Class 1 0.3 13.6 13.3 0.5 21.1 20.6 Class 2 22.1 46.3 24.2 30 55.7 25.7 Learning Gains by Island Group Gains by island group are represented in Table 26. The results indicate that Niuatoputapu island expressed the highest gains in Class 1 (gain of 19 percentage points). In Class 2, Tongatapu ranked first with a gain of 26 percentage points, followed closely by Niuatoputapu (with a gain of 25 percentage points) and Niuafo’ou (with a gain of 24 percentage points). However, in terms of overall reading comprehension, Tongatapu ranked the lowest with a mean score of 48% while Niuatoputapu ranked the highest with a mean score of 79% correct. Table 26: Reading Comprehension Learning Gains by Island Group Class 1 Class 2 Subtask Midline Endline Gains Midline Endline Gains Ha’apai 1 17.4 16.4 34.9 54.8 19.9 Niuafo’ou 0 12.5 12.5 30.7 54.9 24.2 Niuatoputapu 0 19.1 19.1 53.7 79.1 25.4 Tongatapu 0.3 17.1 16.8 21.6 48 26.4 Vava’u 0.3 15.9 15.6 37.3 55.4 18.1 *Island group is significantly different from the others (p<.05) Learning Gains by School Type Reading comprehension gains by school type for Class 1 show substantial improvement from midline to endline with minimal differences in gains between the treatment and control groups. At midline, the treatment group averaged 0.5% correct while the control group scored 0.3% correct. Mean scores increased to 17.4% for the treatment group and 16.5% for the control group at endline. Gains were 17% for the treatment group and 16% for the control group, a difference of 1%. Figure 27: Reading Comprehension Learning Gains by School Type (Class 1) 17.4 16.5 16.9 16.2 Treatment 0.5 0.3 Control Midline Endline Gains Class 1 41 Reading comprehension gains for Class 2 show similar scores at midline, but notable gains for the treatment group at endline. At midline, the treatment group scored 25.8% correct for the treatment group and 24.7% for the control group. Mean scores significantly increased to 53.7% for the treatment group and 45.3% for the control group. The treatment group gained 28 percentage points compared to 21 percentage points for the control group, a difference of 7 percentage points. Figure 28: Reading Comprehension Learning Gains by School Type (Class 2) 53.7 45.3 25.8 27.9 24.7 20.6 Treatment Control Midline Endline Gains Class 2 LISTENING COMPREHENSION SUBTASK The Listening Comprehension subtask assessed students’ ability to listen and comprehend an oral story. The assessor read each student a short passage aloud and then asked five comprehension questions. Different from the reading fluency subtask, this subtask is untimed so all students were given an equal opportunity to respond to all five questions. The scores represent the percentage of correct responses from the total possible score. Learning Gains by Class Listening Comprehension gains by class are presented in Figures 29 and 30. Class 1 scored 9.8% correct at midline and 24.6% correct at endline, a gain of 15 percentage points. Meanwhile, Class 2 correctly responded to 26.2% of comprehension questions at midline and 35.8% at endline, gaining 9.6 percentage points. In terms of the number of questions answered correctly, Class 1 students were able to answer 1.23 questions correctly at endline, while Class 2 correctly responded to 1.79 of the 5 questions asked. These scores are low for Class 2 considering listening comprehension is a pre-reading skill. Figure 29: Listening Comprehension Learning Gains Class 1 Class 1 24.6 14.8 9.8 Midline Endline Gains 42 Figure 30: Listening Comprehension Learning Gains Class 2 Class 2 35.8 26.2 9.6 Midline Endline Gains Learning Gains by Gender Table 27 presents Listening Comprehension gains by gender. The results indicate that girls outperformed boys by 2.2 percentage points in Class 1 (gain of 16 versus 13.8 for boys) and by 6.9 percentage points in Class 2 (gain of 13.6 for girls compared to 6.7 for boys). In Class 1, mean scores for boys increased from 10.4% at midline to 24.2% at endline while girls’ scores improved from 9.1% to 25%. In Class 2, boys’ mean scores increased from 28.8% at midline to 35.5% while girls’ scores improved from 22.6% to 36.2%. Table 27: Listening Comprehension Learning Gains by Gender Listening Boys Girls Comprehension Midline Endline Gains Midline Endline Gains Class 1 10.4 24.2 13.8 9.1 25.1 16 Class 2 28.8 35.5 6.7 22.6 36.2 13.6 Learning Gains by Island Group Gains by island group are represented in Table 28. Niuafo’ou island scored the highest gains in Class 1 (gain of 28 percentage points) and the results were statistically significant. In Class 2, Niuatoputapu ranked highest with a gain of 23.7 percentage points, which was also statistically significant. The gains are notable for both classes when compared to the other islands, which ranged from 10-19 percentage points for Class 1 and between 3 and 13.5 percentage points for Class 2. Table 28: Listening Comprehension Learning Gains by Island Group Class 1 Class 2 Subtask Midline Endline Gains Midline Endline Gains Ha’apai* 13 23.2 10.2 38 31.3 -6.7 Niuafo’ou* 4.2 32.5* 28.3* 36.2 49.7 13.5 Niuatoputapu* 23.3 41.8 18.5 29.6 53.3* 23.7* Tongatapu 8.4 24.6 16.2 24.6 36.4 11.8 Vava’u 11.2 20.7 9.5 28.3 31.2 2.9 *Island group is significantly different from the others (p<.05) 43 Learning Gains by School Type Listening comprehension results by school type for Class 1 show similar gains for the treatment and control groups (See Figure 31). At midline, the treatment group correctly responded to 10.4% of questions while the control group scored 9.1% correct. At endline, scores for both groups increased to over 20% (26% for the treatment group and 22.8% for the control group). The treatment group gained 15.7 percentage points while the control group’s scored increased by 13.7 percentage points (a difference of 2 percentage points). Figure 31: Listening Comprehension Learning Gains by School Type (Class 1) 26.1 22.8 15.7 13.7 10.4 9.1 Treatment Control Midline Endline Gains Class 1 In Class 2, the treatment group scored nearly 5 percentage points higher than the control group at midline with a mean score of 28.2% correct compared to 23.4%, respectively (See Figure 32). At endline, the control group exceeded the treatment group by one percentage point (35.4% for the treatment group and 36.4% for the control group), resulting in higher gains for the control group (gain of 13 percentage point versus 7.2 for the treatment group). Thus, the control group scored nearly 5 percentage points higher than the treatment group between midline and endline. Figure 32: Listening Comprehension Learning Gains by School Type (Class 2) 35.4 36.4 28.2 23.4 13 7.2 Treatment Control* Midline Endline Gains Class 2 DICTATION SUBTASK The dictation subtask measures students’ alphabet knowledge and ability to hear and distinguish individual letter sounds in words and to spell words correctly. For this sub-test, the assessor read aloud a short sentence of 15 words and asked students to write down what they had heard. The assessor read the sentence three times, once before students began writing and twice while they were writing. The test was untimed and scored by the percentage of items written correctly. The scores were calculated using weights to arrive at a maximum score of 100 percent. The spelling items received a weight of 70 percent while other components (spacing, capitalization, comma and 44 full stop) received a weight of 30 percent. Given the focus of the EGRA is to assess reading ability, the spelling component received more weight. Learning Gains by Class Dictation learning gains by class are illustrated in Figures 33 and 34. Class 1 achieved significant gains from midline to endline and compared to Class 2. Mean scores dramatically increased from 6.3% at midline to 55.6% at endline, a gain of 49 percentage points. In Class 2, mean scores increased from 69% at midline to 82.6% at endline, a gain of 13.5%. While Class 1 showed the highest gains among all subtasks, Class 2 also had the highest mean scores compared to all other fluency and comprehension subtasks. Figure 33: Dictation Learning Gains Class 1 Class 1 55.6 49.3 6.3 Midline Endline Gains Figure 34: Dictation Learning Gains Class 2 Class 2 82.5 69 13.5 Midline Endline Gains Learning Gains by Gender The Dictation gains by gender are depicted in Table 29. In Class 1, girls scored 8.9 percentage points higher than boys (gain of 54.2 versus 45.3 for boys). Class 2 boys, however, achieved higher gains than girls (gain of 14.4 compared to 12.4 for girls). Table 29: Dictation Learning Gains by Gender Dictation Boys Girls Midline Endline Gains Midline Endline Gains Class 1 5.8 51.1 45.3 6.9 61.1 54.2 Class 2 66.5 80.9 14.4 72.5 84.9 12.4 45 Learning Gains by Island Group Dictation learning gains by island group are displayed in Table 30. The results demonstrate that students in all island groups for Class 1 gained between 47 and 53 percentage points, with Vava’u showing the highest gains (53%). In Class 2, gains ranged between 1.5 and 17.7 percentage points (both of which were statistically significant). Niuafo’ou island achieved the highest gains (17.7 percentage points). Table 30: Dictation Learning Gains by Island Group Class 1 Class 2 Subtask Midline Endline Gains Midline Endline Gains Ha’apai 11.7 58.7 47 76.9 83.6 6.7 Niuafo’ou* 12.5 63.2 50.7 72.7 90.4* 17.7* Niuatoputapu* 14.9 62.7 47.8 86.6 88.1* 1.5* Tongatapu 4.9 53.9 49 67.2 82.1 14.9 Vava’u 7 60.1 53.1 71.7 83.4 11.7 *Island group is significantly different from the others (p<.05) Learning Gains by School Type According to the Dictation results by school type for Class 1 in Figure 35, both groups showed significant gains from midline to endline, but the control group achieved higher gains overall. Mean scores at midline were 6.1% correct for the treatment group and 6.7% correct for the control group. At endline, mean scores increased to 53.3% for the treatment group (gain of 47.2 percentage points) and 58.3% for the control group (gain of 51.6 percentage points). Thus, the control group exceeded the treatment group’s endline score by 5 percentage points and achieved 4.4 more percentage points than the treatment group overall. Figure 35: Dictation Learning Gains by School Type (Class 1) 58.3 53.3 51.6 47.2 Treatment 6.1 6.7 Control Midline Endline Gains Class 1 Dictation learning gains by school type for Class 2 are shown in Figure 36. Gains for Class 2 were much smaller than Class 1 considering the high mean scores at midline. Both groups scored 69% correct at midline. At endline, mean scores increased by 14.3 percentage points for the treatment group (to 83%) and by 12.5% for the control group (to 82%). Therefore, the treatment group scored 1.8 percentage points higher than the control group. 46 Figure 36: Dictation Learning Gains by School Type (Class 2) 83.2 81.6 68.9 69.1 14.3 12.5 Treatment Control Midline Endline Gains Class 2 INTER-SUBTASK CORRELATIONS Table 31 provides the correlation indices between the subtasks in the TEGRA instrument. Indices range from 0 (no relationship) to 1 (absolute relationship), with 0.7 or greater suggesting a strong positive relationship. Cells with correlation indices of 0.7 or greater are shaded in the table for ease of reference. Gains in letter names were strongly correlated with gains in letter sounds, familiar words and nonwords. This means that students who gained more knowledge of letter names tended to show increased knowledge of familiar words and nonwords. Gains in oral reading fluency were positively associated with gains in Letter Sound, Familiar Word and Nonwords subtasks showing that as fluency in lower-level skills increased so did oral reading fluency. While there was strong correlation among the lower and higher-level reading fluency subtasks, there was low correlation between the reading fluency subtasks and reading comprehension. The lower observed correlations does not mean that there is no correlation between letter- and word-reading subtasks and reading comprehension; it could mean that the gains were not significant enough to draw a positive correlation. The same conclusion could be drawn about the dictation subtask. Table 31: Inter-subtask Correlations Sub-tests 7 1 2 3 4 5 6b 8 6a Listen Letter Initial Letter Familiar Non- Read Dict- ORF -ing Name Sound Sound Words words Comp. ation Comp. rs 1 1. Letter Name N 2274 rs .140** 1 2 Initial Sound N 2274 2274 rs .856** .141** 1 3 Letter Sound N 2274 2274 2274 rs .796** .102** .911** 1 4 Famili Words N 2274 2274 2274 2274 rs .825** .049* .958** .949** 1 5 Non Words N 2271 2271 2271 2271 2274 rs .622** .070** .746** .790** .777** 1 6a ORF N 2272 2272 2272 2272 2271 2274 .226* rs .060* .224** .095** .258** .094**2 1 6b Reading Comp. * N 2274 2274 2274 2274 271 2274 2272 47 rs .078** .168** .050* .031 .024 -.012 .104** 1 7 Listening Comp. N 2274 2274 2274 2274 2271 2272 2274 2274 rs .296** .333** .149** .116** .059** .057* .057* .129** 1 8 Dictation N 2274 2274 2274 2274 2271 2272 2272 2274 2274 ** Correlation is significant at the 0.01 level (2-tailed). r s = Pearson correlation value between the sub-tests N = sampled observations. Significant positive correlations occur between sub-tests where rs >0.04. For example, between reading of Familiar words and Reading comprehension there is a significant positive correlation (rs = .258 851, N = 2274, p < .01, two-tailed). 48 Chapter 4 – Student Contextual Factors This section describes the student contextual factors that were collected through the student context questionnaire. Regression analysis did not reveal any statistically significant correlations between any of the factors and reading fluency11; nonetheless, the student contextual factors are important to consider, as they may provide insights into literacy environments at home and at school. The 34 questions on the student questionnaire have been grouped into six categories: 1) household assets, 2) student characteristics, 3) teacher-student interaction, 4) home support, 5) time spent reading, and 6) reading materials at home and school. The results below indicate the proportion of students who responded positively to each question and thus possessed the particular characteristic. Household Assets Table 32 shows the proportion of students possessing each of the household assets listed in the student questionnaire. Household assets have often been used as a proxy for wealth and socioeconomic status. The rationale is that the more assets a student possesses, the higher perceived socioeconomic status the student holds, which in turn has been associated with higher literacy rates. The results show that nearly all (96%) students have electricity in their home. The majority of students have a phone (90%), a refrigerator (83%), and a television (82%). More than three-quarters of students have a radio (78%) and vehicle (78%), and most have a mobile phone (69%). Less common is a toilet inside the house (55%), cable TV (27%) and a motor boat (18%). Overall, the majority of students have seven of the ten household assets, indicating a high degree of perceived socio-economic status. Table 32: Household Assets Household Assets Proportion SE Electricity 96% 1.0% Phone 90% 2.0% Refrigerator 83% 1.7% TV 82% 1.8% Radio 78% 1.1% Vehicle 78% 2.1% Mobile phone 69% 2.4% Toilet inside the house 55% 2.6% Cable TV 27% 1.9% Motor boat 18% 1.4% 11 Please note that regression analysis by subgroup (gender, island group or school type) also did not reveal any statistically significant or strongly correlated findings between variables and oral reading fluency. 49 Student Characteristics Based on the results in Table 33, the majority of students (87%) ate breakfast before arriving to school and taking the TEGRA. Nearly 70% attended an early childhood education program (e.g., kindy, pre-school or nursery) before entering Class 1. However, over half of students (55%) were absent from school at least one day the week before the TEGRA. Only 7% of students assessed had repeated a grade. Table 33: Student Characteristics Student Characteristics Proportion SE Ate any food before arriving at school today 87% 1.2% Attended a kindy, pre-school or nursery school before 69% 1.7% entering Class 1 Were you absent from school any day last week 55% 1.6% Repeater 7% 1.2% Student-Teacher Interaction Table 34 reports how the teacher reacts when the student performs well in class. When students do well on a test or during a lesson, half of the teachers praise students or ask classmates to clap; 10% gives the student a prize; and 11% of teachers do nothing. About 22% of students did not provide any answer. Table 34: What does the teacher do when students do well on a test or during a lesson? What does the teacher do when students do well on Proportion SE a test or during a lesson? Nothing 11% 1.2% Praise me 32% 2.0% Ask classmates to clap 20% 1.5% Gives me a prize 10% 1.4% Other 8% 0.8% Don’t know/no answer 22% 1.6% When the students are unable to answer a question correctly, 46% of teachers correct the student or explain the question again (See Table 35). Seven percent of teachers encourage the student to try again or ask the question to another student. Meanwhile, another 7% of teachers do nothing. Twenty percent of students did not provide an answer. Table 35: What does the teacher do when students are unable to answer a question correctly? What does the teacher do when student are unable Proportion SE to answer a question correctly? Nothing 7% 1.0% Correct the student 30% 1.7% Explain the question again 17% 1.5% Encourage the student to try again 7% 0.9% Ask another student 7% 1.0% Ask the student again 7% 0.6% Other 14% 1.2% Don’t know/no answer 20% 1.6% 50 Home Support Table 36 provides results to the question, “Does anyone help you to do your homework? If yes, who?� According to the responses below, 84% of students receive help with their homework. Of those who receive help from a family member, the majority (54%) receive help from their mother, 19% from their father, 16% from their sister, and 13% receive help from their brother. Table 36: Does anyone help you to do your homework? Does anyone help you to do your homework Proportion SE Yes 84% 1.6% No 16% 1.6% If yes, who? Mother 54% 1.8% Father 19% 1.4% Sister 16% 0.8% Brother 13% 1.0% Any other persons 2% 0.3% Don’t know/no answer 1% 0.2% Table 37 presents results to the question, “Does anyone else know how to read in your family? If yes, who?� About 60% of students have a literate person in their family or household. The most common literate person in the household is the mother (18%) or sister (24%). Only 10% of students have a literate father in the household. Table 37: Does anyone else know how to read in your family? Does anyone else know how to read in your house Proportion SE or family Yes 58% 1.6% No 42% 1.6% If yes, who? Mother 18% 2.0% Father 10% 1.1% Sister 24% 1.4% Brother 18% 1.2% Any other persons 1% 0.3% Don’t know/no answer 0% 0.1% According to the results in Table 38, the majority of parents, both mothers and fathers, do ask students what they did at school when they come home. Table 38: When student comes home from school, does mother and father discuss what he/she did at school? Question Proportion SE When student comes home from school, does mother 92% 3.1% and father discuss what he/she did at school? Because the majority of parents do discuss what occurred at school with their child, most parents are aware of when their children receive a good grade on a school test. When students receive a good grade, 39% of parents give them a reward, 24% of parents congratulate or encourage their child, and 18% of parents do nothing (according to students). 51 Table 39: Last time student got a good grade on a test in school did your parents know that you did well? Last time you got a good grade on a school test did Proportion SE your parents know that you did well? Yes 85% 1.6% No 16% 1.6% If yes, what did they do? Yes, did nothing 18% 1.3% Yes, congratulate or encourage me 24% 1.5% Yes, gave me a reward 39% 1.7% Other 3% 0.5% Don’t know/no answer 7% 1.2% Time Spent Reading Tables 40-42 shows the time spent reading at school and at home. Per the results in Table 40, over 90% of students have time to read books in their classroom or school library every day. At home, about 50% of students sometimes read aloud to someone at home or are read to by someone at home. One-quarter of students never read aloud to someone at home, and 31% are never read to by someone at home. Meanwhile, 26% of students read aloud to someone at home everyday and 20% have someone at home who reads to them on a daily basis. Table 40: Do you have time to read books in your classroom or in your school library every day? Do you have time to read books in your classroom Proportion SE or in your school library every day? Yes 91% 2.0% No 9% 2.0% Table 41: How often do you read aloud to someone at home? How often do you read aloud to someone at home? Proportion SE Never 24% 1.7% Sometimes 50% 1.8% Every day 26% 1.4% Table 42: How often does someone read to you at home? How often does someone read to you at home? Proportion SE Never 31% 1.8% Sometimes 49% 1.7% Every day 20% 1.2% Reading Materials at Home and School According to the responses to questions in Table 43, 67% of students bring reading books home from school. In addition, half of students have access to other reading materials at home (e.g., books, newspapers, etc.). However, the majority of students (63%) do not have the school textbook. 52 Table 43:Reading Materials at home and school Question Proportion SE Do you bring home reading books from your classroom 67% 1.7% or from the school library? Do you have the school textbook? 37% 2.9% Apart from your school textbook, are there other books, 51% 1.7% newspapers or other things to read at your house? Based on the results in Table 44, 45% of students have reading books in the Tongan language and 8% have reading materials in English. Table 44: What language is used in these books? What language is used in these books? Proportion SE Tongan 45% 1.6% English 8% 0.8% Any other language 0% 0.01% Don’t know/no answer 2% 0.4% 53 Chapter 5 - Conclusions Summary of Key Findings Considering the short timeframe between midline and endline (7 months), the learning gains are impressive. Below is a summary of the gains achieved for each class, gender, island group, and school type. ZERO SCORES Class 1 students showed a significant reduction in zero scores of 40-70 percentage points across all subtasks, with the greatest reduction in Dictation (-70.6). Zero scores on fluency subtasks decreased by 60 percentage points and on listening and reading comprehension subtasks by 40 percentage points. Class 2 students reduced zero scores by 2-31 percentage points between midline and endline. The most substantial reduction was in Reading Comprehension (-30.8), where zero scores dropped from 42% at midline to 11% at endline. In Class 1, both boys and girls showed similar rates of reduced zero scores, ranging from 37 to 70 percentage points. Girls showed greater decreases among Familiar Word, Nonword, ORF and Reading Comprehension subtasks. The subtask with the greatest gender disparity was the ORF subtask, in which girls’ zero scores reduced by 14 percentage points more than boys. In Class 2, girls showed a greater reduction of zero scores in four of the nine subtasks while boys showed a greater reduction in four subtasks (Familiar Word, Nonword, Oral Reading Fluency and Reading Comprehension) and both scored similarly on Dictation. The most striking difference in performance was in the Oral Reading Fluency and Reading Comprehension subtasks. On the Oral Reading Fluency subtask, boys’ zero scores dropped by 20% compared to 14% for girls, a difference of 6 percentage points. In Reading Comprehension, the percentage of zero scores declined by 34% for boys and 26% for girls, a difference of 8 percentage points. Of the nine subtasks assessed, the treatment group in Class 1 showed a greater reduction in zero scores among four subtasks (Letter Name, Initial Sound, Letter Sound, and Listening Comprehension) while the control group achieved larger decreases in three subtasks (Familiar Word, Oral Reading Fluency, and Dictation). On the remaining two subtasks (Nonword and Reading Comprehension), both groups expressed similar rates of decline. In Class 2, the control group showed a greater decline in six of nine subtasks, while the treatment group had a greater reduction in zero scores on two subtasks (Letter Name and Reading Comprehension). MEAN SCORES In Class 1, the overall sample increased performance from midline to endline on all subtasks with gains ranging from 6.4 more cnwpm on the Nonword subtask to 30 more clpm on the Letter Name subtask. On the untimed subtasks, gains in Class 1 ranged from 14.8 percentage points in Listening Comprehension to 27.9 percentage points on Initial Sound. The most striking increase was in 54 Dictation mean scores, which jumped from 6.3% at midline to 55.6% at endline – a gain of 49.3 percentage points. Class 2 students exhibited remarkable gains in ORF and Reading Comprehension subtasks with gains of 20 cwpm and 25 additional percentage points, respectively. Students in both classes had the lowest gains in Nonword reading and Listening Comprehension, which are two skills required for reading comprehension. In order to achieve reading comprehension, research 12 shows that students must have strong decoding skills and strong language comprehension abilities. Based on the learning gains by gender, girls performed higher than boys across all subtasks in Class 1 and across eight of the nine subtasks in Class 2 (with the exception of the Dictation subtask). Class 1 girls achieved notably higher gains on the Dictation (+9 percentage points), Letter Name (+7 clpm) and Initial Sound subtasks (+6 percentage points) compared to gains for boys. In Class 2, girls exhibited notable gains in ORF and Reading Comprehension subtasks compared to boys. Class 2 girls read 6.5 more cwpm than boys on the ORF subtask and gained 7 more percentage points than boys in Reading Comprehension. Overall, the results indicate that both boys and girls improved, but girls showed higher gains across all subtasks in Class 1 and across eight of the nine subtasks in Class 2 (with the exception of Dictation). These differences in performance should not be overlooked because they are consistent and can be taken as a sign of systematic, yet not well understood, differences in the learning opportunities and experiences offered to boys and girls. There may be cultural or gender barriers that affect boys’ interest and engagement in reading activities. These results are reflective of international trends, which show that girls consistently outperform boys on standardized tests in the early primary grades in almost every country in the world. 13 There were mixed results by Island Group. In Class 1, Niuatoputapu ranked highest in five subtasks while Vava’u ranked highest on two subtasks (Letter Name and Dictation) and Hapai ranked highest on one subtask (Letter Sound). Three island groups scored similarly for Letter Sound: Hapai (23.8), Niuatoputapu (23.4) and Vava’u (23.2). Finally, Niuafo’ou scored the highest on Listening Comprehension. In Class 2, the island group results were more distinct. Niuatoputapu ranked highest on seven subtasks while Hapai scored highest on one subtask (Letter Name) and Niuafo’ou ranked highest on one subtask (Dictation). The island groups that demonstrated the lowest learning gains were Niuafo’ou and Tongatapu. Niuafo’ou students achieved the lowest gains on three subtasks in Class 1 (Initial Sound, Letter Sound and Reading Comprehension) and four subtasks in Class 2 (Letter Name, Familiar Word, Nonword, and ORF), but they showed the highest gains in Listening Comprehension in Class 1 and substantial gains in Class 2. Tongatapu had the lowest gains in three subtasks in Class 1 (Familiar Word, Nonword and ORF), but showed the highest gains in Reading Comprehension in Class 2. Based on these results, students on the island of Niuatoputapu tend to perform better in reading than students on the other islands. These students had the highest reading fluency and comprehension scores, while students on the island of Niuafo’ou tended to perform the lowest across some subtasks, although they showed the highest gains on listening comprehension in Class 1. These students, therefore, may have had initial difficulties understanding the language of 12 Farrell, L., Davidson, M., Hunter, M, & Osenga, T. (2010). The simple view of reading. Research of importance to all educators. Available at: http://www.cdl.org/articles/the-simple-view-of-reading/ 13 Loveless, T. (2015). Brown Center Report on American Education: How well are American students learning? Washington, DC: Brookings Institute. Available at: https://www.brookings.edu/wp-content/uploads/2016/06/2015-Brown- Center-Report_FINAL-3.pdf 55 instruction, which may have hindered their reading skills development. Meanwhile, Tongatapu island had the lowest gains in three subtasks in Class 1, but showed the highest gains in Reading Comprehension in Class 2. Therefore, the reading instruction in this island was highly effective. When the results are presented by school type, the treatment group achieved higher gains in eight of the nine subtasks in Class 1 and in seven of the nine subtasks in Class 2. In Class 1, students in the treatment group demonstrated the greatest gains on the Letter sound and Initial Sound subtasks. Class 1 students identified 15 more clspm and scored 9 percentage points higher on the Initial Sound subtask compared to the control group. In Class 2, students in the treatment group exhibited the highest gains in Reading Comprehension, scoring 7 more percentage points than the control group. The student contextual factors indicate that on average students do have a supportive literacy environment at school and at home. The majority of students have time and materials to read at school. At home, most students receive help with homework from a literate family member and are supported to read books that they bring from school or have at home. However, the majority of students (63%) do not have a school textbook. As part of the ongoing World Bank impact evaluation of the CLRW, classroom observation data from monitoring and coaching may be used to further assess the quality of reading instruction, availability of teaching and learning materials and assessment methods. Conclusions and Recommendations Learning gains in Class 1 and Class 2 students during the 2017 school year are evident in the decreasing proportion of zero scores and increasing mean scores from midline to endline. As students progress through the grades, they are acquiring more knowledge of letters, words and decoding skills resulting in increased fluency and comprehension. The progress achieved in Class 1 (e.g., learning letter names, initial sounds and letter sounds) serves as a stepping-stone for students to become more fluent readers in Class 2. The fact that fluency in letter sound knowledge manifests into more fluent readers with higher comprehension rates in Class 2 confirms lower-level skills are predictors of reading fluency. The difference in treatment and control groups gains demonstrates that improvements can be made to strengthen reading instruction and outcomes in both grades. It is important to note that both the treatment group and control group improved from Class 1 to Class 2, and Class 2 gains were small, however notable in letter sounds and initial sounds indicating some effect from the intervention. What is most impressive is the rate at which students are advancing in reading comprehension. If students continue to advance at the same rate, they will be well within international standards for fluency (60 cwpm) and comprehension (80% correct). Nonetheless, weaknesses in decoding and listening comprehension could affect comprehension of more difficult content and thus need to be strengthened before the end of Class 3. Additionally, differences between gender, island groups and school types will need to be addressed in order to achieve equitable outcomes in Oral Reading Fluency and Comprehension. Finally, it is recommended that the World Bank Impact Evaluation team complement these findings with classroom observation monitoring data to explore the quality of instruction, availability of textbooks and reading materials, and reading assessment methods, in order to identify more concrete ways to improve reading instruction and overall learning outcomes. 56 Annex: Test reliability measures Annex Table 1 presents indicators of test reliability for the second round of the 2017 TEGRA. The first indicator of reliability is the “item -test� correlation which is the correlation between each sub - domain and a composite measure. The composite measure is the sum of the standardized scores of each sub-domain following the Cronbach’s Alpha methodology. The second indicator is the “item-rest� correlation which is the correlation of each sub-domain with a composite measure excluding the sub-domain. The composite measure for this second indicator is the sum of the standardized scores for all sub-domains excluding the sub-domain in question. These two indicators help identify sub-domains that are less correlated with the EGRA test as a whole in order to identify potential outlier sub-domains. Finally, Cronbach’s Alpha is calculated for the sub - domains that are not timed; these are scored as percent correct in EGRA. RTI (2009:82) does not recommend the Cronbach’s Alpha test for timed sub-domains as this may inflate the measure of test reliability14. This indicator provides an overall measure of the correlation between the non- timed sub-domains; a typical benchmark in research studies is 0.7. For the October 2017 TEGRA, initial sounds and listening comprehension stand out as being less correlated with the other domains. Their item-test correlations are 0.54 and 0.61, respectively, compared to a range of 0.77 to 0.89 for the other domains, and their item-rest correlation is 0.42 and 0.51, respectively, compared to a range 0.70 to 0.85 for the other domains. However, Cronbach’s Alpha for the non-timed sub-domains which includes these two sub-domains is 0.74 and exceeds the typical reliability benchmark of 0.7. Annex Table 1. Reliability measures Item-test Item-rest correlation correlation Correct letters per minute 0.77 0.70 Correct letter sounds per minute 0.73 0.64 Correct words per minute 0.88 0.84 Correct invented words per minute 0.49 0.36 Oral reading fluency 0.65 0.54 Initial sounds (percent correct) 0.74 0.65 Reading comprehension (percent correct) 0.85 0.80 Listening comprehension (percent correct) 0.51 0.38 Dictation (percent correct) 0.71 0.61 Cronbach's alpha (percent correct items only) 0.77 14 RTI (2009). Early Grade Reading Assessment Toolkit. Research Triangle Park, N.C.: RTI International 57