64801 Report No. 64801-TH Learning Outcomes in Thailand What Can We Learn from international Assessments? January 2012 East Asia and the Pacific Region and Human Development Network, Education The World Bank i Learning Outcomes in Thailand What Can We Learn from International Assessments? Disclaimer This volume is a product of the staff and consultants of the international Bank for Reconstruction and Development / The World Bank. The findings, interpretations, and conclusions expressed in this report do not necessarily reflect the views of the Executive Directors of The World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgement on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. January 2012 East Asia and the Pacific Region and Human Development Network, Education The World Bank ii Learning Outcomes in Thailand What Can We Learn from International Assessments? Table of Contents Acknowledgements .......................................................................................................................... VII Abbreviations and Acronyms ........................................................................................................ VIII Executive Summary ........................................................................................................................... 1 Chapter 1 : Improvement in education quality for Thailand’s disadvantaged groups? A first look at PISA 2009 ................................................................................................................... 7 Trend Reversal? ............................................................................................................................ 7 Disadvantaged Groups ................................................................................................................. 10 References .................................................................................................................................... 13 Appendix: Production function modeling to account for differences in student backgrounds .... 14 Chapter 2: What International Student Assessments Tell Us about Improving Basic Education in Thailand ............................................................................................................ 19 introduction................................................................................................................................... 19 Identifying the Challenges ........................................................................................................... 20 Conclusions .................................................................................................................................. 26 References .................................................................................................................................... 27 Annex 1......................................................................................................................................... 28 Chapter 3: Educational Quality in Thailand as Measured by International Tests (PISA and TIMSS) ............................................................................................................................ 31 introduction................................................................................................................................... 31 The Test Results ........................................................................................................................... 32 Trends from PiSA 2000 to PiSA 2006 ......................................................................................... 33 Trends from the National Tests .................................................................................................... 34 How the Other Countries in the Region are Faring ..................................................................... 35 Results from TiMSS..................................................................................................................... 36 Performance Distribution of Thai Students ................................................................................. 36 Disparities in Performance between Geographic Areas .............................................................. 38 Urban and Non 0urban Schools ................................................................................................... 38 Big Schools and Small Schools ................................................................................................... 38 Public and Private Schools .......................................................................................................... 39 Performance Differences between Girls and Boys in Thailand.................................................... 40 Gender Differences in Other Countries ....................................................................................... 41 Trends in the Gender Gap in Thailand between PISA 2000 and PISA 2006 ............................... 42 The Analysis of School Variables and Student Performance........................................................44 School Factors .............................................................................................................................. 44 iii Learning Outcomes in Thailand What Can We Learn from International Assessments? Admitting, Grouping, and Selecting Policies .............................................................................. 45 Institutional Differentiation and Grade Repetition....................................................................... 46 Ability Grouping Within Schools ................................................................................................ 47 The Relationship between School Admittance, Selection, Ability Grouping and Student Performance .................................................................................................................... 48 School Management and Funding: Public and Private Financing ............................................... 48 Performance of Students from Public and Private Schools ......................................................... 50 Parental Influence on Schools ...................................................................................................... 50 Accountability .............................................................................................................................. 51 Giving Feedback on Student Performance to Parents .................................................................. 52 The Impact of Accountability Policies ......................................................................................... 52 School Management and School Autonomy ................................................................................ 52 School Resources and their Impact on Learning .......................................................................... 53 Human Resources: Qualified Teachers ......................................................................................... 53 Material Resources ....................................................................................................................... 53 How Resources have changed between PISA 2000 and PISA 2006 ............................................ 55 Computers for Instruction ............................................................................................................ 56 The Relationship between School Resources and Student Performance ..................................... 57 Conclusion and Implications for Policy ....................................................................................... 58 References .................................................................................................................................... 61 Annex 1......................................................................................................................................... 62 Annex 2......................................................................................................................................... 67 Annex 3......................................................................................................................................... 71 iV Learning Outcomes in Thailand What Can We Learn from International Assessments? List of Tables Table 1.1 : 2006 to 2009 Reading, Math, and Science Change 8 Table 1.2 : Urban and Rural Differences in PISA Reading 11 Table 1.3 : Wealth Differences in PISA Reading 12 Table 1.4 : Gender Differences in PISA Reading 13 Table 1.5 : Oaxaca-Blinder decomposition 16 Table 1.6 : Linear Regression Estimates 17 Table 2.1 : Over-time Oaxaca-Blinder Decomposition of PISA Reading Scores 29 Table 2.2 : Urban Rural PISA reading Oaxaca-Blinder decomposition 29 Table 3.1 : Students’ Performance in PISA 2006 and TIMSS 2007 62 Table 3.2 : Percentage of Students at Each Proficiency Level on Science Scale 62 Table 3.3 : Trends in Thailand’s National Test Results for Grade 9 Students 62 Table 3.4 : Trends from PISA 2000 to PISA 2006 63 Table 3.5 : Students’ Performance in Science of by Level of Parents’ Education in Thailand and Korea 63 Table 3.6 : Differences in Test Scores between Groups of Schools in Thailand 64 Table 3.7 : Percentage of Thai Students at Low and High Reading Proficiency Levels by Gender 64 Table 3.8 : Performance of Thai Boys and Girls in PISA 2006 65 Table 3.9 : Gender Difference in Science in PISA 2006 and TIMSS 2007 in Asian Countries 65 Table 3.10 : Performance of Boys and Girls from PISA 2000 to PISA 2006 65 Table 3.11 : How the Gender Difference 66 Table 3.12 : Simple Regression Analysis of Selected School Variables (Thailand) 66 Table 3.13 : Multilevel Models - Admitting, Grouping and Selecting 67 Table 3.14 : Multilevel Models - School Management and Funding 67 Table 3.15 : Multilevel Models - Parental Pressure and Choice 68 Table 3.16 : Multilevel Models - Accountability Policies 68 Table 3.17 : Multilevel Model - School Autonomy 69 Table 3.18 : Multilevel Model - School Resources 70 List of Figures Figure 1.1 : East Asia PISA Reading Overtime 8 Figure 1.2 : Change in distribution by reading proficiency level 9 Figure 1.3 : How has the tested student population changed from 2006 to 2009? 9 Figure 1.4 : How much do changes in the student population explain change over time? 10 Figure 2.1 : PISA Reading Performance over Time – Selected Countries 20 Figure 2.2 : TIMSS Mathematics Performance over Time – East Asia 20 Figure 2.3 : TIMSS Intermediate Benchmark Attainment – East Asia 21 V Learning Outcomes in Thailand What Can We Learn from International Assessments? Figure 2.4 : Thailand Benchmark Attainment over Time 21 Figure 2.5 : Reading Performance Differences over Time by Household Wealth 22 Figure 2.6 : Decomposing the Over-time Decline in PISA Reading Scores from 2000 to 2006 22 Figure 2.7 : Bangkok versus the Rest of Thailand 23 Figure 2.8 : Decomposing Differences in PISA Reading Scores by Urban-Rural Location 23 Figure 2.9 : Percentage Increase in the Number of Grade 8 Students Achieving the TiMSS intermediate Benchmark in Mathematics 24 Figure 2.10 : Increases in Figure 2.9 Attributable to Changes in Quality and Enrollment 24 Figure 2.11 : Increases in Figure 2.9 Attributable to Urban and Rural Areas 25 Figure 2.12 : Kernel Distribution of Achievement for Bangkok, the Rest of Thailand, and the United States 26 Figure 3.1 : Asian Students’ Performances in PISA and TIMSS 32 Figure 3.2 : Proportion of Students in Asian Countries at Each Reading Proficiency Level 33 Figure 3.3 : Change in Thailand’s Score between PISA 2000 and PISA 2006 34 Figure 3.4 : Performance Trends from Thailand’s National Test 35 Figure 3.5 : Change in Reading, Mathematics, and Science Scores 35 Figure 3.6 : Changes in Science Score from TIMSS 1995 to TIMSS 2007 36 Figure 3.7 : Association of Mother’s Education with Student Performance (Thailand) 36 Figure 3.8 : Education of the Mothers of Thai and Korean Students, Primary or Secondary level (ISCED1, 2) and Tertiary Level (ISCED 5, 6) 37 Figure 3.9 : Change in the Performance of the Top and the Bottom Performers 38 Figure 3.10 : Change in Student Performance in Urban and Non-urban Schools 39 Figure 3.11 : Change in Student Performance in Big Schools and Small Schools 39 Figure 3.12 : Change in Student Performance between Public and Private Schools 40 Figure 3.13 : The Performance of Boys and Girls in Thailand in PISA 2006 40 Figure 3.14 : Gender Differences in Science Scores in PISA 2006 and TIMSS 2007 41 Figure 3.15 : Gender Gaps between Thai Boys and Girls from PISA 2000 to PISA 2006 42 Figure 3.16 : Change of Gender Differences from PISA 2000 to PISA 2006 43 Figure 3.17 : Ability Grouping in Different Groups in Thailand 47 Figure 3.18 : Sources of Funding for Schools in Thailand 49 Figure 3.19 : Relationship between Teacher Shortage Index and Science Score 53 Figure 3.20 : Relationship between Resource Availability Index and Science Score (Thailand) 54 Figure 3.21 : The Availability of Resources among Different School Groups (Negative index represents the resource shortage) 54 Figure 3.22 : Performance of Students in Different School Groups in Science 55 Figure 3.23 : Change in the Resource Availability Index of the Best and Worst Performing Schools 56 Figure 3.24 : Change in the Resource Availability Index of Big and Small Schools 56 Figure 3.25 : ICT and Student Performance, OECD average 57 Vi Learning Outcomes in Thailand What Can We Learn from International Assessments? Acknowledgements This report was prepared by a team led by Harry Anthony Patrinos and consisting of Kevin Macdonald and Suhas Parandekar, under the guidance of Elizabeth King, Emmanuel Jimenez, Robin Horn and Eduardo Velez. Comments and feedback from workshop participants in Bangkok and Washington DC are gratefully acknowledged, especially from Shaista Baksh, Deborah Bateman, Raja Bentaouet Kattan, Emily Brearley, Hana Polackova Brixi, Sofia Busch, Annette Dixon, Pichaya Fitts, Juliana Guaqueta, Maria Ionata, Chutima Lowattanakarn, Vachraras Pasuk- suwan, Piriya Pholphirul, Emilio Porta, Omporn Regel and Mathew Verghis. This report summarizes the research funded by an EPDF grant. The grant funded three individual studies and a series of dissemination and policy development workshops. Chapter 1 was written by Kevin Macdonald. Chapter 2 was written by Kevin Macdonald, Harry Anthony Patrinos and Suhas Parandekar. Chapter 3 was written by Sunee Klainin of the PiSA Thailand National Centre, Institute for the Promotion of Teaching Science and Technology, Bangkok, Thailand. The grant has facilitated dialogue on quality of education issues in both countries, in Indonesia where the portfolio is very active, and in Thailand where the work follows on the recommendations from the secondary education work, which highlighted the need for more attention to quality of education. Thai policy in recent years addresses these key issues, as they focus on expanding access to higher levels, reducing the cost of attendance, and improving quality. The grant has also generated active interest in learning from international experience and using the results of comparisons of policies in other countries to inform domestic policy. This is facilitated through other projects such as System Assessment and Benchmarking for Education Results (SABER), an HDNED-activity which is being piloted with the close cooperation of the Education team in East Asia and the Pacific, and the Korea funded program that brings policymakers in the region to the annual Global Human Resources Forum in Seoul each fall. The Benchmarking program received its impetus from this grant. Presentations of comparative work using the international assessments led to demands for more systematic knowledge on policies that produce results. Thus, we used the grant to initiative the East Asia pilot of the global benchmarking project. The systems approach developed in the benchmarking program is closely aligned with the Bank’s new education sector strategy, Learning for All: Investing in People’s Knowledge and Skills to Promote Development. The demand for the benchmarking work in East Asia led to the development of a specific project, initially supported by this grant, now financed by the Korea Trust Fund in the region, which was launched with the conference in Singapore in June 2010. Vii Learning Outcomes in Thailand What Can We Learn from International Assessments? Abbreviations and Acronyms BKK Bangkok Metropolitan Administration BMA Schools run by The Bangkok Metropolitan Administration CONAFE Secondary schools run by universities EPDF Education Program Development Fund ESCS Economic, Social and Cultural Status ICT Information and Communication Technology IPST Institute for the Promotion of Teaching Science and Technology ISCED International Standard Classification of Education LOC Local Education Administration OBEC Office of Basic Education Council OECD Organization for Economic Co-operation and Development PISA Program for International Student Assessment PRV Private Schools SABER System Assessment and Benchmarking for Education Results SATiT Secondary schools run by universities TiMSS Trends in Math and Science Study VOC Vocational institutions Viii Learning Outcomes in Thailand What Can We Learn from International Assessments? Executive Summary Education quality is crucial to Thailand’s future economic success. While lower-income countries in East Asia are experiencing a bulge in their youth population, Thailand’s youth labor force is expected to decline by 10 percent over the next decade. As a result, the labor-intensive comparative advantage that contributed significantly to Thailand’s past economic performance will diminish. This means that it is essential for Thailand to develop the human capital of its declining young work force to ensure the country’s future competitiveness and economic growth. Education is a significant component of human capital development, the economic benefits of which are firmly established in the policy literature to have a positive effect on economic growth. However, the success or failure of education in terms of increasing productivity and growth depends crucially on its quality. Thailand has participated in two international student assessments to measure the quality of education: OECD’s Program for International Student Assessment (PISA) and the IEA’s Trends in Mathematics and Science Study (TIMSS). 2006 marked a turning point in Thailand with a break in the consistent decline in learning achievement. Figure 1 presents PISA reading achievement for East Asian participants. Achievement in PISA reading had declined consistently from 2000 to 2006 in Thailand. However, from 2006 to 2009, learning achievement increased slightly, suggesting that the decline in learning achievement may have stopped. There exist large disparities in learning achievement between Bangkok and the other areas in Thailand. Figure 2 compares the distribution of learning achievement in Bangkok with that of the other areas of Thailand. As demonstrated, the distribution in Bangkok greatly exceeds that of the rest of Thailand matching that of the United States. 1 Learning Outcomes in Thailand What Can We Learn from International Assessments? Too many Thai 15 year-olds have insufficient reading ability. This is shown in Figure 3 which documents the distribution of students according to PISA’s proficiency levels. These proficiency levels are criterion-references that describe what students are able to do given their performance on the PISA tests. For example, in 2009, 11 percent of students scored below Level 1A where students are “able to recognize the main theme in a text about a familiar topic.� Still, this is lower than the 15 percent recorded in 2006. However, in 2009, nearly half of all 15 year-old students were unable to “locate information which may need to be inferred or meet several conditions.� This also declined slightly from its 2006 level. These levels of achievement signal a high level of functional illiteracy in the future labor force. Improvement among the poorest 50 percent of the student population explains the reversal of the downward trend in learning achievement. Among the richest 50 percent of students, PISA reading achievement declined by 2 points from 2006 to 2009 while learning achievement increased by 10 points for the poorest 50 percent of students resulting in the overall increase of 4 points. In 2006 there was a substantial gap in reading achievement between the richest and poorest students. This gap still exists even when controlling for differences in the household factors related to learning between these two groups; consequently, a large portion of the gap between the richest and poorest students can be attributed to differences in the quality of the education that the poorest have access to. However, from 2006 to 2009 this gap reduced. It also reduced when controlling for differences in household factors related to learning which suggests that the quality of the education system for the poorest 50 percent has increased creating a more equitable distribution of schooling quality in Thailand. 2 Learning Outcomes in Thailand What Can We Learn from International Assessments? The decrease in learning achievement until 2006 is attributed to declines in the quality of the education system and not to changes in the student population. Figure 4 presents a decomposition of the difference in learning achievement between 2000 and 2006 into components explained by differences in background characteristics of students and those due to the value-added or quality of the education system. While PiSA reading achievement declined from 433 to 418 points between 2000 and 2006, a 6 percent increase in achievement would have occurred had the ability of the education system to transform student characteristics into learning remained the same over time. in fact, the student population improved in terms of background characteristics such as parental education. The ability of the education system to transform these personal “inputs� into learning offset this increase by 22 points resulting in a net decline in learning. Thus it is the education system that is responsible for the decline in achievement from 2000 to 2006. There was no decline in the quality of the education system after 2006. From 2006 to 2009 there was a small increase in PISA reading. In the decomposition presented in Figure 4, above, there was a large decrease in the quality of the education system offset by a small increase in the student background characteristics. To confirm that the small increase in reading achievement from 2006 to 2009 was not the result of a large decline in the quality of the education system offset by a large improvement in the background characteristics of the student population, Figure 5 presents the same decomposition. As shown, there is no significant decline in the quality of the education system from 2006 to 2009. 3 Learning Outcomes in Thailand What Can We Learn from International Assessments? The difference in learning achievement Figure 6. Difference in PISA reading between urban and rural between urban and rural areas is areas attributed mostly to differences in system attributed primarily to differences quality and not student characteristics in the quality of the education system and less to differences in the student population. Figure 2, above, reveals a large disparity in learning achievement between Bangkok and elsewhere in Thailand. Figure 6 presents a decomposition of PISA reading achievement between urban and rural areas. The difference in learning achievement is 44 points. However, differences in background characteristics account for only 13 points of that difference; the remaining 31 points are due to differences in the education system’s ability to translate students’ backgrounds into learning. There are large inequalities in resources by school type. Figure 7 presents the PISA school resource and teacher shortage indices for each type of school in Thailand. A lower level on the index represents large shortages. For example, principals from the Office of Basic Education Council (OBEC 1) schools reported high levels of both resource shortages and teacher shortages that inhibit learning. The Bangkok Metropolitan Administration (BKK) schools also reported high teacher and resource shortages. Both of these types of schools are low achieving and cater to students from lower socio-economic backgrounds. The OBEC 1 schools are based in rural areas, while BKK schools have students generally from disadvantaged backgrounds. Public vocational schools (VOC 2) also report high levels of teacher and resource shortage, though less than OBEC 1 or BKK schools. Conversely, SATIT schools; high achieving schools run by the universities, tend to enjoy relatively fewer shortages of either teachers or resources. In general, high performing schools tend to be facing less resource constraints then low performing schools that cater to disadvantaged segments of the population. This helps explain both the urban / rural and wealth differences presented previously. 4 Learning Outcomes in Thailand What Can We Learn from International Assessments? A menu of policy options The results of this study warrant consideration of targeting disadvantaged students and schools. The above findings reveal major disparities in learning achievement and in the quality of schooling between urban and rural areas as well as between rich and poor students. Resource and teacher shortages were felt hardest in the poorest performing schools. However, there have been recent improvements in learning achievement among the poorest 50 percent of the population, as well as improvementsfor the poorest performing students which need to be understood better. If the education system of the rest of Thailand performed as well as Bangkok, Thailand’s education system would perform as well as the United States. To achieve this however, policy makers would need to focus on targeting poor performing schools and students. There are several international examples of compensatory and targeted programs that have been effective at improving education outcomes. For example, Chile’s P-900 program was shown to have improved test scores by 0.2 standard deviations by providing technical and material support to the lowest ranking 10 percent of schools in the country. Mexico’s CONAFE program targeted schools in poor communities and was also shown to improve test scores. This study also supports policy dialogue on improving accountability mechanisms for publically- funded private schools. The above findings reveal that public schools outperform private schools even though private schools receive public funding. The implication is that the accountability mechanism for private schools that receive public funding needs to be improved in order to ensure that private schools are providing high quality services. There are several international examples of strong accountability mechanisms for publically funded private schools. One example is the voucher scheme used in the Netherlands. This provides schools with equal funding per student with which schools have considerable freedom on how to use this funding; however, they must meet specific performance requirements. School performance is also a condition for contract renewal of private operators running public schools in Colombia. 5 Learning Outcomes in Thailand What Can We Learn from International Assessments? Policy dialogue on selection into vocational schools is also supported by this study. Public vocational schools are poor performing and do not qualify their graduates to attend post secondary education; however, they report to have less of a shortage of resource than OBEC 1 and Bangkok Metropolitan Authority schools which generally cater to disadvantaged students. International studies show that delaying the age at which children are tracked into vocational programs improves their learning achievement. Policy makers may wish to consider the resources they allocate to vocational tracks and whether students should be tracked in vocational schools later – currently this happens at age 14 in Thailand. Poland delayed comprehensive education until after age 15, and this resulted in significant improvements in learning achievement for students that would otherwise have been in vocational programs. Finland implemented a similar reform in the 1970’s and is among the top performing countries in PiSA. Next steps This study helps identify the strengths and weaknesses of Thailand’s education system in terms of learning achievement. These strengths and weaknesses imply several future research priorities. 1. Examine Bangkok’s education system: As shown above, the education system in Bangkok is shown to be equally as good as that of the United States, while the rest of Thailand lags behind. There are potentially many lessons that can be transferred from Bangkok to other areas of Thailand to improve education quality. 2. Examine recent improvements in learning achievement among the poorest: There was a significant increase in learning achievement for the poorest 50 percent in Thailand from 2006 to 2009. Understanding how this occurred may help policy makers identify and strengthen the programs or practices that have been successful at improving education quality for disadvantaged groups. 3. Comparing education policies and practices internationally: While there is a lot to learn within Thailand on how to improve education quality, using the policies and practices of other countries as benchmarks will also help guide Thai policy makers on how to continually improve their education system. The application of the Bank’s new instrument, System Assessment and Benchmarking for Education Results (SABER) and more detailed comparison work can provide this type of benchmarking. 6 Learning Outcomes in Thailand What Can We Learn from International Assessments? Chapter 1: Improvement in education quality for Thailand’s disadvantaged groups? A first look at PISA 2009 The quality of education is crucial to Thailand’s future. The country’s youth labor force is projected to decline 10 percent by 2018, while neighboring lower middle income countries are experiencing a youth bulge; as a result, the labor-intensive competitive advantage that helped drive Thailand’s past economic success is now subsiding, increasing the relative importance of human capital (World Bank 2008) and the requisite education quality. Additionally, ensuring equal access to high quality education has been recognized by the Government of Thailand as crucial for political stability and peace. Previous studies of education quality in Thailand using international assessment data have revealed large disparities between rich and poor, and between those residing in Bangkok and other urban areas, and those in rural areas. They have also revealed a consistent decline in education quality since 2000. But the recent release of the OECD’s Program for International Student Assessment (PISA) has shown a slight increase in PISA achievement, suggesting an end to, and even the reversal of this. This paper takes an initial look at how the disparities in education quality have changed since 2006. There is evidence of notable progress towards reducing the gap in the standard of education received by disadvantaged groups. The five main findings are: 1. The percent of poor performing students has dropped from 15.6 to 11.1; 2. There has been a minor increase in reading score which seems to be explained by changes to the education system, and not just changes to the student population’s background; 3. The minor increase in reading achievement was driven by an increase in achievement in rural areas; 4. The reading achievement gap between the wealthiest 50 percent and poorest 50 percent has declined by about 25 percent; and, 5. Men, who typically lag women in all subjects in PISA, witnessed a sizeable reduction in the gap for reading achievement by about 30 percent. Trend Reversal? PISA reading achievement has increased since 2006, breaking the downward trend since 2000. As with the first round of PISA in 2000, the focus of PISA 2009 was reading, and there now exists nearly a decade of comparable reading scores. Figure 1.1 presents PISA reading from 2000 to 2009 for East Asia participants. High income countries and regions such as Korea, Hong Kong, Singapore 7 Learning Outcomes in Thailand What Can We Learn from International Assessments? and Japan are clustered above 500, while Thailand and Indonesia are lower, at around 400. Shanghai, China, had the highest reading score in East Asia. As shown in the figure, though Thailand’s score has been declining since 2000, since 2006 there has been an increase which suggests the end of the downward trend and may indicate a reversal. From 2006 to 2009, there have been small but statistically inconclusive increases in PISA results in Thailand. Table 1.1 presents the differences between 2006 and 2009 for PISA math, reading, and science. Table 1.1: 2006 to 2009 Reading, Math, As shown in Table 1.1, the increase in and Science Change reading score from 2006 to 2009 was Math Reading Science the largest of the three subject areas 2006 417.07 416.75 421.01 with a difference of 4.62 points; however, this difference is not statistically 2009 418.58 421.37 425.30 significant. Science had a similar Difference 1.51 4.62 4.28 increase in score, and mathematics (4.19) (4.07) (3.84) saw a small increase. Source: author’s calculations PISA 2006 and 2009 The percent of lowest achieving students has been reduced. Figure 1.2 presents how the distribution of reading achievement has changed since 2006. The percent of students below proficiency level 1A (those unable to recognize the main theme in a text about a familiar topic) has declined from 15.6 percent to 11.1 percent. This is evidence that the gap in learning attainment for disadvantaged groups has declined. However, the figure also shows that the percent of students at the highest levels of proficiency have also declined. 8 Learning Outcomes in Thailand What Can We Learn from International Assessments? In 2009, more 15 year-olds enrolled in school were from better educated families, and attended schools in urban areas than in 2006. The change in PISA scores overtime (or lack of a decline) can be attributed to either the system being able to better educate the student population, but also to changes in the backgrounds of the student population. Figure 1.3 describes how the student population targeted by PISA (after non-response) has changed since 2006. As can be seen, the number of students has increased from 644,000 to 692,000. The number of students in rural schools has declined, possibly as a result of internal migration, and the proportion of students with a highly educated mother (as a proxy for socio-economic status) has increased from 38 percent to 42 percent. 9 Learning Outcomes in Thailand What Can We Learn from International Assessments? Figure 1.4 shows that improvements in the household and personal characteristics of the student population from 2006 to 2009 increased PISA reading scores by 2.9 points. The ability of the education system to transform these characteristics into learning added an additional 1.7 points. This Oaxaca-Blinder decomposition is based on a cognitive production function conceptualization of learning achievement: household and personal characteristics are inputs into learning, while the education system adds value by transforming these inputs into learning. Improvements over time can either occur because of an improvement in the household or personal characteristics -- the inputs into learning -- or because of improvements in the education system’s ability to transform these characteristics into learning. The Appendix contains the full specification of the model and decomposition. The second study of this report (see below) found that from 2000 to 2006, there was an improvement in household characteristics of the student population but a large offsetting decrease in the value-added of the education system using this same type of decomposition. Even though the increase in reading achievement was small from 2006 to 2009, Figure 1.4 shows that there was no significant decrease in the value-added of the education system. Source: author’s calculations using a Oaxaca-Blinder decomposition with Thailand PISA 2006 and 2009. See the Appendix Table 1.5 for full specification Disadvantaged Groups The small increase in reading was driven by rural, not urban areas. Table 1.2 presents the estimates of two linear regressions where a rural school is one with a community of less than 15,000 people. The first regression model is defined to estimate urban and rural differences in 2006 and 2009, as well as the over-time change in these differences; the second regression estimates the same thing, but this time controlling for student background characteristics such as relative wealth, education of the child’s mother, the number of books at home, and gender. As shown in the table, the change in urban reading achievement from 2006 to 2009 is negative, -2.28, while for rural areas, the change in achievement is positive, at 5.55 points. However, in both cases, the changes are small and statistically insignificant. Also, the urban rural difference in 2006 is 44 points, and in 2009 it is 36 points; however, it is not statistically conclusive that this gap was reduced. 10 Learning Outcomes in Thailand What Can We Learn from International Assessments? Table 1.2: Urban and Rural Differences in PISA Reading Regression Model: (1) (2) Change in urban score from 2006 to 2009 -2.28 -6.60 (5.92) (4.57) Urban and rural difference in 2006 -43.59*** -20.32*** (7.02) (5.77) Change in urban and rural difference 7.83 8.51 (9.62) (7.66) Controlling for student background characteristics No Yes Constant 437.23*** 348.89*** (7.02) (19.99) Observation 12,417 12,061 R-Square 0.07 0.25 Change in rural score from 2006 to 2009 5.55 1.90 (6.71) (5.70) Urban and rural difference in 2009 -35.75*** -11.82** (5.18) (4.78) Source: author’s calculations using Thailand PISA 2006 and 2009. Standard errors noted in brackets. Statistical significance at the 1, 5, and 10 percent levels denoted by ***, **, *. See Table 1.6 for control variable estimates. The gap in reading achievement between the wealthiest 50 percent and poorest 50 percent of students decreased by 25 percent over the period. Table 1.3 presents the results of two regression models analogous to Table 1.2, except for the wealthiest and poorest 50 percent instead of urban and rural. In 2006 the poorest 50 percent of students lagged behindin reading by 40.61 points, which is about the equivalent of a year of schooling. In 2009, however, this lag was 28.65 points in 2009, or about three quarters of what it was in 2006. The decline in this difference was 11.95 points which was statistically significant. PISA reading achievement among the wealthiest 50 percent declined by 8.92 points while among the poorest 50 percent, the increase was 9.60 points. The wealthiest and poorest 50 percent are relative to the students sampled in each year and were identified using PISA’s wealth index which is based on home possessions. 11 Learning Outcomes in Thailand What Can We Learn from International Assessments? Table 1.3: Wealth Differences in PISA Reading Regression Model: (1) (2) Change in reading score from 2006 to 2009 among the wealthiest 50% -1.92 -8.05 (5.09) (4.30) Difference between wealthiest and poorest 50% in 2006 -40.61*** -17.81*** (4.11) (3.56) Change in the wealth difference 11.95** 11.16** (5.69) (4.97) Controlling for student background characteristics No Yes Constant 435.72*** 349.15*** (3.54) (16.65) Observation 12,409 12,061 R-Square 0.04 0.26 Change in reading score from 2006 to 2009 among the poorest 50% 10.03** 3.11 (4.47) (4.01) Difference between wealthiest and poorest 50% in 2009 -28.65*** -6.65** (3.55) (3.06) Source: author’s calculations using Thailand PISA 2006 and 2009. Standard errors noted in brackets. Statistical significance at the 1, 5, and 10 percent levels denoted by ***, **, *. See Table 1.6 for control variable estimates. The gap between men and women in reading achievement has notably decreased. In 2006, women outperformed men in all subjects. Table 1.4, similar to the previous two tables, presents the differences between men and women and how this difference changed overtime. In 2006, men lagged behind women in reading by 54.30 points, which is about 1.25 times one year of schooling in Thailand (see Table 1.4). In 2009, this dropped considerably to 37.64 points. Men showed an increase in reading score by about 14.45 points; however, when controlling for their background characteristics, this difference declines suggesting that much of the gain by men is due to changes in the population of those enrolled. In fact, when controlling for these background characteristics, women witness a decline in reading achievement. For science, there is a similar pattern but it is not statistically significant since the gap is smaller; and for math, the gap is still smaller so there is no clear pattern. 12 Learning Outcomes in Thailand What Can We Learn from International Assessments? Table 1.4: Gender Differences in PISA Reading Regression Model: (1) (2) Change in reading score from 2006 to 2009 among males 14.45** 2.79 (5.50) (4.63) Gender difference (females - males) in 2006 54.30*** 45.48*** (4.66) (3.93) Change in the gender difference -16.66*** -10.11** (6.02) (4.90) Controlling for student background characteristics No Yes Constant 385.57*** 343.74*** (3.95) (19.76) Observation 12,417 12,061 R-Square 0.09 0.26 Change in reading score from 2006 to 2009 among females -2.21 -7.31* (4.55) (3.78) Gender difference (females - males) in 2009 37.64*** 35.37*** (3.75) (2.97) Source: author’s calculations using Thailand PISA 2006 and 2009. Standard errors noted in brackets. Statistical significance at the 1, 5, and 10 percent levels denoted by ***, **, *. See Table 1.6 for control variable estimates. References Todd, P. E. and K. I. Wolpin. 2003. “On the specification and estimation of the production function for cognitive achievement.� Economic Journal 113:F3-F33. World Bank. 2008. Thailand Social Monitor on Youth 2008: Development and the Next Generation. World Bank, Washington, DC. 13 Learning Outcomes in Thailand What Can We Learn from International Assessments? Appendix: Production function modeling to account for differences in student backgrounds Cognitive production functions have been used extensively in academic literature to model and estimate the determinants of learning. They treat the production of cognitive achievement analogously to that of a firm: various inputs are combined according to some type of production technology to create cognitive ability. The inputs of a cognitive production function are everything that can affect a child’s learning from prenatal until present including the pedagogical materials at home, support from parents, teachers, pedagogy at school, motivation, innate ability, etc. The production technology is how these separate inputs combine to create learning. Modeling learning as the product of separate inputs conceptualizes the quality of an education system as the effectiveness of the school inputs as wells as the education system’s ability to combine inputs into learning (analogous to the production technology). Estimating a cognitive production function would then provide a measure of the effectiveness of an education system and allow the comparison of education quality overtime or between different systems. The problem with estimating cognitive production functions in practice is the lack of required data. Todd and Wolpin (2003) examine the challenges of estimating cognitive production functions and the severe assumptions needed for their estimation using data like PISA where information is only available about students’ current household and school characteristics. One assumption, for example, is that either only contemporaneous inputs matter to cognitive achievement or that inputs do not change overtime. For PISA data, it is unlikely that contemporaneous school characteristics have a large effect on achievement because students are tested quite late, at age 15, after 8 or 9 years of schooling excluding preschool. Most sampled schools in Thailand offer 6 grades of schooling up to grade 12; consequently, the Thailand PISA sample excludes school characteristic data for 6 out of the 8 or 9 previous years of schooling. Given these data limitations, this paper models household and personal characteristics of the student as the inputs into the cognitive production function and the ability of the education system to translate these characteristics into learning as the production technology. The personal and household characteristics include characteristics that are stable overtime such as gender of the student, education of the mother, whether the student is from the poorest 50 percent of households. The number of books at home is also included to proxy for a household’s interest in reading. Statistically, cognitive achievement is modeled as a stochastic process whose conditional mean is a linear function of these variables. 14 Learning Outcomes in Thailand What Can We Learn from International Assessments? Estimating this model provides comparisons on how effective different education systems are at translating student characteristics into learning. Comparing these estimates between 2006 and 2009, for example, can distinguish how much of the change in PISA reading stems from improvements in the household and personal characteristics of the student population and how much stems from improvement in the quality of the education defined as the education system’s ability to translate these personal and household characteristics into reading ability. Table 1.5 presents the estimates of a Oaxaca-Blinder decomposition underlying Figure 1.4. A cognitive production function is estimated for each year including the household and personal variables listed as well as fixed effects for schools. School fixed effects are needed to isolate the correlation between each variable and learning; in absence of school fixed effects, these variables would be correlated with both learning and the ability of households to select into better schools. As shown in the table, improvements in the student’s household characteristics increased reading ability in Thailand by 2.83 points and the ability of the system to translate these characteristics into learning increased it by 1.82 points. Table 1.6 presents estimates of cognitive production functions used to examine the overtime changes in reading achievement between different subpopulations controlling for household characteristics; these are presented in a truncated form in Tables 1.2, 1.3 and 1.4. These cognitive production functions are specified similarly to that used in Table 1.5; however, they are designed to estimate the difference in the education system’s effectiveness of adding value to the household and personal inputs between subpopulations and overtime. Binary variables for time and the subpopulation of interest as well as their interaction are included in the model to accomplish. The interpretation of the main findings is addressed above; however, as can be seen in Table 1.6, the household and personal characteristics of the students are large in magnitude and statistically significant. 15 Learning Outcomes in Thailand What Can We Learn from International Assessments? Table 1.5: Oaxaca-Blinder decomposition 2006 2009 Decomposition Variable Average Coefficient Average Coefficient Explained Unexplained mother completed 0.08 -3.34 0.11 -4.15 -0.12 -0.07 non-academic secondary mother completed lower 0.13 -7.29 0.12 -8.11 0.09 -0.10 secondary mother completed primary 0.53 -3.21 0.49 -8.01 0.28 -2.54 mother did not complete 0.10 3.67 0.08 -2.09 0.05 -0.57 primary 11 - 25 books at home 0.34 6.79 0.33 3.82 -0.05 -1.00 26 - 100 books at home 0.28 6.97 0.31 10.44 0.24 0.98 101 - 200 books at home 0.07 11.70 0.09 13.25 0.24 0.12 201 - 500 books at home 0.03 7.78 0.04 23.77 0.22 0.51 more than 500 books at home 0.01 18.89 0.02 11.53 0.10 -0.10 poorest 50 percent 0.47 -2.06 0.43 4.81 -0.20 3.23 Female 0.58 38.73 0.57 29.79 -0.19 -5.17 in grade 9 0.30 27.42 0.23 13.99 -1.01 -4.07 in grade 10 0.68 60.85 0.76 39.37 3.16 -14.69 constant 1.00 343.84 1.00 369.12 0.00 25.28 Total 2.83 1.82 Source: author’s calculations using PISA Thailand 2006 and 2009. Coefficients are estimated using a linear regression model with fixed effects for each school. The dependent variable are the reading “plausible values� which are five estimates of each students reading ability. Base categories are mother completed secondary education and 0 to 10 books at home. 16 Learning Outcomes in Thailand What Can We Learn from International Assessments? Table 1.6: Linear Regression Estimates Model 1 Model 2 Model 3 Variable Coefficient Std Err. Coefficient Std Err. Coefficient Std Err. rural area x year 2009 8.51 7.66 poorest 50 percent x year 2009 11.16** 4.97 female x year 2009 -10.11** 4.90 year 2009 -6.6 4.57 -8.05* 4.30 2.79 4.63 mother completed non-academic -3.3 3.12 -3.45 3.14 -3.49 3.14 secondary mother completed lower -23.79*** 3.01 -24.03*** 3.00 -23.6*** 3.01 secondarymother completed -22.76*** 2.48 -22.82*** 2.48 -22.66*** 2.49 primary mother did not complete -19.54*** 3.97 -19.62*** 3.99 -19.53*** 4.00 primary 11 - 25 books at home 10.67*** 2.21 10.61*** 2.21 10.64*** 2.20 26 - 100 books at home 18.75*** 2.74 18.63*** 2.73 18.72*** 2.74 101 - 200 books at home 30.45*** 3.35 30.56*** 3.35 30.5*** 3.34 201 - 500 books at home 42.56*** 5.09 42.53*** 5.12 42.52*** 5.08 more than 500 books at home 41.22*** 7.36 41.47*** 7.34 41.28*** 7.34 in a rural area -20.32*** 5.77 -16.39*** 3.6 -16.26*** 3.71 poorest 50 percent -12.44*** 2.20 -17.81*** 3.56 -12.27*** 2.21 female 40.3*** 2.49 40.23*** 2.46 45.48*** 3.93 in grade 9 44.51** 18.84 45.24*** 18.71 44.75** 18.82 in grade 10 76.27*** 19.46 76.83*** 19.32 76.32*** 19.45 small school -0.27 9.99 -0.34 10.15 0.38 10.27 constant 348.89*** 19.99 349.15*** 19.65 343.74*** 19.76 Observations 12,061 12,061 12,061 R-Square 0.25 0.26 0.26 Source: author’s calculations using PISA Thailand 2006 and 2009. Coefficients are estimated using a linear regression model with fixed effects for each school. The dependent variable is the reading “plausible values� which are five estimates of each students reading ability. Base categories are mother completed secondary education and 0 to 10 books at home. 17 Learning Outcomes in Thailand What Can We Learn from International Assessments? 18 Learning Outcomes in Thailand What Can We Learn from International Assessments? Chapter 2: What International Student Assessments Tell Us about Improving Basic Education in Thailand Introduction Because of the demographic changes that are happening in Thailand, the development of human capital is going to play a critical role in Thailand’s economic future. While lower-income countries in South East Asia are experiencing a bulge in the youth population, Thailand’s youth labor force is expected to decline by 10 percent over the next 10 years. As a result, the labor-intensive comparative advantage that contributed significantly to Thailand’s past economic performance will diminish (World Bank, 2006 and 2008). This means that it is essential for Thailand to develop the human capital of its shrinking young work force to ensure the country’s future competitiveness and economic growth. Education is a significant component of human capital development, the economic benefits of which have been thoroughly established in the literature (Psacharopoulos and Patrinos, 2004) as well as its positive effects on economic growth (Barro, 1997 and Romer, 1990). However, the success or failure of education in terms of increasing productivity and growth depends crucially on its quality (Hanushek and Woessmann, 2007). Thailand’s education system has been able to achieve a secondary enrollment rate as high as 72 percent according to the latest data from 2009 (World Bank EdStats), but its new challenge is improving educational quality. Analyzing scores on international student assessment tests such as the Trends in Math and Science Study (TIMSS) and the OECD’s Program for International Student Assessment (PISA) is one way to assess the current quality of secondary education in Thailand, in relation to other countries, over time, and internally. TIMSS measures achievement according to content generally taught across countries while PISA is focused on the ability of young people to apply their knowledge and skills in real-life situations. Our analysis of this data has revealed the challenges that Thailand faces in trying to increase its human capital. Two key challenges are, first, an over-time decline in the number of individuals reaching acceptable levels of educational achievement and, second, a large disparity in the effectiveness of the education system in urban versus rural areas. We found that, in those countries whose test scores had increased to acceptable levels of achievement, improving educational quality had been a crucial factor, especially in rural areas. In addition, we found that the distribution of learning achievement in one part of Thailand – Bangkok – is equivalent to that in high-income countries such as the United States. These international comparisons suggest that disadvantaged sub-populations such as those in rural areas should not be ignored by policymakers wishing to improve educational quality in Thailand and that they may learn some useful lessons by examining the education system in Bangkok. 19 Learning Outcomes in Thailand What Can We Learn from International Assessments? Identifying the Challenges Figures 2.1 and 2.2 capture the educational challenges facing Thailand. In the last three rounds of PISA, Thailand’s reading achievement score has declined, while countries with similar per capita income levels, including Indonesia, have increased their scores. Similarly, since 1999, Thailand’s TiMSS math achievement score has also declined. The achievement score alone is not a very intuitive measure; it is an abstract measure of ability estimated using item response theory with an arbitrary mean and standard deviation. However, it can be used to estimate what proportions of students are able to attain various proficiency standards. TIMSS, for example, has four proficiency benchmarks for mathematics. Students who attain the second proficiency standard, the intermediate math benchmark, are able to “apply basic mathematical knowledge in straightforward situations� (Mullis et al, 2008, p. 69) which is a seemingly prerequisite skill for participating in the knowledge economy of today. 20 Learning Outcomes in Thailand What Can We Learn from International Assessments? Figure 2.3 presents the proportion of grade 8 students in East Asia who were able to attain the intermediate math benchmark in the 2007 TIMSS. Viewing TIMSS performance from this perspective (as opposed to the overall scores used in Figure 2.2) changes the order of the country rankings, with Korea, Singapore, and Japan filling the top three spots. In Thailand, 34 percent of grade 8 students were able to achieve the intermediate benchmark in 2007, but this is 11 percentage points lower than the 45 percent who achieved it in 1999. As Figure 2.4 shows, the number of students in grade 8 in Thailand increased by only 10 percent during this period, which means that the total number of students who could achieve the intermediate math benchmark actually dropped by 17 percent. This is an important finding because the decrease in the number of young individuals meeting this minimum standard suggests a decrease in the growth rate of human capital in Thailand. Since enrollment increased during this period, this decline must be solely attributable to decreases in quality. The PISA data also contain a measure of wealth estimated from student responses to questions about their possessions. Figure 2.5 presents the changes in reading achievement score over time of students from above and below the median wealth point. While virtually no change was detected among students from the top half of the wealth distribution, there was a substantial change among those in the poorest half of the distribution. In other words, the decline in reading achievement happened only in the case of the least advantaged. 21 Learning Outcomes in Thailand What Can We Learn from International Assessments? One way to find out more about this decline is to decompose how much of it is due to changes in the characteristics of the student population and how much is due to changes in how effectively the education system is transforming its students’ initial characteristics into learning. So we followed the cognitive production function literature (see Todd and Wolpin 2003) and conceptualized learning as a production process. Using this concep- tualization, the background characteristics of the student population can be thought of as inputs into the production process and the education system as the production process that transforms these inputs into learning. There are several decomposition methods used in labor economics that can be used to separate the proportion of the over-time difference due to changes in the inputs from the proportion due to changes in how effectively the education system is transforming these inputs into learning. Figure 2.6 presents the results from an Oaxaca-Blinder (Oaxaca, 1973 and Blinder, 1973) decomposition, in which we decomposed the 16-point drop in PiSA reading scores into one portion explained by the observable background characteristics of the student and another unexplained portion attributable to changes in the education system’s effectiveness. (See Annex 1 for details of the methodology used.) The student background characteristics included gender, grade level, socio-economic status as captured by the number of books at home, household wealth, and the education level of the mother as well as the language or dialect spoken in the household. As Figure 2.6 reveals, the change due to student characteristics is positive; in other words, if the education system had been just as effective at transforming students’ background characteristics into learning in 2006 as it was in 2000, then the score in Thailand would haveincreased. This suggests that the decline in the quality of education in Thailand is not due to changes in the characteristics of the student population but to the education system becoming less effective at educating students of similar backgrounds. This is an important observation because it suggests that education policymakers may be able to influence the problem. Had the decline been entirely due to the background characteristics of the students such as socio-economic status or gender, then education policymakers would have had much less scope to affect the problem. 22 Learning Outcomes in Thailand What Can We Learn from International Assessments? In addition to the decline in educational quality measures in Thailand, the second major challenge is the disparity in education outcomes within the country. Figure 2.7 presents the large disparity between TIMSS math scores by students in Bangkok and the scores of those from other areas of Thailand. A similar decomposition can be used to understand this difference, and this is presented in Figure 2.8. Of the 44-point difference in PISA reading scores, only 14 points are due to the background characteristics of students, such as the lower average socio- economicstatus levels in rural areas. The bulk of the difference, 31 points, can be attributed to differences in how effectively the education system is transforming student characteristics into learning. Like the result of the over-time decomposition of Figure 2.6, this observation is important because it reveals that the urban-rural difference in Thailand is mostly due to differences in the education system and not to differences in demographics. Finding the Solutions So what should education policymakers in Thailand do to improve the quality and performance of the education system over time and to eliminate the urban-rural disparity? Analysis of international assessment data alone cannot answer this question, but it can reveal some useful facts to inform policy dialogue. A major problem that is illustrated in Figure 2.8 is the over-time decline in the number of students who were able to achieve the TiMSS intermediate math benchmark. We analyzed the international assessment data to identify which countries had the largest increases in the number of students achieving the intermediate benchmark and to understand why they had improved. The percentage increases achieved by the top TIMSS countries, excludinghigh-income economies, are presented in Figure 2.9. Morocco, Syria, and Ghana all achieved tremendous increases of around 200 percent, or double, while more modest increases between 30 and 40 percent were achieved by Lebanon, Jordan, and Tunisia among others. However, when we used a decom- position method similar to that of Oaxaca-Blinder, we found that much of this increase was due to improvements in the quality of education and not to increases in enrollment (see Figure 2.10). (See Annex 1 for details of the methodology used.) For example, in all countries shown except Morocco, Palestine, and Botswana, the increase in quality accounted for most of the increase in the number 23 Learning Outcomes in Thailand What Can We Learn from International Assessments? of students able to achieve the intermediate math benchmark. in Tunisia and Armenia, the number of grade 8 students actually declined, but the increase in quality offset that decline, resulting in a net increase in the number achieving the benchmark. Therefore, if policymakers in Thailand wish to increase the number of students who can achieve the math intermediate benchmark and thereby foster the growth of human capital in Thailand, they should focus more on improving the quality of education than on increasing enrollment. 24 Learning Outcomes in Thailand What Can We Learn from International Assessments? Another characteristic of the most improved countries is presented in Figure 2.11. We decomposed the increases in the number of students who achieved the intermediate math benchmark into urban and rural areas.2 in almost all countries, the contribution to the increases by rural areas is not minor and, although this cannot be shown, it is likely that the contribution made by other disadvantaged groups, such as the poor, is not minor either. This suggests that whatever policies or changes caused these increases in these countries were not experienced only by advantaged groups but also by disadvantaged groups. As a result, policy dialogue would not ignore disadvantaged groups when considering ways to improve education quality in Thailand. Another fact revealed by our analysis of international student assessment data is shown in Figure 2.12, which was adapted from Ahuja et al (2006). We found that the distribution of TIMSS math achievement in Bangkok is almost identical to that of the United States. While this highlights the disparity in educational quality between Bangkok and the other areas of Thailand, it also suggests that many of the problems of educational quality shown in rural areas in Thailand have been solved in Bangkok. Therefore, policymakers may wish to examine Bangkok’s education system to see whether any lessons can be learned that can be applied to other areas, including rural areas. 25 Learning Outcomes in Thailand What Can We Learn from International Assessments? Conclusions The chief challenge identified by this analysis is the decline over time in the number of grade 8 students able to achieve the TIMSS intermediate math benchmark. The implication of this finding is that the human capital growth rate is slowing down, which is of particular concern to Thailand given its need for human capital in order to stay competitive in the future. However, our decomposition analysis revealed that the decline in quality over time is not due to demographic factors such as the education levels of the students’ parents or their general socioeconomic status but rather of the ability of the education system to transform these factors into learning. Therefore, education policymakers may be able to reverse this trend and also address the second challenge revealed by this analysis - the urban-rural disparity in test scores. international student assessment data provides a means for policy makers in Thailand to compare and benchmark the characteristics of their education system to those of other countries. As shown, in almost all emerging economies that experienced increases in the number of students able to attain the TIMSS intermediate math benchmark, the driver was an increase in quality. As a result, policies focusing on increasing the effectiveness of the education system to produce learning among its students would emphasize quality. Another characteristic of most of these countries is that rural areas contributed significantly to this increase. As a result, policies aimed at improving learning in Thailand would not only focus on urban areas but also rural areas. In fact, as shown, the distribution of learning in Bangkok is as good as high income economies like the United States; thus improving the distribution of learning among rural areas is required for Thailand to have education quality levels the same as high income economies. And given the effectiveness of the education system in Bangkok, policy makers may not have to look too far to find solutions to improving education quality in Thailand as a whole. 26 Learning Outcomes in Thailand What Can We Learn from International Assessments? References Ahuja, A., T. Chucherd and K. Pootrakool. 2006. “Human Capital Policy: Building a Competitive Workforce for 21st Century Thailand.� Monetary Policy Group, Bank of Thailand, Bangkok. Barro R.J. 2001. “Human Capital and Growth.� American Economic Review, Papers and Proceedings 91(2): 12-17. Barro, R.J. 1997. Determinants of Economic Growth: A Cross-section Empirical Study. MIT Press. Blinder, A. 1973. “Wage discrimination: Reduced form and structural estimates.� Journal of Human Resources 8(4): 436–455. Hanushek, E. and L. Woessmann. 2007. Education Quality and Economics Growth, World Bank, Washington, DC, 2007. Mullis, I. V. S., M. O. Martin, and P. Foy. 2008. TIMSS 2007 International Mathematics Report. Chestnut Hill, MA: TIMSS and PIRLS International Study Center, Boston College. Oaxaca, R. 1973. “Male-female wages differentials in urban labor markets.� International Economic Review 14(3): 693–709. OECD. 2009. PISA Data Analysis Manual: SPSS and SAS, Second Edition. Paris, France: Organization for Economic Cooperation and Development Psacharopoulos, G. and H.A. Patrinos. 2004. “Returns to Investment in Education: A Further Update.� Education Economics 12(2): 111-134. Todd, P and K. Wolpin. 2003. “On the Specification and Estimation of the Production Function for Cognitive Achievement.� Economic Journal: F3-F33. World Bank. 2008. Thailand Social Monitor on Youth 2008: Development and the Next Generation, World Bank, Washington, DC. World Bank. 2006. Thailand Social Monitor: Improving Secondary Education, World Bank, Washington, DC. UNdata 2009. Total Secondary Net enrolment rate. http://data.un.org/Data.aspx?d=UNESCO&f=series%3ANER_23 Accessed 12/27/2009. 27 Learning Outcomes in Thailand What Can We Learn from International Assessments? Annex 1 This annex describes the methodology behind the Oaxaca-Blinder method that we used to decompose the over-time and urban-rural differences in the PiSA results. We also used it to attribute increases in the number of students who achieved the benchmark to quality factors versus enrollment rates and to urban versus rural areas. The Oaxaca-Blinder method is well documented, and we refer the reader to Oaxaca (1973) for the formulas. The coefficients that we used in the regression models for 2000 and 2006 as well as for urban and rural areas are presented in Table 2.1 and 2.2. We estimated these coefficients using five plausible values for reading proficiency with the Monte-Carlo method outlined in OECD (2009). The model is a school fixed-effects model; in other words, the coefficients capture the correlation between the independent variables and achievement within schools in order to eliminate some of the selection bias. We estimated standard errors using the Balanced Repeated Replication method with a Fays adjustment of 0.5. We used a similar decomposition to attribute the increase in the number of grade 8 students who achieved the intermediate benchmark to educational quality versus enrollment. The following equation defines the number of students who achieved the intermediate benchmark at time t: number attainingt = proportion attainingt × number in grade 8t Using a similar type of decomposition method as Oaxaca-Blinder, we then decomposed the number of students achieving this benchmark over time into the part due to the change in the proportion of grade 8 students attaining the benchmark (or quality) and the part due to any change in the absolute number of grade 8 students. number attaining2006 – number attaining2000 = number attaining2000 × (proportion attaining2006 – proportion attaining2000) + proportion attaining2006 × (number attaining2006 – number attaining2000) We were then able to count directly the number of students who achieved the benchmark in urban and rural areas respectively. 28 Learning Outcomes in Thailand What Can We Learn from International Assessments? Table 2.1: Over-time Oaxaca-Blinder Decomposition of PISA Reading Scores Table 2.2: Urban Rural PISA reading Oaxaca-Blinder decomposition 29 Learning Outcomes in Thailand What Can We Learn from International Assessments? 30 Learning Outcomes in Thailand What Can We Learn from International Assessments? Chapter 3: Educational Quality in Thailand as Measured by International Tests (PISA and TIMSS) Introduction The success of individuals and societies is dependent upon schools providing their students – and the country’s future workers – with a solid foundation of knowledge and skills. Failure to provide citizens with adequate skills to participate in today’s society will prevent children from realizing their potential and cost society more in terms of health care, income support, child welfare, and security (OECD, 2007a). The quality of Thailand’s education outcomes was recently measured by two international studies that allow for comparison across countries. PISA (the OECD’s Program for International Student Assessment) tests the ability of 15-year-old students to apply their knowledge and skills in real-life situations. TIMSS (Trends in Mathematics and Science Studies) focuses more on the content of the curricula taught to students in their schools. PISA assesses literacy in three key subject areas - reading, mathematics, and science - and collects information about students and their schools. TiMSS assesses students’ ability in mathematics and science and also collects information about students and their schools. The TiMSS results referred to in this document are from the TiMSS assessment for 8th- grade students. Our analysis of the PISA and TIMSS data shows that Thailand’s students are still a long way from being able to produce strong scores in key subject areas, especially in comparison with other countries in the East Asia region. This poses a challenge for Thai education policymakers. The analysis below verifies the assessment data going beyond just mean score and ranking to look at those variables which have an impact on learning. We propose some possible factors that negatively affect learning outcomes in Thailand, and outline the benefits of certain measures that could improve education quality. Specifically the analysis of PISA, TIMMS and National Assessment data focuses on: (1) Thailand’s performance in comparison with its regional competitors (2) The distribution of education quality among geographical areas (among regions and be tween urban and non-urban areas), between genders, and among socioeconomic groups (3) Changes in educational quality over time (4) The variables that affect learning outcomes. We aim to inform and guide policymakers on how best to improve student performance and educational quality in Thailand. 31 Learning Outcomes in Thailand What Can We Learn from International Assessments? The Test Results In three key subject areas, namely reading, mathematics, and science, a wide distribution of student performance can be seen among countries in Asia. The results of PISA 2006 show that, while the performance of students from Japan, Korea, Hong Kong (SAR China), and Chinese Taipei were on the top end of the international scale, that of students from Thailand and indonesia were on the lower end (OECD, 2007b). Similar results can be seen in the TIMSS 2007 study (which tested students in the 8th grade). Students from Singapore, Chinese Taipei, Japan, and Korea were in the top five and those from Hong Kong (SAR) were in the top ten, but students from Thailand, Malaysia and Indonesia were below average in all subject areas (Martinet al, 2008 and Mullis et al, 2008). The data from PISA 2006 and TIMSS 2007 are compiled in Figure 3.1 (and Annex Table 3.1). More useful for our analysis than the mean score, PISA also reports results in terms of five levels of reading proficiency, with level 5 being the highest and level 1 being the lowest. Nevertheless, there are a number of students who performed at a level even lower than level 1. On the PiSA scale, Level 2 is designated as the threshold or basic level at which students begin to show that they can benefit from readingin their later lives. The PISA mathematics and science results are also reported in terms of proficiency level, from level 6 to level 1, with level 2 again being the threshold. Those whose proficiency is below level 2 are regarded as being a “risk group.� Results from PISA 2006 indicate that 44.6 percent of 15-year-old Thai students performed below the basic reading level, and fewer than 5 percent reached the high proficiency levels (level 4 and level 5). In contrast, only 5.7 percent of Korean students were below level 2 and around 50 percent of Korean students scored at the high levels, among who one-fifth reached the highest level (level 5). In Hong Kong-China, 45 percent of students performed at the highest proficiency level. While a majority of students from other Asian countries performed above the basic level, the majority of students from Thailand and Indonesia were only at or below #basic levels of proficiency (Annex Table 3.2). 32 Learning Outcomes in Thailand What Can We Learn from International Assessments? Figure 3.2 summarizes the proportion of students at each reading proficiency level, with the bar on the right side representing the percentage of students above the basic level and the bar on the left side representing the percentage of students at or below the basic level. The graphs for Korea, Japan, Hong Kong-China, and Chinese Taipei stretch to the right side, but those for Thailand and Indonesia tend towards the left side. A similar picture can be seen in mathematics and science where the longer bars are on the left side for Thailand and Indonesia, while other countries have longer bars on the right side (Klainin et al, 2008). Trends from PISA 2000 to PISA 2006 On the PISA 2000 test (PISA plus), Thai students performed below the OECD average and ranked near the low end on the international scale. Thailand’s performance in all three subject areas has deteriorated significantly in the subsequent two rounds of the test (Figure 3.3). Between PISA 2000 and PISA 2006, the reading performance of Thai students declined by 14 points. Their performance declined slightly at the higher end but declined markedly at the lower end. The percentage of students below the level of basic reading proficiency (level 2) increased from 37 percent in PISA 2000 to 45 percent in PISA 2006. The scores of Thai students in mathematics and science decreased by 15 points. The percentage of students below the basic level accounted for over one-half (53 percent) in mathematics and nearly one-half (46 percent) in science (Annex Table 3.2). 33 Learning Outcomes in Thailand What Can We Learn from International Assessments? Trends from the National Tests The performance of Thailand’s students has also declined in its national tests. For students in grade 9, the last year of compulsory education, a large decline occurred in scores for Thai language and English between 2000 and 2008 (Figure 3.4 and Annex Table 3.3). 34 Learning Outcomes in Thailand What Can We Learn from International Assessments? How the Other Countries in the Region are Faring in PiSA 2000, indonesian students performed at the low end on the international scale but have since improved their score. For example, their reading performance increased by 22 score points by PiSA 2006. The percentage of Indonesian students below the basic level decreased from 69 percent in PISA 2000 to 54 percent in PISA 2006. Indonesia’s mathematics score increased by 24 points in those six years, but their score in science remained unchanged. These trends are summarized in Figure 3.5 (Annex Table 3.4). 35 Learning Outcomes in Thailand What Can We Learn from International Assessments? Results from TIMSS The TIMSS results also demonstrate the decline in Thailand’s performance. From TIMSS 1995 to TIMSS 2007 (Thailand did not participate in TIMSS 2003), Thailand’s science score dropped by 39 points. It is interesting to note that Thailand and Hong Kong (SAR) had the same score in TIMSS 1995, but their scores have diverged over time. While Hong Kong-China has moved up, Thailand has moved down (Klainin, 2002 and 2009). Malaysia did not take part in PiSA but participated in TiMSS. Like Thailand, since TiMSS 1999 to TIMSS 2007, Malaysia’s scores dropped dramatically by 21 points in science and 45 points in mathematics. Indonesia’s scores remained almost the same in science but declined in mathematics (see Annex Table 3.4). Performance Distribution of Thai Students Disparities among different groups of students are evident in the results of PISA and TIMSS, particularly among students with different social backgrounds. For instance, the data shows that the mother’s education has an impact on student performance. Students whose mothers had completed only primary or lower secondary education (ISCED 1, 2) scored about one level lower in science, mathematics, and reading than those whose mothers had completed tertiary education (ISCED 5). This was true both for Thailand and for the OECD average (OECD, 2007b). 36 Learning Outcomes in Thailand What Can We Learn from International Assessments? Most Thai students (65 percent) have mothers who have completed only primary or lower secondary education, while a majority of Korean students have mothers with an upper secondary education. The number of Korean students whose mothers had completed tertiary education was nearly three times higher than the number of Thai students. On the other hand, Thailand had more than four times the number of mothers with a primary or lower secondary education (Figure 3.8 and Annex Table 3.5). This may have been one of the factors that influenced the different test scores achieved by the two countries. One study of how Thai students have performed on the PISA test (Klainin et al, 2008) has shown the performance disparity between students of different socioeconomic backgrounds. Taking PISA’s ESCS (Economic, Social and Cultural Status) mean index of two groups of Thai students - the top performing university secondary schools (SATIT) and the lowest performing schools (OBEC1) - the mean index of the former group was 0.61 and that of the latter was much lower at -2.26. The performance gap between the two groups was more than one standard deviation and equal to about two proficiency levels in science. The mean score of the first group was higher than the OECD average, while that of the latter group was at the lower end of the scale. 37 Learning Outcomes in Thailand What Can We Learn from International Assessments? Disparities in Performance between Geographic Areas Student scores varied widely between the different geographical regions in Thailand. In PISA 2006, students in Bangkok and its outskirts (Bangkok Metropolitan Area) performed better than all of their counterparts from other regions. Students from the North-East region had the lowest scores on all tests. The differences between the top performers and bottom performers were one proficiency level in reading (74 points) and close to one level in mathematics and science (68 and 64 points). Bangkok students outperformed those from provincial areas, while urban students outperformed non-urban students in all three subject areas (Klainin, et al, 2008). Between PISA 2000 and PISA 2006, the gap between the top and the bottom performers grew wider (Figure 3.9). While the top group improved its performance, the scores of the bottom group declined. In PISA 2000, the gap in reading was 26 points, less than half of a proficiency level, but this gap had reached one proficiency level (74 score points) by PISA 2006. The gaps in science and mathematics were close to one proficiency level (Annex Table 3.6). Urban and Non-urban Schools in PiSA 2000, students from urban schools performed better than their non-urban counterparts. The scores of both groups declined in PISA 2006, but the scores of students from non-urban schools declined by more (Figure 3.10). In PISA 2000, the gaps were a little less than half of a proficiency level, ranging from 29 to 30 points in the three subjects. In PISA 2006, the gaps ranged from 39 to 43 points or more than half a proficiency level in all subject areas (Annex Table 3.6). Big Schools and Small Schools In terms of school size, students from big schools – in terms of student population – outperformed those from small schools in all three subject areas. In PISA 2000, the gaps ranged from 43 to 52 points. Since then, the scores of students from both types of schools have declined, so the size of the gaps was more or less the same in PISA 2006 as in PISA 2000. In science, there is some evidence that the gap became narrower (from 52 points to 43 points). 38 Learning Outcomes in Thailand What Can We Learn from International Assessments? Public and Private Schools In PISA 2000, public schools outscored private schools by 13 points in reading and 14 points in science. The performance of both school types decreased over time, but the scores of public school students decreased by more. As a result, by PISA 2006, the gap had become much narrower, two points in reading and five points in science. 39 Learning Outcomes in Thailand What Can We Learn from International Assessments? In conclusion, in all measurements and criteria, the results indicate two points. Firstly, they indicate a deterioration in the quality of education over time in Thailand. Secondly, they show that disadvantaged students have become more disadvantaged over time. Ever since the year 2000, when the Education Reform Act was passed in Thailand, there has been no improvement in education quality and no narrowing of gender and geographical equity gaps in achievement, which was one of the goals of the Act. Performance Differences between Grils and Boys in Thailand Girls and Boys in Thailand The PISA 2006 results show that, in all countries, girls significantly outperformed boys in reading. In Thailand, this difference was equivalent to one level of reading proficiency. Among low- performers on the reading test, there were more boys than girls, but among the high performers, there were more girls than boys (Annex Table 3.7). These results indicate that high-achieving girls performed better than high-achieving boys, and low-achieving boys lagged behind low-achieving girls. Among those who scored more than 600 points, 86 percent were girls and 14 percent were boys. Of the top ten students in reading, nine were girls, but among the bottom ten, all were boys. Only in mathematics did more boys than girls score in the top ten, but among the bottom ten, there were more boys than girls. In science, girls also outperformedboys; there were more girls than boys among the top ten scorers, but the bottom ten were all boys (Annex Table 3.8). Also, girls performed better than boys in every geographical region of the country (Klainin et al, 2008). This same trend was evident in the results of TIMSS 2007 in which girls performed better than boys in both science and mathematics (Klainin et al, 2009). 40 Learning Outcomes in Thailand What Can We Learn from International Assessments? Gender Differences in Other Countries The analysis here focuses on gender differences in science because these differences have become an international concern. The results of TiMSS 2007 show that girls from Malaysia, Singapore, and Hong Kong-China performed better than their male counterparts (Malaysia and Singapore did not participate in PISA 2006). In Indonesia, Japan, and Chinese Taipei, boys outperformed girls in both PiSA 2006 and TiMSS 2007. it is interesting that, in Korea, girls performed better than boys in PiSA, but boys performed better in TiMSS. The reverse was true in Hong Kong where boys performed better in PISA and girls performed better in TIMSS. (Readers are reminded that the TIMSS test focuses on the content of the school curriculum, whereas PISA focuses on the application of knowledge in real-life situations.) As we have seen, in Thailand, girls did better than boys in both PISA and TIMSS (Figure 3.14 and Annex Table 3.9). 41 Learning Outcomes in Thailand What Can We Learn from International Assessments? Trends in the Gender Gap in Thailand between PISA 2000 and PISA 2006 In Thailand, although the performance of both boys and girls declined between PISA 2000 and PISA 2006, boys declined more than girls in reading, meaning that the gender gap in reading has become wider over time. In mathematics, the gender gap remained more or less the same, and in science, the gap grew a little wider but not as much as in reading (Figure 3.15 and Annex Table 3. 10). In terms of science performance, the change in gender differences varied from country to country. Korea had the widest gap in favor of boys in all Asian countries in PISA 2000, but this gap changed in favor of girls by PISA 2006. In Hong Kong-China, the gap in favor of boys became narrower over time, while in Indonesia, the gap in favor of boys widened. In Japan, girls had higher scores than boys in PISA 2000, but by PISA 2006, boys scored higher than girls. In Thailand, the gender gap in favor of girls continued but widened (Figure 3.16 and Annex Table 3.11). 42 Learning Outcomes in Thailand What Can We Learn from International Assessments? international studies, particularly in the developed countries, have generally found that girls are under-represented and are under-achievers in the physical sciences (Comber and Keeves 1973; Keeves and Kotte 1992 and 1996). In the western world, science is seen as a masculine enterprise (Harding 1986), which leads to the assumption that “science is for boys� and to the assumption that gender differences were inherent (Lin and Peterson 2004, cited in Fensham 2004). In Thailand, however, this generalization does not hold. Girls are not under-represented nor do they under-achieve. On the contrary, girls perform at least as well as boys, and in many cases they outperform their male counterparts in science. Long before the PISA study, research in Thailand showed that girls performed as well as boys in science, and in many cases better than boys in science and mathematics (Klainin and Fensham 1987; Klainin et al. 1989). These results have been criticized in the western world as false, biased, and not credible (Rennie 2002). Now the results of international studies show that this false image of girls as being bad at science has begun to change. PISA 2006 showed that girls and boys performed equally well in science in many countries, including Australia, the USA, and Ireland. Boys did better than girls in 10 countries, but in 12 countries, including Thailand, girls did better than boys by a much wider margin. In the rest of the 35 participating countries, the differences were not significant either way. Now the focus of education policymakers has begun to shift from the need to ensure that girls are learning science to the need to ensure that boys are learning science. For example, PISA suggested to policymakers to more closely examine science competencies among male students (OECD, 2007b). 43 Learning Outcomes in Thailand What Can We Learn from International Assessments? The Analysis of School Variables and Student Performance Policymakers and the public often look only at the mean score and their country’s ranking in the results of these international assessments. As a result, much useful information is ignored or neglected, and thus opportunities to improve the quality of learning have been missed. In our analysis in this section, we look more deeply into the results and investigate the role that various factors, particularly school-related, play in influencing learning quality. By revealing a clearer picture of what has been happening in the education system and locating any weak points, we hope provide useful information to inform future policymaking. Variations in student performance within any one country can have a variety of causes, including the socioeconomic background of students and of schools, the way in which teaching is organized, the availability or lack of resources, system-level factors, and organizational policies and practices. Our analysis builds on the OECD’s PISA database and PISA’s Thailand database as well as on data and results from PISA-related research on Thailand (Klainin, 2002, and Klainin et al, 2007; 2008; and 2009). Since the overall impact of student-related and school-related factors on student performance is similar for reading, mathematics, and science, the analysis focuses mainly on their impact on science performance. School Factors Six groups of school-related and system-related factors were collected from school principals in the PISA questionnaires: The Admittance , selection, and grouping of students School management and funding Parental pressure and choice Accountability policies School autonomy School resources In order to estimate how each school factor influences learning, we used a simple regression analysis (see Annex Table 3.12). This model makes it possible to estimate the association of learning with the variable in question while holding other variables fixed. This helps isolate the relationship between the specific variable and learning. Since various aspects of a school system can be interrelated, the OECD uses a multilevel model to analyze the data from PISA 2006 for each factors. This model can reveal how much of the variation in student performance is associated with school-level and system-level factors and how much is due to other factors, including each student’s socioeconomic background. The model uses student data from 55 PISA countries (around 14,000 schools) and gives each country an equal weight. In this report, we analyze the multilevel models that estimated the relationship between performance in 44 Learning Outcomes in Thailand What Can We Learn from International Assessments? science and six groups of school factors, both before and after accounting for socioeconomic variables at the student, school, and system levels. The relationship before accounting for socioeconomic variables is referred to as gross effects, and after accounting for socioeconomic status, it is referred to as net effects. Only the factors that have a significant relationship with performance were selected. The PISA results (OECD, 2003, 2004, and 2007b) show that in almost all countries, there is a clear advantage associated with attending schools whose students are from a more advantaged background. Regardless of their own socioeconomic background, students attending schools with a high average household income level tend to perform better than those who are enrolled in schools with a below- average household income level. Admittance, Grouping, and Selection Policies PISA asked school principals what factors they consider when admitting students. The criteria they came up with included the student’s residence in a particular area, the student’s academic record, a recommendation from feeder schools, pressure from the parents, a student’s need for a special program, or the attendance at the school of other family members. The most frequent factor cited by the principals was the students’ residence in a particular area. On average, 47 percent of 15-year-old students are enrolled in schools according to this geographical criterion in all of the 55 PISA countries. In Thailand, school principals reported that a higher than average percentage of students (71 percent) were admitted according to this criterion. However, in other high-performance countries in Asia, the percentage was lower than the OECD average. This criterion was used most by schools run by the Bangkok Metropolitan Authority (BMA), the Office of Basic Education (OBEC), and the Local Education Administration (LOC). Secondary schools run by universities (SATIT schools) did not take into account where students lived, with around 80 percent of students in these schools being selected according to other criteria. Others that did not consider residence in their admission criteria were private schools (PRV) and vocational institutions (VOC). The second most frequently reported criterion used to determine admission was the student’s academic record. In the OECD, an average of 27 percent of students were admitted according to this criterion, but the percentages for the high-performing countries in Asia were higher than the OECD average - Japan (86.3 percent), Hong Kong (82.7 percent), Macao-China (66.4 percent), Korea (59.1 percent), and Chinese Taipei (52.7 percent). However, there is no data on what happened to the low-performing students who were not admitted, or on how they performed on the international assessments. Klainin et al, 2009 found that Thailand used academic records as an admission factor less than other countries in the region, and this criterion was mostly used by the best performing schools, such as the university SATiT schools. 45 Learning Outcomes in Thailand What Can We Learn from International Assessments? The situation in Thailand is complex. Principals reported that they need to use many criteria in determining which students to admit. While putting a high priority on where students live, the principals also considered their academic records and parental pressure. Therefore, the figures in the PISA data for admission criteria for Thai schools may be confusing because of the use of complex criteria in practice. A student’s need for a specific program was the third most common admission criterion (the OECD average is 19 percent). In Thailand, normal schools did not use this criterion because they all offer the same curriculum. Only vocational colleges used this criterion in selecting students because they are offer a variety of different programs preparing students for specific occupations 1 Having a family member already at the school was the criterion that was least frequently mentioned by school principals. This was also the case for Thailand, where only private schools take this factor into account for admitting students. Institutional Differentiation and Grade Repetition Our analysis of the PISA data showed that the number of school types or distinct education programs in a country are not related to student performance. In countries with a large number of distinct types of program, stratification of students across program types tends to be associated with socioeconomic segregation. In Thailand, there are only two school types for 15-year-old students - basic education and vocational education. The proportion of all students in vocational education in Thailand is 11.3 percent, while the proportion of 15 year olds is 14.7 percent. A number of studies have found that grade repetition has little benefit for students. In Thailand, education policy does not favor grade repetition. Teachers are expected to help low-performing students to move up to the next grade with their classmates. Thus, the grade retention rate in Thailand is very low. The OECD’s analysis of the PISA 2006 data suggests that countries with a more stratified education system tend to perform less well, but the difference is not significant (OECD, 2007b). 46 Learning Outcomes in Thailand What Can We Learn from International Assessments? Ability Grouping Within Schools On average in OECD countries, 14 percent of 15 year olds attend schools that group students by ability in all subjects, 54 percent are in schools that have ability grouping forsome subjects, and 33 percent are in schools with no ability grouping. The PISA 2006 results indicate that, in six OECD countries and four OECD partner countries, science performance in schools that group students by ability in all subjects was lower than in the other two kinds of schools. After accounting for students’ backgrounds, those in schools that have no or only some ability grouping outperformed those from schools with ability grouping in all subjects in a majority of countries (OECD, 2007b). In high-performing countries like Finland, Korea, and Japan, principals repoted that only a minority of students taking the PISA test attended schools with ability grouping in all subjects (2.1 percent, 6.8 percent, and 9.8 percent respectively). In Hong Kong and Chinese Taipei, 17.3 percent and 8.2 percent of students were in schools with ability grouping in all subjects. By contrast #in Thailand, principals reported that about half of all PiSA 2006 students were in schools with ability grouping in all subjects, and 42 percent were in schools with ability grouping in some subjects. That is, over 90 percent of Thai students were in schools with some form of ability grouping. According to the principals’ responses to their PISA questionnaires, in Thailand, most ability grouping occurred in the lowest performing schools (OBEC 1), while there was much less in the highest performing schools, the university SATIT schools (Klainin etal, 2009). 47 Learning Outcomes in Thailand What Can We Learn from International Assessments? The Relationship between School Admittance, Selection, Ability Grouping and Student Performance We ran a regression analysis to find the association between student performance and selected school factors. Our results are summarized in Annex Table 1.12. One rather unexpected result was that students who were admitted to schools on the basis of their academic record scored lower in the PISA tests than students who were admitted according to other criteria. This is the opposite of what has been found in other studies using multilevel models, which showed that schools that emphasize academic achievement as a selection criterion tended to perform better. Those studies found that schools with high academic selectivity had a 30.4-point advantage over other schools on the PISA science scale. The net effect was reduced to 18.1 points after accounting for demographic and socioeconomic factors (Annex Table 3.13). However, those results do not provide a true picture of schools’ selection and grouping policies because some of the differences can be attributed to their selection processes. For example, admitting all students from a particular residential area is supposed to give every student an equal educational opportunity. However, the socioeconomic status of the residents of any given area may vary widely, thus affecting their ability to make the most of that opportunity. Schools in particularly advantaged areas are likely to have an intake of students who are from households with higher incomes. Students from disadvantaged socioeconomic backgrounds tend to be directed towards low-performing schools. Moreover, the PISA results for Thailand show no positive association between selecting students on the basis of their academic record and learning. Therefore, it is not clear that academic selection has an impact on student performance. In a similar analysis, the OECD found that students in schools with ability grouping in all subjects tended to perform worse than those in other schools. in Thailand, students in schools with ability grouping had a disadvantage of 10.1 points (Annex Table 3.13) and of 4.9 points after accounting for their contextual backgrounds #(Annex Table 3.13). The OECD average disadvantage on the science scale for these schools was about 8.8 points, and after taking into account the PISA index of economic social and cultural status, the OECD average disadvantage is 6.9 points. Unlike the OECD average, in the high-performing countries in Asia, students in schools with ability grouping in all subjects scored higher than students in schools with some or no ability grouping, but the difference was not significant (Annex Table 3.13). However, in these countries, only a small percentage of students attended schools that practice ability grouping. School Management and Funding: Public and Private Financing Encouraging private education is often seen as a way to increase resources in the education system and to make education more cost-effective. On average in OECD countries, only 4 percent of 15 year olds are enrolled in schools that are privately managed and predominantly privately financed (independent private schools). In Asian countries, private education is more common. In Japan and 48 Learning Outcomes in Thailand What Can We Learn from International Assessments? Chinese Taipei, about one-third of students are enrolled in independent private schools, but in Korea, Macao-China, indonesia, and Thailand, smaller proportions of students are enrolled in this type of school. Globally, private schools that are financed through public money (defined as government-dependent private schools) are more common than schools that are fully funded by the private sector. In the OECD, an average of 11 percent of 15 year olds are enrolled in these schools, but the proportion is higher in East Asian economies. In Hong Kong - China and Macao-China, the proportion of students in this type of private school is 91 percent and 69 percent respectively. in Korea, it is 31.5 percent. In Thailand, only 6.1 percent of 15 year olds are enrolled in government-dependent schools. When the proportion of students in the two types of private schools is combined, the percentage of students in private schools in Asia is as follows: Macao-China (96 percent), Hong Kong-China (92.5 percent), Korea (46 percent), Indonesia (39.3 percent), Chinese Taipei (35 percent), Japan (30 percent), and Thailand (16.5 percent). Schools in Thailand, whether publicly or privately managed, are financed through the government’s budget. Even independent private schools receive nearly one-half of their funding from the government. The majority of Thai students are enrolled in public schools, with 10.5 percent enrolled in independent private schools and 6 percent enrolled in dependent private schools. The Thai schools that receive most of their funding from the government come under many different jurisdictions, including the Bangkok Metropolitan Administration (BMA), the Basic Education Office (OBEC), and the Local Education Administration (LOC). Some private schools (PRV) and university secondary schools (SATIT) have other sources of funding, such as fees, fund-raising activities, donors, or parent-teacher associations. Although SATIT schools are government schools, less than 40 percent of their funding comes from the government, and the rest comes from other sources (Klainin et al, 2009). 49 Learning Outcomes in Thailand What Can We Learn from International Assessments? Performance of Students from Public and Private Schools The PISA results suggest that, before adjusting for demographic and socioeconomic factors, privately managed schools in PISA countries are associated with better academic performance. Students from private schools (government-dependent and government-independent combined) performed better than those in public schools by 25 points in science in PISA 2006. However, 56 once the demographic and socioeconomic factors of students are taken into account, the advantage became 8 points. The picture changed further when we also took the socioeconomic background of the schools (the ocio-economic background of its total intake) into account. This gave public schools an advantage of 12 points (Annex Table 3.13). This suggests that private schools have an advantage due to the socioeconomic background of their intake, which may enable them to create an environment that enhances learning. Conversely in all PISA countries in the Asia region, public schools performed better than private schools, with significant differences in Hong Kong-China, Indonesia, and Chinese Taipei and no significant differences in Japan, Korea, and Thailand. Only Macao-China was an exception. There private schools had a significant advantage over public schools of 49 points. After accounting for the socioeconomic background of students, the advantage in science performance changed. Public schools had a significant advantage in Japan, Hong Kong-China, Indonesia, Chinese Taipei, and Thailand. This advantage increased further after accounting for the socioeconomic background of students and of schools, except in Hong Kong where the advantage became insignificant (Annex Table 3.13). In addition, the results of our multilevel models suggest that privately managed schools are associated with better performance, but this advantage disappeared once we accounted for socioeconomic factors. In Thailand, public schools outperformed private schools by 5 points in science in 2006. Among students with similar socio-economic status, this increased to 14 points and at schools with similar socio-economic status this increased further to 21 points. In other words, among students from similar schools and household backgrounds, those in public schools perform better than those in private schools. Parental Influence on Schools Parents may indirectly influence how schools operate when they are choosing a school for their child. According to the international test data, an average of 60 percent of students’ parents in OECD countries had a choice between two or more schools for their children. in indonesia, Japan, Hong Kong-China, and Chinese Taipei, more than 80 percent of students were enrolled in schools where the school principals reported that parents had at least two alternatives to their own school. in Korea, the equivalent proportion was 76 percent, and in Thailand it was 66 percent. Researchers at PISA used multilevel models to estimate the gross and the net association between school choice and student performance (OECD, 2007). Their results showed that students in education systems where 50 Learning Outcomes in Thailand What Can We Learn from International Assessments? parents have more school choice scored 18 points more than their peers in other education systems with less choice, but this advantage disappeared after demographic and socioeconomic context was taken into account (OECD, 2007b). In Thailand (Klainin et al, 2007b), the picture of school choice may differ from the situation in other countries. In fact, parents do not have much choice in their children’s education, and more parents want places for their children in good schools than there are places available. Therefore, low-performing schools tend to end up with students from disadvantaged backgrounds and/or poor academic records. This situation is particularly prevalent in vocational institutions. Over 90 percent of the principals of VOCs reported that good schools on the basic academic track attract most of good students, and those who are left over have no choice but to enroll in vocational colleges. This is also true for low-performing schools whose principals reported that other schools had attracted better students. These schools, particularly the OBEC1 and the BMA school groups, are not able to attract good students. Therefore, they end up with a concentration of low-performing students from disadvantaged backgrounds. Most of the OBEC1 schools are small and are located in rural areas, meaning that they are already at a disadvantage. They have been segregated first by residential area and then by economic, social, and cultural status. The same is true for the BMA schools. Even though they are located in the Bangkok area, they are not popular with parents and thus their intake consists mainly of students from disadvantaged backgrounds. This implies that the goal of achieving equality and equity in education in Thailand is still far beyondreach. The low-performing groups and schools have been overlooked both by education policymakers and by the public. There is an urgent need for policymakers to ensure that these low-performing students get the essential assistance needed to help them to improve their academic performance. Accountability PiSA 2006 collected data on the accountability mechanisms that were in place in schools, and the way in which schools used information to hold themselves accountable to parents and local communities. Researchers at PiSA asked school principals to indicate whether or not they used achievement data as a way of showing school accountability. The results of the survey of Thai school principals showed that 72 percent of 15-year-old students were enrolled in schools where achievement data were made available to the public. Also, 82 percent were enrolled in schools where the school tracked its assessment results over time (the OECD average being 65 percent), 72 percent in schools where data was used to evaluate the principal’s performance, 86 percent in schools where data was used to evaluate teachers’ performance, and 76 percent in schools where achievement data was used in decisions about the allocation of instructional resources. 51 Learning Outcomes in Thailand What Can We Learn from International Assessments? Giving Feedback on Student Performance to Parents On average in OECD countries, principals reported giving feedback to parents about their child’s performance in relation to the performance of other students at the school in the case of 54 percent of students. in Thailand, principals reported that the proportion was 83 percent for feedback in relation to other students, and 52 percent for feedback in relation to national or regional benchmarks. The Impact of Accountability Policies Other accountability practices include using achievement data to evaluate principals and teachers, to allocate resources, to track achievement data over time, and the use of standards-based examinations. The way in which accountability policies relate to performance is not always clear since they are interrelated with other school policies. An estimate of this relationship in the multilevel models found that only two policies had a significant impact - when achievement data is posted publicly and when standards-based examinations exist. The gross positive impact on performance of the former is 15 points and on the latter is 36 points. In the case of the net impact, only the former is significant (see Annex Table 3.16). School Management and School Autonomy School autonomy involves the devolution of decision-making responsibility #and accountability to the schools themselves, particularly to principals. Devolved responsibilities can include decisions about the hiring and firing of teachers, the courses offered and their content, the choice of textbooks, teacher assessment policy, and the formulation and allocation of budgets. We developed three indices of school autonomy from the principals’ reports covering school autonomy over staffing, educational content, and budgeting. The OECD examined the association between school autonomy and student performance in a multilevel model for PISA countries (Annex Table 3.17). Using this data we found that school autonomy over educational content and budgeting both had a significant association with student performance. Students in education systems that give more autonomy to schools to formulate and allocate their budgets tended to perform better than others (an increase of one unit of the index corresponded to an increase of 22.5 points in science test scores). Students in education systems that give schools more autonomy over educational content also tended to perform better (an increase of one unit of the index corresponded to an increase of 20.3 points). For school autonomy over staffing, the association was weaker. 52 Learning Outcomes in Thailand What Can We Learn from International Assessments? School Resources and their Impact on Learning Among the three major factors that have been found to have an association with student performance - teaching and instruction, organization and management, and school resources - the last factor seems to have the strongest association in Thailand. School resources here include both teachers and educational materials. Human Resources: Qualified Teachers In terms of the adequate supply of teachers; principals reported that 2 percent of Thai students were enrolled in schools where one or more science teacher positions were still vacant (the OECD average was 3 percent). The rest were enrolled in schools that had no vacancies. Even when principals reported having no vacancies in their schools, they still reported a lack of qualified teachers in key subjects that seriously hindered student learning. The index of teacher shortages in Thai schools was 0.65, which was much higher than the OECD average index (0.00), while the indexes for Japan and Korea were -0.51 and those for Chinese Taipei and Hong Kong-China were -0.31 and -0.20 respectively. On average in the OECD, teacher shortages have been proven to have a strong negative effect on student performance, with a one unit change in the teacher shortage index being associated with a 9 point decrease in test scores. in Thailand, this relationship is twice as large, with a one unit increase in the index being associated with a -18.2 point change in test scores. Material Resources Material resources include science laboratories and equipment, instruction materials such as textbooks, computers, internet connectivity, computer software for instruction, library materials, and audio visual resources. Among OECD countries, only a minority of students attended schools where principals reported that a shortage of these educational resources had hindered learning (OECD, 2007b). In Thailand, however, the principals reported that over one-half of students were attending such schools (Klainin et al, 2009).. 53 Learning Outcomes in Thailand What Can We Learn from International Assessments? Principals in the OECD expressed little concern about the shortage or inadequacy of computers and internet connectivity but expressed more concern about the supply of laboratory equipment. Thai principals were most concerned about the availability of laboratory equipment, audio-visual equipment, and library materials (at schools covering 59 percent, 58 percent, 54 percent of students respectively). We prepared an index of educational resources to summarize the principals’ responses regarding the adequacy or shortage of educational resources. We inverted the index so that positive values reflected below-average concern that the inadequacy of the resource hindered student learning. In Japan, Chinese Taipei, and Hong Kong-China, the principals did not experience any shortage of educational resources (a positive index value), while principals in Korea expressed more concern (index – 0.19). Principals in Thailand and Indonesia expressed a great deal of concern about resource shortages (index -0.67 and -1.63 respectively) 54 Learning Outcomes in Thailand What Can We Learn from International Assessments? On average, educational resources exhibited an almost linear relationship with student performance: the greater the shortage, the lower the performance. For Thailand, a one unit change in the index was associated with a score change of 18.5 points (Klainin et al, 2009). Thus, low student performance stems not only from the socioeconomic background of the school’s intake, but also from a shortage of educational resources. As mentioned earlier, the insufficiency of educational resources can also explain the low performance of students in the OBEC 1 schools. This is illustrated in Figures 3.21 and 3.22. Figure 3.21 compares the index of school resources among different school groups in Thailand. In this figure, the teacher shortage index is inverted from the original index so that a positive index indicates a slight shortage or no shortage and a negative index indicates a greater shortage. Figure 3.22 illustrates the performance of students in different school groups in science compared with their country’s mean performance in relation to the resources available at those schools. The data indicates that the fewest resources were available at the OBEC1 schools, which were the schools with the students with the worst performance, while there was no shortage of resources at the SATIT schools, which was the best performing group of schools. Most of the OBEC 1 schools are small schools located in rural areas, which means that they suffer from those two disadvantages in the first place. Add to those disadvantages a resource shortage, and they seem to have had little chance to improve their students’ academic performance. How Resources have changed between PISA 2000 and PISA 2006 The data showed that the level of educational resources available to the most disadvantaged groups has remained unchanged. The change or otherwise in the availability of resources to the top and the bottom performing schools is summarized in Figure 3.23. 55 Learning Outcomes in Thailand What Can We Learn from International Assessments? Between PiSA 2000 and PiSA 2006, advantaged schools received more educational resources, but disadvantaged schools received the same amount. in terms of school size, big schools also received more educational resources, but there was no increase for small schools. in fact, small schools experienced some decrease in resources, though not a dramatic one. Computers for Instruction On average, in OECD countries, the data from 2006 and 2007 show that the ratio of computers used for instructional purpose to students was 1:7. In Thailand, the ratio was 1:13. In general, school principals did not express much concern about the inadequacy of this resource (OECD, 2007b). Two studies have analyzed the use of iCT and its effect on student performance (PISA, 2003 and OECD, 2004), and neither study found that it had a positive impact. Students who reported the most frequent use of computers (education programs and software) did not score the highest marks in tests. In fact, almost the exact reverse was found. This was true for the OECD average for performance in reading, mathematics, and science (OECD, 2005). Similar results were also found in Thailand (Klainin et al, 2007c). 56 Learning Outcomes in Thailand What Can We Learn from International Assessments? The Relationship between School Resources and Student Performance Since the various aspects of school resources are interrelated, the total impact can be estimated only by examining all of the various factors jointly to estimate their collective impact on student and school performance. Our multilevel model analysis yielded the gross effect of school resources (before adjusting for socioeconomic factors) and the net effect of school resources (after adjusting for demographic and socioeconomic factors). The factors that were included in the model were: the index of teacher shortage, the student-teacher ratio, the index of school educational resources, the ratio of computer use for instructional purpose to the number of students, learning time in schools, time spent on homework, time spent on out-of-school lessons, and the existence of school activities to promote learning (Annex Table 3.18). The models suggest that four factors were correlated with greater learning before accounting for the socioeconomic context. Another five school factors and one system factor were correlated with greater learning both before and after accounting for the socioeconomic context. The school factors that were associated with performance before taking students’ demographic and socioeconomic background into account were: School principals’ reports regarding the level of funding from government; -3.2 score points with additional 10 percent of public funding (Annex Table 3.14). A lack of qualified teachers; - 3.5 score points for one additional unit index (Annex Table 3.17) The quality of educational materials; +3.9 score points for one additional unit of the index (Annex Table 3.18). The school factors that were associated with performance after taking students’ demographic and socio- economic background into account were: Other schools in the area that compete for students; +7 score points with additional 10 percent of competitive schools (Annex Table 3.15). 57 Learning Outcomes in Thailand What Can We Learn from International Assessments? Schools factors that were associated with performance both before and after (or gross and net effects) taking students’ demographic and socioeconomic background into account can be summarized as follows: The practice of ability grouping for all subjects within schools; gross -10.2, and net -4.5 score points (Annex Table 3.13). Having high academic selectivity; gross 30.4, and net 18.1 score points (Annex Table 3.13) Posting school achievement data publicly; gross 14.7, and net 6.6 score points (Annex Table 3.16). Higher average student learning time for regular lessons in schools (one additional hour per week); gross 14.3, and net 8.7 score points (Annex Table 3.18). Higher average student learning time in out-of-school lessons (one additional hour per week); gross -12.9, and net -9.0 score points (Annex Table 3.18). Higher average student learning time on self-study or homework (one additional hour per week); gross 3.8, and net 3.1 score points (Annex Table 3.18) Higher average index of school activities to promote students’ learning of science (effect of one standard deviation of the index); gross 7.1, and net 2.89 score points (Annex Table 3.18). The same factors at the system level that were associated with performance both before and after (or gross and net effects) taking students’ demographic and socioeconomic background into account can be summarized as follows 65 Education systems where schools had a high degree of autonomy over budgeting (for one additional standard deviation); gross 27.2, and net 22.5 score points (Annex Table 3.17). Education systems where schools had a high degree of autonomy over educational content (for one additional standard deviation); gross 22.1, and net 20.3 score points (Annex Table 3.17). Conclusion and Policy Implications How Thailand’s students perform academically today will determine the roles they will play in tomorrow’s world and how competitive Thailand’s economy will be in the future. Low academic achievement can have negative consequences for students’ future labor-market and income prospects and for their capacity to participate fully in society. it is essential, therefore, to monitor how well the nation has provided its young adults with the fundamental skills necessary to play a full part in the economy and society. 58 Learning Outcomes in Thailand What Can We Learn from International Assessments? Analysis of the data from international studies shows that Thailand’s students are still a long way from achieving strong educational scores in key subject areas. Thai students have performed less well on international tests than young people in other countries and have scored at the lowest proficiency levels on the international scales in all key subject areas. Within Thailand, students from the Bangkok Metropolitan area outperformed their peers from other areas, particularly students from the North-East regions, and this gap has widened over time. Big schools outperformed small schools, and urban students did better than those from rural schools. There is no evidence in the data to suggest that any material assistance was given to the disadvantaged schools to help them improve over time. This implies that in Thailand the goals of equity and equality in education have yet to be met. Various factors at the school level and system level appear to have a significant impact on the learning outcomes of students. Knowing more about these effects will provide policymakers with useful information to help them to improve the educational performance of Thailand’s students. Our analysis has pinpointed some school factors that lie behind the deterioration in student learning. These include a shortage of educational resources, a lack of school autonomy over budgeting and education content, and a lack of out-of-school learning time. All these factors have a negative effect on students’ academic performance. There are several policy options that can help achieve higher academic achievement by Thai students, and increase equity in Thai schools. Target the factors that affect learning quality. Our analysis has pinpointed some school factors that lie behind the deterioration in student learning, including, a shortage of educational resources, a lack of school autonomy over budgeting and education content, and a lack of out-of-school learning time. All these factors have a negative effect on students’ academic performance. Policymakers ought to consider these factors when deciding what policies to implement. For example, providing more educational resources and increasing the supply of qualified teachers are both steps that are urgently needed. Since it is clear that coaching schools in which students are taught how to get the right answer on an objective test does not help them to learn the skills necessary to cope with life, should these schools be eliminated from the education system or continue to be promoted? The answer to this question should be based on research, not on opinion. Target low-performing schools. The analysis shows that in Thailand, schools are segregated by the socio-economic backgrounds of their student intake. Small schools and non-urban schools are normally located in poorer neighborhoods, and face considerable disadvantages. More over, The PISA and TIMSS data show that low-performing schools tend to cater to students from low-income communities and are lacking in resources. As might be expected, educational resources have a positive impact on learning. Therefore, it is recommended that 59 Learning Outcomes in Thailand What Can We Learn from International Assessments? the Government allocates more educational resources to disadvantaged schools. However, care must be taken about which resources are chosen, for example our results showed that the possession of computers by schools has no positive impact on learning. A bigger impact can be gained by focusing on filling vacant teacher positions especially for science, and en suring that poorer schools have access to non-ICT resources such as laboratories and text books. Target low-performing students. Within schools low-performing students can be targeting through the creation of programs that intervene early enough in a student’s schooling to prevent them from falling behind, as well as remedial programs and special curricula for special needs students. Give schools more autonomy. Our results also suggest that students perform better in schools that have autonomy over their own budget decisions. Perhaps, the system where schools have only basic funding will not be able to raise student performance as expected. Therefore, Thai policymakers need to consider the merits of introducing more school autonomy instead of the Free Education scheme, especially as any form of fund raising by schools is currently prohibited. Raise standards for all students. This is an essential step for low-performing students. All students need to master reading and mathematics because these are the foundation of all other learning. Policymakers can introduce measures to raise the acceptable minimal standard in these subjects by altering the content and pace of teaching, increasing the amount of time a student spends on each subject, funding activities that support learning, and improving as sessment criteria. They can also increase the supply of factors that have been proven to have a positive effect on learning, such as instruction resources and the number of qualified teachers. In sum, education excellence in Thailand is still a challenge. The low scores of Thai students on international tests should serve as a wake-up call to the nation, public, government, and the education system as a whole. It is time to take serious action to improve the quality of education or the future citizens of Thailand will lack the skills needed to participate in society and to be competitive in the international community. Without those skills, they are also likely to be a drain on public resources because of their need for living support and welfare payments. 60 Learning Outcomes in Thailand What Can We Learn from International Assessments? References Ahuja, A., T. Chucherd and K. Pootrakool. 2006. “Human Capital Policy: Building a Competitive Workforce for 21st Century Thailand.� Monetary Policy Group, Bank of Thailand, Bangkok. Barro R.J. 2001. “Human Capital and Growth.� American Economic Review, Papers and Proceedings 91(2): 12-17. Barro, R.J. 1997. Determinants of Economic Growth: A Cross-section Empirical Study. MIT Press. Blinder, A. 1973. “Wage discrimination: Reduced form and structural estimates.� Journal of Human Resources 8(4): 436–455. Hanushek, E. and L. Woessmann. 2007. Education Quality and Economics Growth, World Bank, Washington, DC, 2007. Mullis, I. V. S., M. O. Martin, and P. Foy. 2008. TIMSS 2007 International Mathematics Report. Chestnut Hill, MA: TIMSS and PIRLS International Study Center, Boston College. Oaxaca, R. 1973. “Male-female wages differentials in urban labor markets.� International Economic Review 14(3): 693–709. OECD. 2009. PISA Data Analysis Manual: SPSS and SAS, Second Edition. Paris, France: Organization for Economic Cooperation and Development Psacharopoulos, G. and H.A. Patrinos. 2004. “Returns to Investment in Education: A Further Update.� Education Economics 12(2): 111-134. Todd, P and K. Wolpin. 2003. “On the Specification and Estimation of the Production Function for Cognitive Achievement.� Economic Journal: F3-F33. World Bank. 2008. Thailand Social Monitor on Youth 2008: Development and the Next Generation, World Bank, Washington, DC. World Bank. 2006. Thailand Social Monitor: Improving Secondary Education, World Bank, Washington, DC. UNdata 2009. Total Secondary Net enrolment rate. http://data.un.org/Data.aspx?d=UNESCO&f=series%3ANER_23 Accessed 12/27/2009. 61 Learning Outcomes in Thailand What Can We Learn from International Assessments? Annex 1 Table 3.1: Students’ Performance in PISA 2006 and TIMSS 2007 PISA 2006 Score* TIMSS 2007 Score** Countries Reading Mathematics Science Mathematics Science Chinese Taipei 496 549 532 598 561 Hong Kong 536 547 542 572 530 Indonesia 393 391 393 405 427 Japan 498 531 570 554 Korea 556 547 522 597 553 Malaysia dnp dnp dnp 474 471 Singapore dnp dnp dnp 593 567 Thailand 417 417 421 441 471 Sources: *OECD 2007c, **Martin et al 2008, Mullis et al, 2008 Note: dnp = did not participate. Table 3.2: Percentage of Students at Each Proficiency Level on Science Scale Below level 1 Level 1 Level 2 Level 3 Level 4 Level 5 Level 6 Japan 3.2 8.9 18.5 27.5 27.0 12.4 2.6 Korea 2.5 8.7 21.2 31.8 25.5 9.2 1.1 Hong Kong-China 1.7 7.0 16.9 28.7 29.7 13.9 2.1 Indonesia 20.3 41.3 27.5 9.5 1.4 0.0 0.0 Macao-China 1.4 8.9 26.0 35.7 22.8 5.0 0.3 Chinese-Taipei 1.9 9.7 18.6 27.3 27.9 12.9 1.7 Thailand 12.6 33.5 33.2 16.3 4.0 0.4 0.0 Source: OECD, 2007c Table 3.3: Trends in Thailand’s National Test Results for Grade 9 Students (Percentage correct) Year Thai language Mathematics Science English (% correct) (% correct) (% correct) (% correct) 2000 53.1 31.2 no test 39.0 2003 54.0 35.0 38.1 37.9 2006 43.4 31.1 39.4 30.8 2008 41.0 34.6 39.4 32.6 Source: Educational Testing Office, Ministry of Education (bet.obec.go.th/equal) 62 Learning Outcomes in Thailand What Can We Learn from International Assessments? Table 3.4: Trends from PISA 2000 to PISA 2006 Scale Reading Mathematics Science PiSA 2000 2003 2006 2000 2003 2006 2000 2003 2006 Korea 525 534 556 547 542 547 552 538 522 Hong Kong -China 525 510 536 560 550 547 541 539 542 Japan 522 498 498 557 534 523 550 548 531 Chinese Taipei dnp dnp 496 dnp dnp 549 dnp dnp 532 Macao - China 498 492 dnp 527 525 - 525 511 Thailand 431 429 417 432 417 417 436 429 421 Indonesia 371 395 393 367 360 391 393 395 393 Source: OECD 2007c Note: dnp = did not participate Table 3.5: Students’ Performance in Science of by Level of Parents’ Education in Thailand and Korea Country ISCED 0 ISCED 1,2 ISCED 3 ISCED 5,6 % student Science % student Science % student Science % student Science score score score score Thailand 10.2 402 64.8 411 15.9 436 9.0 490 Korea 1.5 - 14.9 497 58.9 520 24.7 549 Notes: ISCED 0 = without complete primary education ISCED 1 or 2 = complete primary or lower secondary education ISCED 3 = complete upper secondary education ISCED 5, 6 = complete tertiary education 63 Learning Outcomes in Thailand What Can We Learn from International Assessments? Table 3.6: Differences in Test Scores between Groups of Schools in Thailand Group (in terms of) Reading Math Science PiSA PiSA PiSA PiSA PiSA PiSA 2000 2006 2000 2006 2000 2006 Performance (Top /Bottom) Top performing schools (Bangkok) 446 466 449 464 456 463 Bottom performing schools (North-east) 420 392 424 396 428 399 Different Gap (Top - Bottom) 26 74 25 68 28 64 Location (Urban / Non urban) Urban schools 451 446 453 449 458 452 Non-urban schools 422 407 424 406 428 410 Different Gap (Urban - Non urban) 29 39 29 43 30 42 School size Big schools 461 439 470 443 463 440 Small schools 414 395 418 400 416 395 Different Gap (Big-small schools) 47 44 52 43 47 45 Public / private Public schools 433 417 439 422 436 418 Private schools 420 415 425 417 417 411 Different Gap (Public – Private) 13 2 14 5 19 7 Gender Boys 407 386 429 413 429 411 Girls 448 440 435 420 442 428 Different Gap (Boys – Girls) -41 -54 -6 -7 -13 -17 Source: PISA, Thailand database Table 3.7: Percentage of Thai Students at Low and High Reading Proficiency Levels by Gender Reading in % at Low proficiency level % at High proficiency level (below level 1 + level 1) (level 4+level 5) Boys Girls Boys Girls PISA 2000 51.1 27.3 3.5 6.5 PISA 2003 57.2 33.2 3.0 6.0 PiSA 2006 61.0 32.6 2.2 6.2 64 Learning Outcomes in Thailand What Can We Learn from International Assessments? Table 3.8: Performance of Thai Boys and Girls in PISA 2006 Performance Reading Mathematics Science Boys Girls Boys Girls Boys Girls Average score 386 440 413 420 411 428 Proportion of students who score 14 86 52 48 46 54 above 600 Proportion of students who score 1 9 8 2 4 6 at top 10 Proportion of students who score 10 0 8 2 10 0 at bottom 10 Table 3.9: Gender Difference in Science in PISA 2006 and TIMSS 2007 in Asian Countries Country PISA 2006 science score* TIMSS 2007 science score ** Boys Girls Difference Boys Girls Difference (Boys-Girls) (Boys-Girls) Chinese Taipei 536 529 7 563 559 4 Hong Kong 546 539 7 528 533 -5 Indonesia 399 387 12 435 432 1 Japan 533 530 3 556 552 4 Korea 521 523 -2 557 549 8 Malaysia dnp dnp dnp 466 475 -9 Singapore dnp dnp dnp 563 571 -8 Thailand 411 428 -17 462 480 -18 Sources: *OECD 2007c, ** Martin et.al (2008) Note: dnp – did not participate Table 3.10: Performance of Boys and Girls from PISA 2000 to PISA 2006 PiSA Score Group Reading Mathematics Science PiSA 2000 PiSA 2006 PiSA 2000 PiSA 2006 PiSA 2000 PiSA 2006 Boys 407 386 429 413 429 411 Girls 448 440 435 420 442 428 Difference -41 -54 -6 -7 -13 -17 (Boys – Girls) Source: Klainin et al, (2008) 65 Learning Outcomes in Thailand What Can We Learn from International Assessments? Table 3.11: How the Gender Difference Changed from PISA 2000 to PISA 2006 Gender difference (Boys/ Girls) in science Country PiSA 2000 PiSA 2006 Korea 20 -2 Hong Kong-China 9 7 Indonesia 5 12 Japan -7 3 Thailand -13 -17 Table 3.12: Simple Regression Analysis of Selected School Variables (Thailand) Grouping, selecting policy Change in score Read Math Science Selection from academic record (1= academic record, 0 = others) -8.7 -3.77 -3.08 Ability grouping (1= grouping for all subjects, 0 = no grouping) -15.22 -11.66 -10.06 Management & funding Private, public management(1= private, 0 =government) -2.26 -7.81 -4.67 Parent pressure & school choice One or more schools compete for students (1 = yes, 2 = no) 16.12 14.89 14.51 Schools under parents’ pressure (1 = yes, 2 = no) 23.7 28.62 25.67 Accountability policy Feedback on student performance to parents relative to other students in schools (1 = yes, 2 = no) 6.33 10.37 6.79 Feedback on student performance to parents relative to other students in other schools (1 = yes, 2 = no) 2.3 8.25 -0.23 Feedback on student performance to parents relative to national or regional (1 = yes, 2 = no) 9.61 14.73 5.81 Assessment results are posted publicly (1 = yes, 2 = no) 22.93 23.58 22.22 Achievement data used in evaluation of principal’s performance (1 = yes, 2 = no) 2.48 5.25 3.84 Achievement data used in evaluation of teachers’ performance (1 = yes, 2 = no) -9.13 -5.77 -4.26 Achievement data is used in decision about instruction of resource allocation to the schools (1 = yes, 2 = no) 4.88 7.2 1.88 Achievement data is tracked over time (1 = yes, 2 = no) 8.49 3.3 4.41 School resources Students / teacher ratio # -1.21 -1.24 -0.82 Teacher shortage index -18.03 -19.17 -18.21 No of students per computer for instruction 168.67 63.28 101.16 Resource availability index 21.25 18.55 18.54 Learning time at schools 8.55 9.18 8.24 Out of school learning time -0.16 1.94 0.68 Self studying time / homework 2.69 3.11 3.07 Index of enhanced learning activity 19.79 19.12 16.96 66 Learning Outcomes in Thailand What Can We Learn from International Assessments? Annex 2 Table 3.13: Multilevel Models - Admitting, Grouping and Selecting Admitting, grouping and selecting and Gross Net student performance Change in score p-value Change in score p-value School with ability grouping for -10.2 (0.000) -4.5 (0.002) all subjects (1= yes, 0= no) School with high academic selectivity of 30.4 (0.000) 18.1 (0.000) school admittance (1= yes, 0= no) School with low academic selectivity of -14.5 (0.000) -1.6 (0.264) school admittance (1= yes, 0= no) School with early selection -4.2 (0.331) -0.4 (0.927) (each additional year between the age of selection and the age of 15) System level number of school type or 6.9 (0.357) 3.3 (0.607) distinct educational program available to 15 year olds Source: OECD (2007b) Note: Shaded boxes = statistically significant Table 3.14: Multilevel Models - School Management and Funding (Public or Private) School management and funding and Gross Net student performance Change in score p-value Change in score p-value Schools being privately manage 20.0 (0.002) -2.6 (0.353) (1 = private, 2 = public) Schools with high proportion of funding -3.2 (0.000) -0.3 (0.46) from government (each additional 10% of funding from government sources) Source: OECD (2007b) Note: Shaded boxes = statistically significant 67 Learning Outcomes in Thailand What Can We Learn from International Assessments? Table 3.15: Multilevel Models - Parental Pressure and Choice Parental pressure and choice and Gross Net student performance Change in score p-value Change in score p-value Schools with high level of competition 17.9 (0.000) 1.9 (0.245) (1 = one or more schools compete for students; 0 = no schools compete for students) Schools with high level of perceived 11.2 (0.000) 2.0 (0.228) parental pressure (1 = there is pressure; 0 = pressure absent System with high proportion of 3.1 (0.525) 6.7 (0.076) competitive schools (each additional 10% of competitive schools) Source: OECD (2007b) Note: Shaded boxes = statistically significant Table 3.16: Multilevel Models- Accountability Policies Accountability policies and Gross Net student performance Change in score p-value Change in score p-value School informing parents of children’s 4.7 (0.140) 2.8 (0.139 ) performance relative to other students in the school (1- yes, 2= no) School informing parents of children’s 4.2 (0.100) 18 (0.228) performance relative to national benchmarks (1- yes, 2= no) School informing parents of children’s -5.0 (0.013) -1.4 (0.352) performance relative to other schools (1- yes, 2= no) School posting achievement data publicly 14.7 (0.000) 6.6 (0.000) (1- yes, 2= no) School using achievement data for -2.3 (0.076) 0.5 (0.711) evaluating principals (1- yes, 2= no) School using achievement data for 4.3 (0.076) -0.5 (0.711) evaluating teachers (1- yes, 2= no) School using achievement data for -4.8 (0.034) -4.3 (0.007) allocating resources (1- yes, 2= no) School with achievement data tracked -2.4 (0.327) -1.2 (0.443) over time (1- yes, 2= no) System with standard based examination 16.1 (0.028) 17.1 (0.266) Source: OECD (2007b) Note: Shaded boxes = statistically significant 68 Learning Outcomes in Thailand What Can We Learn from International Assessments? Table 3.17: Multilevel Model - School Autonomy School autonomy and Gross Net student performance Change in score p-value Change in score p-value School autonomy index in staffing 9.5 (0.000) -3.4 (0.005) (effect of one standard deviation of the index) School autonomy index in educational 0.9 (0.573) -0.8 (0.368) content (effect of one standard deviation of the index) School autonomy index in budgeting 1.1 (0.457) 1.5 (0.045) (effect of one standard deviation of the index) System average of school autonomy 0.7 (0.396) 1.5 (0.829) index in staffing (effect of one standard deviation of the index) System average of school autonomy index 22.1 (0.019) 20.3 (0.004) in educational content (effect of one standard deviation of the index) System average of school autonomy 27.2 (0.056) 22.5 (0.048) index in budgeting (effect of one standard deviation of the index) Source: OECD (2007b) Note: Shaded boxes = statistically significant 69 Learning Outcomes in Thailand What Can We Learn from International Assessments? Table 3.18: Multilevel Model - School Resources School resources and and Gross Net student performance Change in score p-value Change in score p-value Human resource indicators School average number of students per 0.33 (0.122) -0.16 (0.304) teacher (one additional student per teacher) School level index of teacher shortage -4.14 (0.000) -1.55 (0.073) (effect of one standard deviation of the index) Material resources School average number of computer for -12.5 (0.359) 2.5 (.0817) instruction per students (one additional computer per student) School level index of quality of educational 5.14 (0.000) 0.17 (0.789) resources (effect of one standard deviation of the index) Educational resources indicators School average student learning time for 14.3 (0.000) 8.7 (0.000) regular lessons in schools (one additional hour per week) School average students learning time -12.9 (0.000) -9.0 (0.000) for out-of-school lessons (one additional hour per week) School average students learning time for 3.8 (0.004) 3.1 (0.001) self study or home work (one additional hour per week) School average index of school activity 7.07 (0.000) 2.89 (0.000) to promote students’ learning of science (effect of one standard deviation of the index) Source: OECD (2007b) Note: Shaded boxes = statistically significant 70 Learning Outcomes in Thailand What Can We Learn from International Assessments? Annex 3 Glossary 1) For countries INDO = Indonesia, JPN = Japan, KOR = Korea, HKG = Hong Kong-China, MAC = Macao-China, SNG = Singapore, CH-T = Chinese Taipei, THA = Thailand. 2) For schools in Thailand under the responsibility of six jurisdictions: OBEC (The Office of Basic Education Council). For the purpose of data analysis, this group is separated into OBEC1 (which transferred from opportunity extension schools) and OBEC2 (which were in the General Education Department before the education reform). PRV (Private schools) BMA (Schools run by The Bangkok Metropolitan Administration) LOC (Schools under the Bureau of Local Education Administration in the Ministry of the Interior) SATIT (Secondary schools run by universities) VOC (Vocational schools and colleges in which 15-year-old students enroll. VOC.1 are private and VOC.2 are public.) 71 Learning Outcomes in Thailand What Can We Learn from International Assessments? 72 Learning Outcomes in Thailand What Can We Learn from International Assessments?