Indonesia MAIN REPORT A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning ÷ + 5 1 Main Report KEMENTERIAN PENDIDIKAN NASIONAL i The Word Bank Office Jakarta Indonesia Stock Exchange Building, Tower II, 12th floor Jl. Jend. Sudirman Kav. 52-53 Jakarta 12190 Tel : (+6221) 5299-3000 Fax : (+6221) 5299-3111 http://www.worldbank.org/id Published in December 2015 Indonesia – A Video Study of Teaching Practices in TIMSS Eight Grade Mathematics Classrooms: Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning is a product of the staff of the World Bank. The findings, interpretation, and conclusions expressed herein do not necessarily reflect the view of the World Bank, its board of Executive Directors, officers or any of its member countries. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map of this work do no imply any judgment on the part of the World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. The work is funded by the Government of the Kingdom of the Netherlands under the Dutch Education Support Program (DESP) Trust Fund. DESP provides support to the Government of Indonesia through the World Bank for the purpose of developing policies, studies, and programs that help the government achieve its education strategic plan. Indonesia A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning MAIN REPORT May 27, 2015 Table of Contents TABLE OF CONTENTS Acknowledgements......................................................................................................................xiii List of Abbreviations....................................................................................................................xvi Preface on the Indonesia Video Study Series.................................................................................xviii Executive Summary......................................................................................................................xx Study Background................................................................................................................................... xx Key Findings on Teacher Knowledge....................................................................................................... xxi A Bigger Toolbox:......................................................................................................................... xxi Better Tools: ................................................................................................................................. xxi Key Findings on Teaching Practices......................................................................................................... xxii Interaction.................................................................................................................................... xxii Questioning.................................................................................................................................. xxii Methods....................................................................................................................................... xxiii Use and Approach for Mathematical Problems............................................................................. xxiii Key Findings on Trends in Teaching Practices......................................................................................... xxiv Key Findings on the Beliefs-Knowledge-Practices Relationship............................................................... xxvi Implications: Relevance in the Context of Recent Policies on Teaching Practices and the 2013 Curriculum.................................................................................................................... xxvi Learning Approach....................................................................................................................... xxvi Teaching Approach....................................................................................................................... xxvi The 2013 Curriculum in the Context of Mathematics.................................................................. xxvi Promising Approach, but a Big Gap to Fill................................................................................... xxvi Implications: The Challenge of Changing Practices................................................................................. xxvii Effective Change in Practices Requires Realigning Teacher Knowledge and Beliefs........................ xxvii Implications: Relevance for the Teacher Professional Management System.............................................. xxviii Effective Use of Competency Testing Requires Political Will…..................................................... xxix …As Well as a Reliable and Effective Instrument.......................................................................... xxix Continuous Professional Development (CPD).............................................................................. xxix From Research to Action: A CPD Module Based on Study Findings....................................................... xxx An Innovative Pilot as One Potential In-service Approach............................................................ xxx Pre-service Training Must Also be in Line with the New Approaches............................................ xxxi Overall Key Messages.............................................................................................................................. xxxi Main Report v A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms Background and Context.............................................................................................................1 1.1 Teacher Reform in Indonesia........................................................................................................ 3 1.2 Mathematics Teaching in Indonesia.............................................................................................. 4 1.3 Video Study of TIMSS Classrooms in Indonesia........................................................................... 5 Overview of Design, Methods and Analytical Approach...............................................................7 2.1 Introduction................................................................................................................................. 8 2.2 Objectives..................................................................................................................................... 8 2.3 Research Questions....................................................................................................................... 9 2.4 Why a Video Study?..................................................................................................................... 10 2.5 Capturing Trends Over Time........................................................................................................ 10 2.6 Conceptual Framework................................................................................................................. 12 2.6.1 Incorporation of Student Learning into the Analysis......................................................... 15 2.7 General Methodological Aspects................................................................................................... 15 2.7.1 Mixed Methods Approach................................................................................................ 15 2.7.2 Unit of Study and Analysis............................................................................................... 16 2.7.3 Sampling.......................................................................................................................... 16 2.8 Instruments.................................................................................................................................. 19 2.9 Case Studies.................................................................................................................................. 20 2.9.1 Length and Scheduling of Visit ....................................................................................... 22 2.10 Linking Components.................................................................................................................... 22 2.10.1 Overview of Relationship Mapping.................................................................................. 22 2.10.2 Analytical Steps Framework.............................................................................................. 24 2.11 Macro to Mezzo to Micro Analysis................................................................................................ 25 2.12 Regression Modelling Approach.................................................................................................... 27 2.12.1 Selection of Control Variables........................................................................................... 27 2.12.2 Progressive Building-Up of the Model.............................................................................. 28 2.12.3 Presentation of Results in Relationship Matrices............................................................... 29 2.12.4 Data Mining for Patterns.................................................................................................. 30 2.12.5 Confounding Factors........................................................................................................ 31 Teaching Practices .......................................................................................................................33 3.1 Proportions, Trends and Cross-Country Comparisons.................................................................. 34 3.2 Relationships between Teaching Practices and Learning Outcomes............................................... 39 vi Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Table of Contents Mathematical Knowledge in Teaching.........................................................................................41 4.1 Teacher Subject and Pedagogy Competency Assessment............................................................... 44 4.2 Teacher Mathematical Scenarios Interview.................................................................................... 45 4.2.1 Summary of Case Studies on the Mathematical Scenarios................................................. 46 4.3 Linking MKiT Assessment and Scenarios Results......................................................................... 48 4.4 Relationship of Teacher Knowledge with Student Learning Outcomes.......................................... 48 4.5 Key Points on Teacher Knowledge................................................................................................ 50 Teacher Beliefs on Teaching and Learning....................................................................................51 5.1 Teacher Mathematical Beliefs Survey............................................................................................ 53 5.2 Case Study Interviews on Teacher Mathematical Beliefs................................................................ 56 5.3 Linkages Between Beliefs Survey and Beliefs Interview.................................................................. 58 5.4 Beliefs Reconceptualised............................................................................................................... 59 Exploring the Beliefs-Knowledge-Practices Relationship..............................................................61 6.1 The Beliefs-Knowledge Link......................................................................................................... 63 6.1.1 Macro View: Linking the Beliefs Survey and Teacher Assessment Results.......................... 63 6.1.2 Case Study Insights on the Beliefs-Knowledge Relationship............................................. 70 6.2 The Beliefs-Practices Link............................................................................................................. 72 6.2.1 Exploring Patterns in Beliefs-Practices.............................................................................. 73 6.2.2 Beliefs Theme: The Role of the Teacher............................................................................ 75 6.2.3 Beliefs Theme: Tools and Resources, Management and Evaluation................................... 77 6.2.4 Beliefs Theme: Practices for and Processes of Learning...................................................... 78 6.2.5 Case Study Teacher Insights on Beliefs-Practices........................................................................... 80 6.2.6 Key Concluding Points on Beliefs-Practices...................................................................... 84 6.3 The Knowledge-Practices Link...................................................................................................... 85 6.3.1 Linking Practices with MKiT Assessment Scores.............................................................. 86 6.3.2 Use of Practices in Relation to Knowledge Indicators....................................................... 87 6.3.3 Case Study Relationships.................................................................................................. 89 6.4 Key Concluding Points on Relationships...................................................................................... 90 Main Report vii A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms A New Conceptual Model of Beliefs-Knowledge-Practices for Indonesia’s Teachers.....................93 7.1 Representing Beliefs, Knowledge and Practices as Discs................................................................ 94 7.2 Disc Size Significance.................................................................................................................... 94 7.3 Disc Interaction............................................................................................................................ 95 7.4 Gravitational and Repelling Forces within the Model.................................................................... 97 7.5 The Cultural Dimension as a Key Driver...................................................................................... 99 7.6 Application of the Model to Specific Instances.............................................................................. 100 7.6.1 Large Dissonant Versus Large Congruent Zone................................................................ 100 7.6.2 Comparison of Higher and Lower Knowledge Teachers.................................................... 101 7.6.3 Key Conjectures from the Model...................................................................................... 103 Themes on Teaching Practices and Policies..................................................................................105 8.1 How Teacher Knowledge is Enacted in the Classroom.................................................................. 106 8.2 Teacher-Student Interaction.......................................................................................................... 113 8.2.1 What is a Student-Centered Approach?............................................................................ 113 8.2.2 Interaction and Recent Policies on Teaching Practices....................................................... 114 8.2.3 Interaction of Teachers and Students................................................................................ 114 8.2.4 What Type of Interaction Really Matters?......................................................................... 118 8.3 Teaching Practices Encouraged over the Past Decade.................................................................... 120 8.4 The National Exam and Other Influences on Teaching Practices................................................... 126 8.4.1 What Do the Trends Tell Us about Teacher Reform and Other Influencing Factors?......... 126 Overarching Conclusions from the Study Findings .....................................................................133 Turning Study Findings into Practice: Video Study Results Converted into a CPD Module.........137 10.1 Using Key Study Findings to Identify Areas for Improvement....................................................... 138 10.2 An Innovative Approach to CPD through the Use Of Video........................................................ 138 10.2.1 In-On-In Approach.......................................................................................................... 139 10.2.2 Elpsa Framework.............................................................................................................. 139 10.2.3 Incorporation of Videos into the Module......................................................................... 140 10.2.4 Facilitators as an Integral Part of Session Activities............................................................ 140 10.3 Implementation in Teacher Working Groups................................................................................ 141 10.4 Impact on Teacher Beliefs, Knowledge and Practices..................................................................... 142 10.5 Key Lessons Learned and Considerations...................................................................................... 143 Content........................................................................................................................................ 143 Implementation............................................................................................................................ 144 Effectiveness................................................................................................................................. 145 Scaling Up and Sustainability........................................................................................................ 146 10.6 Wider Policy Implications Emerging from the Pilot...................................................................... 146 viii Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Table of Contents References....................................................................................................................................149 Appendices...................................................................................................................................163 Coding Layers and Definitions................................................................................................................ 164 Graphical Mapping of Findings for Each Stage of Analysis...................................................................... 167 Distribution of Responses for Each Statement in the Mathematical Beliefs Survey.................................. 168 Likert Scale Responses (Scale Of 1-5) .......................................................................................... 168 Frequency of Teaching Practices in the Teacher’s Lessons............................................................... 170 Ranking Responses....................................................................................................................... 171 Summary of Teacher Responses to Mathematical Scenarios..................................................................... 172 Technical Report as a Separate Appendix................................................................................................. 173 Main Report ix A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms INDEX OF FIGURES Figure 1: Estimated Impact of High vs. Low Performing Teachers on Student Achievement................... 2 Figure 2: Concept of Measuring Trends in Teaching Over Time............................................................. 11 Figure 3: Framework for Analysis of Teacher Practices, Beliefs and Knowledge....................................... 12 Figure 4: Mapping of Instruments by Full Sample vs. Case Study Sample.............................................. 20 Figure 5: Data Shown as Levels of Perspective for MKiT and Mathematical Beliefs................................ 23 Figure 6: Four Stages of Analysis: (1) Individual; (2) Component; (3) Cross-component; and (4) Full-component........................................................................................................... 24 Figure 7: Breakdown of Macro-Mezzo-Micro Approach to Analysis....................................................... 26 Figure 8: Example of Coding of Regression Results to Indicate Relationships......................................... 29 Figure 9: Proportional Breakdown of Lesson Time................................................................................. 34 Figure 10: Duration for Different Activities in Indonesia and Other Countries........................................ 35 Figure 11: Time for Different Learning Activities in Mathematics Lessons............................................... 35 Figure 12: Percentage of Time for Public Interaction and Private Interaction............................................ 36 Figure 13: Public (Whole Class) Interaction Breakdown: Teacher and Student Involvement..................... 36 Figure 14: Average Number of Words Spoken by Teacher and Students During the Lesson...................... 37 Figure 15: Mathematical Time Divided into Problem vs. Non-Problem Segments (Percent of total)......... 38 Figure 16: Scatter Plot of Teacher Subject and Pedagogy Scores on the Teacher Assessment...................... 44 Figure 17: Relationship between Teacher Knowledge and Student Learning............................................. 49 Figure 18: Likert Scale Statements with Large Majority of Responses being Either Positive or Negative.... 53 Figure 19: Distribution of Frequency Scale Responses in the Mathematical Beliefs Survey....................... 55 Figure 20: Frequency Scale Results on a Sample of Teaching Practices...................................................... 55 Figure 21: Proportions of the Relationship between Beliefs Questions and Knowledge............................. 63 Figure 22: Beliefs Statements Having a Relationship with Both Subject and Pedagogical Knowledge........ 65 Figure 23: Beliefs Statements Having a Relationship with Subject But Not Pedagogical Knowledge......... 66 Figure 24: Examples of Beliefs Statements with a Relationship with Pedagogical But Not Subject Knowledge.................................................................................................................. 66 Figure 24: Example of Beliefs Statement with Subject and Pedagogy MKiT Scores in Opposite Directions................................................................................................................ 67 Figure 26: Beliefs Statements where MKiT and Student Learning have Opposite Relationships............... 68 Figure 27: Scatter Plot of Teacher Subject and Pedagogy Scores on the Teacher Assessment...................... 71 Figure 28: Matrix with Beliefs Role of the Teacher related to Actual Practices........................................... 75 Figure 29: Matrix with Beliefs on Teaching Approach, Assessment, and Resources Related to Actual Practices........................................................................................................................ 77 Figure 30: Matrix with Beliefs on Learning Related to Actual Practices..................................................... 78 Figure 31: Teaching Practices of Teachers in Relation to Their Overall MKiT Score................................. 87 Figure 32: Conceptual Model of Beliefs-Knowledge-Practices Interaction................................................ 94 Figure 33: Comparison of Teachers Reacting to Influences....................................................................... 96 Figure 34: Large Dissonant vs. Large Congruent Zone............................................................................. 98 Figure 35: Illustrative Contrasting Example between Higher and Lower Knowledge Teachers.................. 99 x Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Index of Figures Figure 36: Distribution of Scores of Mathematics Teachers in UKG......................................................... 104 Figure 37: Integrated Framework for the Teacher Professional Management System................................. 109 Figure 38: Proportions of Whole-class vs. Group and Seatwork................................................................ 113 Figure 39: Whole-class Work is on the Increase, while Group Work is Declining..................................... 113 Figure 40: Ratios of Words from the Teacher for Every Word from a Student........................................... 116 Figure 41: Proportion of Lesson Time by Methods Used.......................................................................... 120 Figure 42: Problem and Question Contexts.............................................................................................. 120 Figure 43: Theoretical Model Showing Alignment of Teaching Practices with Knowledge and Beliefs....... 123 Figure 44: Proportion of Mathematics Time by Structure of Interaction................................................... 125 Figure 45: Proportion of Whole Class Time by Interaction Type.............................................................. 125 Figure 46: Proportion of Teaching Methods and Approaches Used........................................................... 126 Figure 47: Proportion of Mathematics Time as Problem vs. Non-problem Time...................................... 127 Figure 48: Proportion of Problem Work as Involving Routine versus Non-Routine Problems................... 128 Figure 49: Proportion of Problem Work as Involving Mathematical Language vs. Real World Problems... 128 Figure 50: Differences in Teaching Practices of Certified Teachers vs. Uncertified Teachers in Indonesia TIMSS Video Study of Mathematics for the 2011 Phase......................................... 129 Figure 51: Example of ELPSA for the Topic of Right Triangles................................................................. 138 Main Report xi A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms INDEX OF TABLES Table 1: Research Data, Sources and Sample Grouping......................................................................... 19 Table 2: Case Study Activities and Their Purpose.................................................................................. 21 Table 3: Variables Used in Various Layers for in the Regression Analysis............................................... 28 Table 4: Sources of Data for MKiT analysis.......................................................................................... 44 Table 5: Beliefs Lenses and Description of Categories Used in the Analysis........................................... 52 xii Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Acknowledgements ACKNOWLEDGEMENTS This report is the result of a joint program between the World Bank and the Ministry of Education and Culture (MOEC). The report was authored by Andy Ragatz under the operational leadership of Susie Sugiarti and overall guidance of Susiana Iskandar and Ratna Kesuma, but it is just one piece of an extensive two-phase study that involved a large team and support from many government officials and stakeholders. The original design of the study was done by Dr. Frederick Leung (University of Hong Kong). The design was based heavily on extensive work done for the report, Teaching Mathematics in Seven Countries: Results from the TIMSS 1999 Video Study (Hiebert et al, 2003) in order to allow for comparison of Indonesia’s results to those of other seven other countries that conducted similar studies. Subsequent components were added to the 2011 phase. The implementation was led by Susie Sugiarti, with a core team that consisted of senior mathematics instructors from the Center for Development and Empowerment of Mathematics Teacher and Education Personnel (P4TK), the Educational Quality Assurance Institution (LPMP), and some junior secondary schools. Team members included Adi Wijaya, Budiharjo, Erwin Roosilawati, Al Krismanto, Rachmadi Widdiharto, Puji Iryanti, Reubun, Suhendro, Sri Yuniati, Suwarkono, Sri Yuniarti, Ade Sunawan, and Isfarudi. The study team tirelessly analyzed and reanalyzed the videos to provide the essential coding results, and produced initial analysis and qualitative studies that contributed to this report. Aria Jalil was also a key author of various pieces and conducted the case studies with Andy Ragatz. The MOEC was heavily involved in all stages. Fasli Jalal (former Directorate General of Quality Improvement of Teacher and Education Personnel) provided the vision and initial provisions to launch the study. Baedhowi (former Directorate General for Quality Improvement of Teacher and Education Personnel), Sumarna Surapranata (Director General of Teacher and Education Personnel Development), Totok Suprayitno (Head of Agency of Research and Development), Nizam (Head of Center of Education Assessment) provided supports for the dissemination of the study findings through various events. Bastari, Burhanudin Tola, Nugaan Yulia Whardani Siregar and Hari Setiadi (former Heads of Center of Education Assessment) supported the development of student and teacher assessment instruments and access to TIMSS schools. Abi Sujak, Unifah Rosyidi and Dian Wahyuni (officials of former Agency of Human Resource Development on Education and Culture and Education Quality Assurance) not only provided leadership and guidance, but also continually participated in and supported the activities. Other technical team members are listed on the next page. The report benefited from the excellent comments and advice of several World Bank experts, including Barbara Bruns (peer reviewer), Helen Craig (peer reviewer), Muna Salih Meky (peer reviewer), Susiana Iskandar, Ratna Kesuma, Samer Al-Samarrai, Tazeen Fasih and Cristobal Ridao-Cano, as well as production support from Santi Sugiarti Santobri. But the biggest contributors to this effort were the over 200 teachers who welcomed us into their classroom and participated in the program. Main Report xiii A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms Members of Technical Team of MOEC TIMSS Video Study 2011 Center of Education Assessment (Puspendik) School sampling, student and teacher test instrument development, socialization and instrument piloting, pre-test and teacher assessment, post-test: 1. Rumondang Purwati (Coordinator/Contact Person) 2. Erika Afiani (Contact Person for TIMSS Score results) 3. Ainun Salim 4. Rahmah Zulaiha 5. Rustomo 6. Rina Rosdiani Patria 7. Lilis W 8. Devi 9. Siti Nadhiroh 10. Fahmi 11. Fitri 12. RR Herlina 13. Malik Azis 14. Zul Arsiah 15. Aris Subranto 16. Benny Widaryanto 17. Rahmawati 18. Nonny S 19. Deddy Rokhidin Directorate General for Quality Improvement of Teacher and Education Personnel/Agency of Human Resource Development on Education and Culture and Education Quality Assurance 1. Gogot Suharwoto 2. Emy Fitriani 3. Fitriadi Student pre-test and teacher assessment and post-test (Administrator test) 1. Fitriadi 2. Laura Rossiana 3. Yusrizal xiv Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Acknowledgements Directorate for Management of Junior Secondary Education Student pre-test and teacher assessment and post-test (Administrator test) 1. Evi Maharani 2. Henny Harsian Videotaping 1. Juandanilsyah 2. Nyoman Ekasanagiri Education Quality Assurance Institute & Education Office (Non Core Team) Post test (Administrator test) 1. Soko B. Oswaldus (Pengawas Dinas Pendidikan Ende, NTT) 2. Drs. Ola Sanga Yohanes, MM (Widyaiswara LPMP NTT) 3. Baharudin (Widyaiswara, LPMP Sulawesi Selatan) 4. Amir Daud (Widyaiswara, LPMP Sulawesi Selatan) 5. Hadi (Widyaiswara, LPMP Sulawesi Tengah) 6. Lapahisa (Widyaiswara, LPMP Sulawesi Tenggara) 7. Bobi Abdul Ghani (Widyaiswara, LPMP Gorontalo) 8. Kasim (Widyaiswara, LPMP Kalimantan Barat) 9. Husein Siregar (Widyaiswara, LPMP Kalimantan Timur) Main Report xv A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms LIST OF ABBREVIATIONS AC Action-related Competencies BK Basic Knowledge BKP Beliefs, Knowledge and Practices CCK Common Content Knowledge CPF Connectionist-Progressive-Facilitator (used to describe teachers tending to hold the combined beliefs categorizations) CK Content Knowledge CS Case Study (often used as, for example, CS1 representing Case Study 1) CLCC Creating Learning Communities for Children (a UNICEF program in Indonesia) COACTIV Professional Competence of Teachers, Cognitively Activating Instruction, and the Development of Students’ Mathematical Literacy (study) CTL Contextual Teaching and Learning (which refers to an Indonesian policy initiative) IEA International Association for the Evaluation of Educational Achievement KCS Knowledge of Content and Students METE Mathematics Education Traditions in Europe (project) MKiT Mathematical Knowledge in Teaching. (Also often referred to as Mathematical knowledge for teaching, and combined Content Knowledge and Pedagogical Content Knowledge) xvi Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning List of Abbreviations MOEC Ministry of National Education and Culture in Indonesia (previously MONE: Ministry of National Education) MT21 Mathematics Teaching in the 21st Century (project) PCK Pedagogical Content Knowledge PGCE Post Graduate Certificate in Education PISA Programme for International Student Assessment RC Reflective Competencies RQ Research Question SCK Specialized Content Knowledge TEDS-M Teacher Education and Development Study in Mathematics (cross-country study) TIMSS Trends in International Mathematics and Science Study TTI Traditional-Transmissionist-Instructor (used to describe teachers tending to hold the combined beliefs categorizations) WB World Bank Main Report xvii A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms PREFACE ON THE INDONESIA VIDEO STUDY SERIES This report is one in a series stemming from the Indonesia Trends in International Mathematics Study (TIMSS) Video Study project, a research program to examine in extensive detail the practices of mathematics teachers in eighth grade classrooms across Indonesia. More than 200 mathematics teachers participated in the study from 2007 to 2011. The study involved in-depth research based on quantitative and qualitative data analysis into teaching methods and behaviors in the classroom, and the impact of these various teaching methods on student learning outcomes. The main findings from the study were later used to develop a pilot training module for mathematics teachers to use in regular workshops as part of their continuous professional development (CPD). The study itself was comprehensive and highly complex. The analysis utilized multiple frameworks and sub- frameworks. The dynamic interactions analysis involved literally thousands of regressions and connected qualitative data from interviews of various types with quantitative data involving video coding of teaching practices, teacher competency testing, surveys, and student pre- and post-tests. Because of the study’s complexity, providing details of all the findings while simultaneously conveying key messages in a digestible manner would simply not have been effective; the messages would become lost in the depths of the technical analysis. For this reason it was decided to separate most of the technical analysis from the main report. A Technical Report has been developed to provide the important details of the design, analysis and general findings. This Main Report, on the other hand, does not attempt to provide all details, but rather just key aspects of the design and analysis. Only key findings are presented and they are grouped in a thematic manner. For readers interested in details on the design, instrument development, implementation and results, the Technical Report will be most relevant. For readers interested in a summary of key findings, the Main Report will be of more interest. In addition, a series of briefs and infographics has been created to cover specific topics that would make the study results accessible to different audiences. Finally, a videograph was created that provides an animated story about the video study, how it relates to key policies and how the CPD module was developed based on the study findings to support strengthening of teaching practices. xviii Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Preface on the Indonesia Video Study Series Main Report Technical Report Background and context Full literature review of international research related to the components of the study Brief overviews of international research related to various In-Depth description of design, methods and analytical approach components of the study Overview of Design, Methods and Analytical Approach Full results of all analysis Select results on teaching practices, beliefs and teacher Detailed presentation of results including step-by-step knowledge Select results Full description of new conceptual model for the dynamic beliefs- knowledge-practices relationship Full description of new conceptual model for the dynamic Very little policy discussion beliefs-knowledge-practices relationship Thematic presentation of key teaching practices, trends and Full appendices of instruments, case study profiles, results used for policy implications pattern analysis Very few appendices; references instead to the Technical Report Summary of pilot teacher continuous professional development module that put the study findings into practice Main Report xix A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms EXECUTIVE SUMMARY Indonesia is a committed participant in the Trends in International Mathematics and Science Study (TIMSS), the Program
 for International Student Assessment (PISA) and the Progress
 in International Reading Literacy Study (PIRLS) international student examinations and is one of
the few non-OECD (Organisation for Economic Cooperation and Development) countries to participate so fully. Indonesian student performance in these examinations has been relatively low, even when taking socio-economic levels into account. For example, 8th grade mathematics in Indonesia ranked 39 out of 43 participating countries in TIMSS 2011, and its score was more than one standard deviation below the international average. While the results are useful in providing an indication of Indonesia’s relative standing in student achievement and its progress over time, the real challenge is to take the next step and translate the results into an understanding of the factors driving student learning outcomes and what might be done to enhance student achievement in Indonesia. STUDY BACKGROUND The Indonesia TIMSS Video Study of 8th-grade mathematics teachers was a collaborative study between the Ministry of Education and Culture (MOEC) and the World Bank under the Dutch Education Support Program (DESP). The study aimed to determine not only what takes place in the classroom, but also gain a better understanding of why it takes place and how teaching practices relate to student learning. Detailed coding of videotaped lessons for 200 teachers provided insights on classroom activities. The use of video also allowed for in-depth explorations on key practices such as the handling of student misconceptions and the use of questioning. The study is one of the most wide-reaching and in-depth studies of classroom practices in the world. It followed a coding methodology used by seven countries in the 1999 TIMSS video study, allowing for cross-country comparisons. It also included two phases (2007 and 2011), allowing for analysis of trends in practices. The study goes beyond what happens in the classroom, also attempting to understand why it happens through exploration of the relationship of teaching practices with teacher mathematical knowledge and mathematical beliefs. Practices are also explored in relation to student learning to understand what may be most effective in the Indonesia context. The research was later turned into action, with the findings and video footage used to produce a course module for use in Indonesia’s mathematics teacher working groups. xx Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Executive Summary KEY FINDINGS: TEACHER KNOWLEDGE Of all education system variables, teacher knowledge emerged as having the strongest relationship with student learning outcomes. While the positive relationship itself is not surprising, the study uncovered the important ways in which knowledge plays a role and how it is employed in the classroom. It can be summarized as teachers having a bigger toolbox, as well as better tools. A BIGGER TOOLBOX: Higher knowledge teachers tended to utilize a greater range of practices. Techniques such as investigation, open- ended questioning, the use of mathematical language and symbols and the use of non-routine problems and applications tended to be found more in classrooms with higher knowledge teachers. This appears to be in part because the techniques themselves require a greater amount of subject mastery to be conducted effectively. For example, open-ended questioning requires that the teacher develop questions that could be answered in a variety of ways and tend to be more complex. When in the classroom, the teacher must then be prepared for the different possible responses of students, have contingencies for the different responses and possibly also involve higher-order thinking. BETTER TOOLS: While higher knowledge teachers tended to have a bigger toolbox, they also tended to have better tools. Further in-depth qualitative analysis uncovered important differences in the ways in which higher knowledge teachers used the same practices (giving them better tools). Teachers with higher knowledge tended to: 1. Make fewer mathematical errors 2. Use more accurate language and be more concise in their explanations 3. Come up with new problems that were not originally in the lesson plan 4. Make a greater attempt to probe student thinking 5. Be more able to spot student misconceptions and effectively correct them 6. Involve a greater range of students, including those who might not provide correct answers 7. Demonstrate more confidence and command in the classroom 8. Stimulate more teacher-student interaction, with the interaction involving more higher order thinking 9. Use questioning more and encouraging students to think and share ideas rather than just follow instructions 10. Make connections between procedures and concepts Main Report xxi A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms KEY FINDINGS: TEACHING PRACTICES Teaching practices were explored mainly through the coded video data to determine proportions of lesson time for each practice and counts of frequency of use. These practices were also examined from a qualitative standpoint to better understand how teachers actually utilized these practices. The relationship of these practices were also related to student learning outcomes to provide an indication of whether they might be effective in the Indonesian context, but with an understanding that its effectiveness is highly contextual and dependent on the teacher and the way in which he/she actually enacts the practice in the classroom. INTERACTION One of the key focus areas for the Video Study was the interaction that takes place between teachers and students. Whole-class time tended to be dominated by a teacher-centered approach, with 74 percent of time being teacher- only. Teacher-student interaction in whole class time was only 11 percent, but had a positive relationship with student learning whereas teacher-only time had a negative relationship. For group work and seatwork, an important finding is that students whose teachers spent a higher proportion of time monitoring and facilitating had higher learning outcomes. These findings of the positive relationship of teacher- student interaction across whole-class, group work and individual seatwork highlight the critical importance that teacher-student interaction plays in Indonesia’s mathematics classrooms for student learning. They also indicate that a student-centered approach can be effective in the Indonesian context, while the teacher-centered approach found in most classrooms tends to be less effective. QUESTIONING Questioning can be a powerful instrument in a mathematics teacher’s repertoire of practices, serving various purposes in defining the dynamics of the classroom and developing student learning. An in-depth exploration of how teachers use questioning revealed that the questioning tends to be superficial, with nearly 90 percent of student responses involving a single word and very rarely were the questions involving higher order thinking or explanation of reasoning. The use of open-ended questions was rare, with nearly all questions being closed. Teachers also tended to use pseudo questions or testing questions, very rarely using the more effective questioning techniques of genuine questions or directing and provoking questions. xxii Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Executive Summary METHODS Tied to the fact that a teacher-centered approach tended to dominate in most classrooms, most lesson time involved exposition and lecturing, making up 61 percent of mathematics lesson time. The more enquiry and real-world context methods of investigation and hands-on practical work tended to be very rare, making up only 1 percent and 7 percent, respectively. In terms of relationships with student learning outcomes, exposition/lecturing had a negative relationship whereas investigation and problem-solving had a positive relationship. USE AND APPROACH FOR MATHEMATICAL PROBLEMS There was little variation in the approaches used for mathematical problems, which tended to be nearly all routine and closed, and involving mathematical language and symbols. Problems that were non-routine, open or involving real-world contexts were extremely rare. In terms of relationships with student learning outcomes, non-routine and open problems and problems using the context of mathematical language and symbols had a positive relationship. The negative relationship of student learning outcomes with real-world contexts was surprising, but also may have had to do with the fact that the student tests were based on Indonesia’s National Exam and TIMSS, which tend to focus on procedures and formulas rather than applications and real-world contexts. KEY FINDINGS: TRENDS IN TEACHING PRACTICES Trends in teaching practices between 2007 and 2011 indicate a shift to a more teacher-centered, procedural, routine approach. Teacher-only time as a proportion of whole-class time increased from 59 to 64 percent, while the teaching method of exposition/lecturing as a proportion of mathematics time increased from 52 to 61 percent. Group work decreased from 21 percent of lesson time to 11 percent. New content increased from 47 percent of lesson time to 58 percent. The proportion of problem work involving non-routine problems (vs. routine) decreased from 12 to 4 percent, while problems involving real-world contexts (vs. mathematical language and symbols) decreased from 11 to 7 percent. In problem set-up, use of procedures increased from 40 to 64 percent, while stating a concept dropped from 54 to 30 percent. What might be driving these shifts? Influences are complex and cannot be narrowed down to a couple of factors. Still, there do seem to be some factors that help explain the trends. One appears to be that the annual National Exam (Ujian Nasional, or UN) taken by all primary and secondary students is having greater influence on teaching practices and that teachers are adapting their teaching practices to “teach to the test”. In 2007 and 2011 teachers were asked what factors influenced their teaching. The results indicate that the National Exam had greater influence in 2011 than 2007. Regression analysis indicates that teachers who used more teacher-centered approaches such as exposition, closed-questioning and routine problems also tended to say that the National Exam played a greater role in their selection of practices. Main Report xxiii A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms Another likely factor seems to be the curriculum. Similar to the National Exam, the results indicate that the curriculum had greater influence in 2011 than 2007. Regression analysis indicates that teachers who spent more time on new content and used more exposition also tended to say the curriculum had greater emphasis on their practices. The teacher reform that effectively began in 2006 was a main focus of this study, with the first phase of the study in 2007 being a useful baseline of teaching practices. Certification, the key program of the reform, was examined as a possible influencing factor. Interestingly, there were almost no statistically significant differences between the teaching practices of certified and uncertified teachers when controlling for other factors. The only difference was that certified teachers tended to have a larger proportion of whole-class time. There was also no difference in the student learning outcomes of certified vs. uncertified teachers. Upgrading may have had an effect, with students of teachers holding a four-year degree tending to have better learning outcomes. For practices, teachers with four-year degrees tended to use more open-ended questioning, and a higher proportion of group work, and were less likely to use procedural approaches to problems, all of which actually go in the opposite direction of the trends. While the teacher reform may be having some effect, these changes are not clear when looking at 8th grade mathematics classrooms and do not appear to explain the identified trends. KEY FINDINGS: THE BELIEFS-KNOWLEDGE-PRACTICES RELATIONSHIP The study is, at its heart, an exploration of interplay among three components: teaching practices, teacher mathematical beliefs and mathematical knowledge in teaching. Knowledge and beliefs are two widely researched areas in relation to their relationship with teaching practices. The knowledge relationship with respect to teaching practices was already touched upon. For beliefs, teachers who tended to have a transmissionist or traditional belief orientation tended to also use more teacher-centered and routine practices. Their students also tended to have lower learning outcomes compared with teachers with a connectionist orientation. What was of more interest, though, is that teachers whose students tended to have the highest learning outcomes had what could be described as a very flexible belief system. They actually argued against there being “best practices”, instead saying that what is best practice is highly contextual, depending on factors such as the achievement level and learning affinities of the students, the topic taught and the stage of the lesson. For example, while exposition may not be the most effective approach in general, these teachers recognized value in using it at key stages of the lesson and while investigation may be a positive approach, it is not applicable for every topic. Their flexible beliefs allowed them to adapt their practices to the context and they had a bigger toolbox of practices at their disposal. Through a process of “data mining” by using regressions to explore thousands of combinations of belief statements, practices and teacher knowledge, relationships were identified. In-depth case studies and qualitative analysis were also used to understand their connections. Through this process a new conceptual model of the beliefs-knowledge- practices relationship was developed as a way of thinking about effectiveness of teaching.1 1 The BKP (Beliefs-Knowledge-Practices) Congruency Model was developed as part of a doctoral thesis by Ragatz (2013). xxiv Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Executive Summary Conceptual Model of Beliefs-Knowledge-Practices Interaction PRACTICES USED (actions) Dissonant Zone Kn ence Ab ne ow Zo s sen lief Zo led ne Ab Be ce ge Congruent Zone (practices-knowledge-beliefs all aligned in congruence) Kn s rac o ow d no tice in p ed t an led t u sed t ti ge sed r u ut no sep in dg ld, b ara pra te wle he eo fro tices kno eliefs m c be Practice B KNOWLEDGE BELIEFS lie fs (tools) Absence Zone (perceptions) The following are key points of the model: • Teachers are most effective when the practices they use align with their knowledge (they know from a subject and pedagogical standpoint how to use the practice) and beliefs (they believe the practice is effective in the teaching-learning process), represented by the “Congruent Zone”, where there is full alignment. • Teachers are least effective in using practices that do not align with their knowledge or beliefs. This is represented by the “dissonant zone”. It is an unnatural state that may occur when teachers are required due to curriculum, school policies or other factors that oblige the teacher to teach in a way she or he would not choose on their own. • Effectiveness of a practice (e.g., inquiry-based learning) is relative and contextual. Whereas it can be an effective method for a teacher who has the proper knowledge and beliefs, it can be an ineffective practice for a teacher who does not have this knowledge and/or beliefs. This model was developed largely as a tool for discussion and simulation in relation to Indonesia’s implementation of the 2013 curriculum, which requires teachers to utilize very different practices from those currently used, as evidenced in the Video Study. Main Report xxv A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms IMPLICATIONS: RELEVANCE IN THE CONTEXT OF RECENT POLICIES ON TEACHING PRACTICES AND THE 2013 CURRICULUM Policies on teaching practices over the past decade have tended to encourage progressive practices focused on active learning with greater student interaction and collaboration as well as an emphasis on enquiry-based learning, reasoning, innovation, creativity and higher order thinking. The 2013 curriculum places even greater emphasis on these practices, requiring a distinct shift in the roles of students and teachers and a change in teaching methodology. LEARNING APPROACH The new learning approach encourages students to seek out and make observations. Students are directed to formulate the problem (ask), not just to solve the problem (answer). They are expected to think analytically (make a decision), rather than just in a mechanistic manner (routine). Students are also expected to work together and collaborate in solving problems. TEACHING APPROACH This more open and student-centered teaching/learning paradigm requires that the teacher become more of a guide and facilitator than simply the provider of information through one-way teaching. Classroom practice must foster independent learning, observation, questioning, processing, making judgments, communicating and creating. THE 2013 CURRICULUM IN THE CONTEXT OF MATHEMATICS For mathematics education, this new approach emphasizes the use of investigative and practical (hands-on) teaching approaches with more of a connection to real-world contexts. The lesson activities should encourage higher-order thinking through techniques such as the use of open-ended questioning, non-routine problems, and the application of various enquiry methods. Teachers are expected to use a student-centered learning paradigm with classroom practice to foster independent learning, observation, questioning, processing, making judgments, communicating and creating. PROMISING APPROACH, BUT A BIG GAP TO FILL Such findings of the Video Study demonstrated that teachers will need to teach in a very different style than currently tends to be used and that there is a large gap to fill. In nearly every case, the practices encouraged in the 2013 curriculum are the ones rarely used. The student-centered learning approach encouraged in the 2013 curriculum is clearly in contrast to existing practices. While problem-solving was used 22 percent of the time, approaches of practical work and investigation, which align with the new curriculum, were rare. Open-ended questioning, non- routine problems and real-world contexts were particularly rare. Overall, the results indicate there is a significant gap between practices used in 2011 and what is expected through the 2013 curriculum. xxvi Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Executive Summary IMPLICATIONS: THE CHALLENGE OF CHANGING PRACTICES Changing practices is notoriously difficult, as can be seen in previous Indonesia curriculum reforms. The 1984 curriculum reform was specifically designed for student-centered learning, but subsequent studies indicate that the teacher-centered approach held strong. Teachers take the information and interpret it in the context of their existing beliefs and environmental situation. With weak or limited training, teachers will be even less likely to change their practices as expected. With limited support the teachers will often not implement in the intended manner, possibly with detrimental results. Change requires a long-term, consistent process. EFFECTIVE CHANGE IN PRACTICES REQUIRES REALIGNING TEACHER KNOWLEDGE AND BELIEFS A useful model for understanding teachers’ use of practices is to consider two major influences: their subject and pedagogical knowledge, and their beliefs of how students learn and how to best teach. Teachers will attempt to align their practices to their knowledge and beliefs. The Video Study results indicate that it is when the practices are aligned with knowledge and beliefs that teachers are most effective. When they have to utilize practices for which they do not have sufficient mathematical or pedagogical knowledge or that do not align with their beliefs on the teaching-learning process, then they will not be effective. For example, a teacher who is forced to use a student- centered approach where they are a guide and facilitator rather than an instructor will struggle if he/she does not know how to use the approach and/or he/she does not believe this is the way students learn. Theoretical model showing alignment of teaching practices with knowledge and beliefs PRACTICES Dissonant Zone PRACTICES Congruent Congruent Zone Zone KN O KN S W EF OW LE LI BE DG LE E S DG EF E LI BE Teacher using many practices not aligned Teacher whose practices are mostly with her knowledge set or beliefs aligned with her knowledge set or beliefs Main Report xxvii A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms The implications for the 2013 curriculum, as represented in this model, are crucial. If teachers do not have sufficient knowledge in the new approaches and they do not believe they are effective approaches then teachers will do one of two things: (1) they will revert back to the practices that do align with their knowledge and beliefs; or (2) they will continue to use the practices, but use them ineffectively. This second result could even be more detrimental (in the short term) than if teachers simply use the practices for which they are capable. In order for teachers to effectively implement the new curriculum with the expected practices, teachers need continual, periodic support to progressively build the knowledge and beliefs necessary to utilize the new practices. IMPLICATIONS: RELEVANCE FOR THE TEACHER PROFESSIONAL MANAGEMENT SYSTEM MOEC has undertaken the massive task of developing a Teacher Professional Management System (TPMS), where the key elements of this system form an “integrated framework” of requirements. The system makes the teacher directly accountable for both work performance through annual appraisal (PKG) and professional development through continuous professional development (CPD). The test of minimum standard in professional and pedagogic competency (UKG) must also be passed in order to progress between the professional levels and the rate of progression depends on a credit point total based on the PKG (annual performance-based teacher appraisal) and CPD. The integrated framework of the TPMS What a teacher does Feedback based on observation Performance Appraisal Continuous Competency Professional Test Development What a teacher knows Tools for improvement Identi cation of knowledge areas for improvement Tailored plan based on identi ed needs xxviii Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Executive Summary The results of the Video Study highlight just how important teacher knowledge is in relation to student learning outcomes. The UKG can serve important purposes related to teacher management, including the hiring of high- quality teachers, identifying areas of improvement for incumbent teachers, and providing the appropriate support to improve competencies. The results can also be used to reward teachers who demonstrate high competencies or significant improvements: • Hiring of teachers: Only teachers who can demonstrate a threshold mastery or strong subject and pedagogical knowledge in the UKG should be hired. • Professional Development support of teachers: The UKG is already incorporated into the MOEC’s TPMS, and can be used to identify a teacher’s strengths, as well as areas for improvement. The areas for improvement can then be addressed through the teacher’s annual professional development and results plan. • Progression and promotion: The TPMS also uses the UKG results for teacher progression and promotion. The thresholds have not yet been established, but if the results are to encourage teachers to improve their knowledge then the thresholds must be meaningful and teachers must meet the competency requirements in order to be promoted. EFFECTIVE USE OF COMPETENCY TESTING REQUIRES POLITICAL WILL… If utilized properly, the UKG can serve as a tool to help teachers identify areas for improvement, set goals and targets and seek out ways to improve their competencies. But to encourage improvement in competencies the UKG must also set thresholds and ensure teachers reach those thresholds. This will require strong political will, but if this will can be mustered it can play an important role in strengthening teacher competencies and improving student learning outcomes in the long term. …AS WELL AS A RELIABLE AND EFFECTIVE INSTRUMENT If the UKG is to fulfill its role in the above-mentioned areas, the instrument (and the way it is administered) must be seen as a valid, reliable and credible measure of teacher competency. An invalid tool could lead to poor decisions, poor outcomes and a frustrated teaching workforce. The assessment items must be validated thoroughly and be based on subject and pedagogical knowledge that are truly relevant for teachers for each subject and school level. Investment in ensuring a valid, reliable and credible instrument will ensure all stakeholders accept it. CONTINUOUS PROFESSIONAL DEVELOPMENT (CPD) One of the key roles of the UKG and the performance appraisal is as a feedback mechanism. By identifying areas for improvement, teachers can then seek out professional development activities that will address these areas. This requires teachers to have awareness of, and access to, a wide range of materials that are relevant to the identified improvement areas. This involves the cataloging and mapping of materials available from teacher training centers, district/city governments, third-party vendors and online resources. Teachers should have clear support for CPD, Main Report xxix A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms including a way of receiving credit for their efforts. There needs to be support of multiple avenues for receiving CPD, from teacher working groups (KKG/MGMP) to online courses (for example, the MOEC’s mathematics education teacher training centers [P4TK], the Open University [Universitas Terbuka, or UT], and teaching training colleges [LPTKs]), and central and local government provision of CPD opportunities. A clear strategy and options will be critical for the TPMS to function optimally and truly improve teacher competency and performance. FROM RESEARCH TO ACTION: A CPD MODULE BASED ON STUDY FINDINGS The insights gained through this study highlight the importance of strengthening both teacher-subject and pedagogical knowledge, and also where teaching practices should be enhanced. The study is also particularly relevant in the context of the practices now being promoted in the 2013 curriculum reform. A natural extension of the study was to incorporate the lessons learned into modules that teachers could use in their CPD activities. The over 600 hours of video footage captured natural classroom time, providing useful real-life examples of what teachers do in their classrooms. This unique and invaluable data provided a perfect platform upon which to develop teacher professional development activities. AN INNOVATIVE PILOT AS ONE POTENTIAL IN-SERVICE APPROACH For teachers already in the system, a consistent, periodic support approach is required in order to build the necessary skills required for the new curriculum. This support can come from a variety of avenues, including traditional training, and online training. One of the most relevant methods for Indonesia’s large number of teachers spread out across many islands is through teacher working groups (KKG-MGMP). A teacher CPD course was developed based on the findings of the Video Study. The course utilized innovative approaches to simultaneously strengthen both subject and pedagogical knowledge with a focused goal of changing teacher behavior. • It simultaneously builds subject and pedagogical knowledge. • It uses an approach of Experience, Learning, Picture, Symbol and Application (ELPSA) that links mathematics teaching practices to the new curriculum. • It uses video clips from the Video Study as artefacts for real-life classroom examples and tools for discussion. • It is delivered in a scaffolded manner, with activities involving a process of “in-on-in” where concepts and approaches are learned first in a teacher working-group session, then practiced in the classroom, and finally discussed and reflected upon with other teachers in the subsequent working-group session. • It is designed to change beliefs and mindset through discussion, practicing and reflection. • It uses innovative techniques like teachers videotaping their own classrooms. xxx Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Executive Summary The impact evaluation results indicate that the module did in fact change teacher beliefs about the teaching-learning process and strengthened teacher subject and pedagogical knowledge. More importantly, though, there are also indications of teachers changing their practices in the classroom. P4TK Matematika has converted the module into an online environment with an attempt to retain the interactive and discussion-oriented philosophy. This virtual KKG/MGMP approach has the possibility of reaching a large number of teachers across Indonesia. PRE-SERVICE TRAINING MUST ALSO BE IN LINE WITH THE NEW APPROACHES The coursework for students at teacher training colleges (LPTKs) must be designed to align with the 2013 curriculum and ensure upon graduation they are prepared to utilize the new teaching and learning approaches. Of course this has to be supplemented by a strong, solid induction program at their respective schools. OVERALL KEY MESSAGES 1. Many of the practices encouraged in the past decade and in particular the 2013 curriculum have a positive relationship with student learning outcomes, indicating that proper implementation of the new approach should improve student learning. 2. BUT the practices encouraged in the 2013 curriculum are rarely used, indicating that it will require a major effort to shift in terms of methods and practices. Teachers will need strong support to understand and properly utilize the practices encouraged in the new curriculum. 3. The strong, positive relationship between teacher subject and pedagogical knowledge, and student learning outcomes, indicates it is a critical element in teacher effectiveness. But the national teacher competency test indicates teacher subject and pedagogical skills are generally very low. Support to in-service teachers to improve their knowledge and targeted hiring with teacher competency as a criteria would improve the quality of the teacher workforce. 4. Teacher-student interaction is an important element of student learning. Students in classrooms with greater teacher-student interaction have better learning outcomes. This holds true in whole-class, group work and individual seatwork situations. Teacher-centered practices such as teacher-only time lecturing tend to have a negative relationship with student learning outcomes. 5. The trends in teaching practices from 2007 to 2011 indicate decreases in many practices that have a positive relationship with student learning outcomes. The National Exam and curriculum have increased in influence according to teachers and appear to be playing a role in some of the trends seen. 6. Teachers require support if they are to change their practices. Results from a pilot geometry module implemented in KKG/MGMP indicate that continual support through “in-on-in” activities (see explanation above) with practical applications appear to be an effective way of helping to change teacher beliefs, improve teacher knowledge and change practices in the classroom. Main Report xxxi CHAPTER 1 BACKGROUND AND CONTEXT A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms Over recent years, Indonesia has achieved significant success in facilitating a rapid increase in the rate of enrolment in junior and now senior secondary education. However, despite this increased rate of enrolment, a large proportion of students in Indonesia still have low levels of literacy and cognitive skills. The performance of Indonesian students in international assessments indicates relatively low levels of student achievement, even when taking socio-economic factors into account. In the area of mathematics, Indonesia ranked 36th out of 48 participant countries in the 2007 Trends in International Mathematics and Science Study (TIMSS), with its score of 397 being more than one standard deviation below the international average (Mullis, Martin and Foy, 2008); In 2011 Indonesia’s score declined by 11 points (Mullis, Martin and Foy, 2012) and ranked 38 out of 42 participant countries. In the 2006 Programme for International Student Assessment (PISA), Indonesia had a score of 391 and ranked 51st of the 57 participant countries in mathematics (OECD, 2007); in the 2009 PISA Indonesia’s score decreased 20 points and ranked 68 of 74 countries (Fleischman et al., 2010). In 2012 Indonesia’s score increased 4 points, but ranked 64 of 65 countries. Figure 1: Estimated Impact of High vs. Low Performing Teachers on Student Achievement 100th percentile Student performance on Standardized Exam 90th percentile 53 percentile point di erence 50th percentile 37th percentile 0th percentile Age 8 Age 11 Source: Sanders and Rivers (1996) on the Tennessee Value-Added Assessment System (TVAAS) International evidence shows that students who are taught by effective teachers dramatically outperform those with ineffective teachers. There is a significant body of evidence to show that the quality of teachers plays a key role in determining student learning outcomes (e.g. Hattie, 2003; Bruns and Luque, 2014; Hanusheck and Rivkin 2010). How much difference does a good teacher make compared to a bad teacher in terms of facilitating student learning outcomes? In a seminal study, Sanders and Rivers (1996) conducted research into the Tennessee Value-Added Assessment System (TVAAS) to attempt to determine an answer to this question. The study found that if eight-year old students with average levels of scholastic performance are taught by teachers of varying qualities, their levels of achievement diverge dramatically. Specifically, one group of students was taught by high ability teachers (in the top 20%), while the other group was taught by low ability teachers (in the bottom 20%) over a 3-year period. At the end of the three years, the levels of performance of the two respective groups had diverged by more than 53 2 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Background and Context percentile points. Thus, by age 11, on average, students taught by teachers in the upper group would be scoring in the 93rd percentile, while those taught by teachers in the lower group would be scoring in the 37th percentile. The study also found that students with lower levels of achievement benefit most significantly from having high ability teachers. More recently, Hanusheck and Rivken (2010) found similar impact, where students with a weak teacher may master 50 percent or less of the curriculum for that grade while students with a good teacher get an average gain of one year and students with great teachers advance 1.5 grade levels or more. Cross-country comparative studies have highlighted deficiencies in the level of teacher quality and teacher support in Indonesia. For example, in a study conducted by Martin and Mullis (2006) to compare the quality of teachers in Indonesia, Malaysia and Singapore in the area of mathematics in TIMSS 2003, it was found that Indonesian students actually received a greater number of hours of instruction in mathematics than their counterparts. However, their teachers had a lower average level of formal education; were less likely to have a degree in mathematics; and received less professional support to improve content knowledge and teaching skills. Cultural context plays a critical role in determining what is effective for student learning outcomes. Teaching practices that are effective in one country may not lead to the same outcomes in another country. In the TIMSS 1999 Video Study (Hiebert et al, 2003), it was found that the nature of instructional practices in different countries varied significantly, with no direct correlation between the method adopted and student learning outcomes. For example, Japan and Hong Kong SAR both achieve high rankings in the TIMSS exam. While they both achieve high rankings, the teaching practices and classroom environment in the respective countries vary significantly. This highlights the need to understand the cultural context in order to determine the level of teaching effectiveness. Thus, in the case of Indonesia, it is important to understand what teaching practices are effective within the Indonesian context. 1.1 TEACHER REFORM IN INDONESIA In recent years, Indonesia has implemented major reforms to improve the quality of its teachers. In December of 2005 a major law on teachers was passed, which has become known as the Teacher Law. A cornerstone of this reform process is a policy requiring all teachers to have a 4-year degree and to undergo a process of certification by 2015. This has had major implications for in-service teachers, with 95% of all primary teachers, 45% of all junior secondary teachers, and 29% of all senior secondary teachers holding a lower level of qualification than that prescribed at the time the policy was promulgated (Ragatz and Chen, 2009). Teachers receive a bonus that effectively doubles their salary after passing through the certification process. This has significant financial and budgetary implications. However, there has been a great deal of criticism of the certification program. These criticisms relate both to how the process attempts to measure teacher ability and to whether certification in any form will actually result in improvements to teacher quality or teacher performance. A central goal of the Teacher Law has been to upgrade the quality of teachers and their performance in the classroom. Of course, the ultimate measure of success for this initiative is improved student learning outcomes. As stated previously, Indonesia participates in a number of important comparative international studies, such as the IEA TIMSS and PIRLS and the OECD PISA studies. By determining the level of performance of Indonesian students compared to their counterparts elsewhere over the years, these studies provide a relatively objective of the change in outputs of the system. Main Report 3 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms However, while the ultimate measure of success is improved student outcomes, a more direct measure is the degree to which the initiative has improved levels of teacher competence in the classroom. Therefore, a study that measures these levels will provide an important indicator to determine the effect of the Teacher Law. The focus of Indonesia’s reform initiative has been on improving the level of teachers’ qualifications. However, there is increasing recognition that determining the means to improve the quality of teaching requires an understanding of what takes place in the classroom. Discussions on measures to improve teacher quality must be informed by empirical evidence of what constitutes effective teaching and learning in the Indonesian context. To date, very little such research has been conducted. As a result, important policy decisions are often being made on the basis of anecdotal evidence or studies conducted in other countries. 1.2 MATHEMATICS TEACHING IN INDONESIA The research on mathematics teachers and teaching practices in Indonesia is relatively sparse. However, the limited research that has been conducted provides some perspective on teaching practices. Somerset (1997) described the teaching-learning process as being almost always based on traditional, teacher-centered approaches. Hendayana et al. (2003) agree, stating that teachers tend to dominate mathematics lessons, rather than letting students learn. On the basis of their observation of lessons, they found that activities tended to involve students copying notes and listening to the teachers, with limited interaction between students and a large proportion of students appearing to be disengaged from the process. According to this research, the approach to teaching geometry is usually highly theoretical, with many abstract concepts and formulas being introduced without much attention being paid to aspects such as logic, reasoning, and understanding (Karnasih and Soeparno, 1999; Soedjadi, 2000). Fauzan et al. state that in general, students are passive throughout the lesson; ‘chalk and talk’ is the preferred teaching style; there is an emphasis on factual knowledge; questions tend to require only single word responses, often provided in chorus; there is a lack of learning questioning; only correct answers are accepted and acted upon; and whole-class activities often involve writing, with no hands-on work carried out (2002). On the other hand, in a recent OECD report, Indonesia was ranked as one of the highest countries in an index of innovation of education for mathematics and science. To determine each educational system’s top innovations in pedagogic and organizational practices, data from three international education datasets – Trends in International Mathematics and Science Study (TIMSS), Progress in International Reading Literacy Study (PIRLS), and the Programme on International Student Assessment (PISA) – were analyzed to identify the areas in which each education system has demonstrated emerging or changing organizational and pedagogic practices over a specific period. Major changes in pedagogic and organizational changes were considered innovations (regardless of direction).2 Indonesia’s top organizational innovations included: (i) more use of assessments; (ii) more use of assessment data in secondary education; (iii) more public posting of secondary achievement data; (iv) more parental service on secondary school committees; and (v) more teacher observations of secondary school science classrooms. 2 It is important to note that while the term “innovation” generally has a positive connotation, for this study it refers to any changes, but it could be argued that some of these changes may not necessarily be going in a positive direction. It is also important to note that these are changes as specified in teacher and student surveys, rather than observed changes. 4 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Background and Context Indonesia’s top pedagogical innovations included: (i) more use of textbooks as primary resources in secondary school science; (ii) more grouping by ability in secondary education; (iii) more relating of secondary school lessons to everyday life; (iv) more use of answer explanation in secondary mathematics; and (v) more observation and description in secondary school science lessons (OECD 2014). 1.3 VIDEO STUDY OF TIMSS CLASSROOMS IN INDONESIA An increasingly popular approach for the analysis of teacher behavior and instructional practices is through the use of video. In the TIMSS 1999 Video Study (Hiebert et al., 2003), it was found that instructional practices in classrooms across countries varied significantly, with each country having its own unique “lesson signature.” Even among high achieving countries, the lesson signature differed significantly, suggesting that there is no single correct method of teaching mathematics. Due to social, economic and cultural complexities, teaching practices that may be effective in one country will not necessarily be effective in another. It is important to understand how students learn within a country’s own unique contexts and to determine the most appropriate forms of teaching in those contexts. With financial support from the Royal Government of the Netherlands, the Ministry of National Education (MOEC) and the World Bank conducted a two-phase Video Study to examine and assess the practice of mathematics teachers in Indonesian classrooms. The study involved the production of video footage of 100 classes that participated in the 2007 TIMSS international assessment, and of another 104 classes that participated in the 2011 assessment. The design of the Video Study, including the method for coding videos, is based upon the methodology used by a TIMSS 1999 Video Study of seven countries (Hiebert et al., 2003). An initial report was produced on the basis of the 2007 phase of the study. This report examined teaching practices in Indonesia and compared them to practices in the seven countries that participated in the TIMSS 1999 Video Study (Hiebert et al., 2003). One of the most striking findings from the study was how few words both teachers and students uttered throughout the period of the class. Indonesia also stood out in terms the proportion of class time dedicated to specific activities. Amongst the comparator countries, on average, at least 96% of class time was dedicated to actually teaching mathematics. By contrast, in Indonesia, the proportion of class time spent on actually teaching mathematics was only 89%. A significantly higher proportion of the lesson time was spent on organizational work (8%) and on non-mathematics activities (3%) than in other countries. The proportion of time spent on teaching mathematics was also broken down into problem-solving and non-problem-solving time. In Indonesia, 76% of the time spent on teaching mathematics was devoted to problem-solving, while in the comparator countries, this proportion ranged from 81% to 96%. In Indonesia, teachers also set relatively few independent problems,3 with 3.3 problems on average per lesson. In the comparator countries, the number ranged from 3 to 13, although Indonesian students did tend to spend more time on average on each independent problem, with this time running to an average of 6.6 minutes compared to a range 3 to 5 minutes in the comparator countries (Leung and Ragatz, 2010). 3 Counts of words spoken focused on only full-class interaction (rather than group or individual work) and were standardized to indicate the number of words spoken over 50 minutes. Indonesian teachers spoke fewer than half the number of words as in other countries, with only 2,633 words in an average lesson compared to 5,198–5,902. Main Report 5 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms Following the 1999 TIMSS study (Hiebert et al., 2003), the study identified a “lesson signature” on the basis of the pattern of practices typically used in the classroom. Typically, classes were segmented into three stages, these being: (i) the introduction stage; (ii) the development stage; and (iii) the closing stage. The introduction stage typically involved the review of homework from the previous class. This was followed by the development stage, which introduced new content. Teachers typically began this stage by motivating students with an exhortation on the importance of the material in the lesson, followed by questions related to the prerequisite knowledge required to engage with the new material. During the closing stage, teachers summarized the day’s lesson, sometimes with a certain level of student involvement, and assigned tasks for home work. There was a great deal of consistency, with nearly every observed lesson following this pattern (Leung and Ragatz, 2010). The coding structure used in the study enabled the identification of an initial profile of what takes place in the classroom in terms of the time spent on different activities, approaches to problems, and teaching methods used. However, it must be acknowledged that there are some limitations to this approach. The indicators are low inference. As noted by Watson (2008) regarding the TIMSS Video Study approach used to identify mathematical coherence, rationale and complexity: “…these methods did not give access to what was coherent or how it was achieved” (p. 129). She adds that: “There are studies that probe ‘typicality’ in mathematics classrooms, and studies that show how certain modes of teaching make a difference to students’ learning, but nothing that looks at how mathematical ideas unfold in successful classrooms of all kinds.” (p. 130). This point highlights the need to probe beyond the TIMSS Video Study codes in order to uncover the deeper influences of teaching practices on levels of student attainment. Another limitation is that the video analysis provided insights into what takes place in the classroom, without explaining why it takes place. Understanding the underlying motivations of teachers would provide a critical foundation upon which to gain a deeper understanding of what takes place in the classroom. The 2011 phase of the study was modified to enable a deeper exploration into the motives of teachers for their teaching practices. Within educational research, two areas have been identified as being particularly important in understanding the motivations and influences behind teaching practices in mathematics education. These are teacher mathematical beliefs and mathematical knowledge in teaching. 6 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning CHAPTER 2 OVERVIEW OF DESIGN, METHODS AND ANALYTICAL APPROACH A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms 2.1 INTRODUCTION This study was intended to examine the mathematics teaching-learning process in the Indonesian context and to determine why it takes place, how it has changed over time, and how it relates to student learning outcomes. The following section on the methodology of the research first defines its conceptual framework and specifies its objectives. It then elaborates on the research design, first by explaining key general methodological issues, and then by explaining the purpose, design and implementation of the instruments. The study data has three levels, with a large sample of more than 200 teachers (100 in 2007 and 105 in 2011) and a smaller sample of 10 case study teachers. In addition there is an in-depth examination of three of the case studies. As will be explained in this chapter, these three levels are referred to as the Macro, Mezzo and Micro levels. The unique iterative, two-phase design of the Video Study has allowed it to evolve in ways that allowed for modifications to its approach and objectives on the basis of its initial findings. 2.2 OBJECTIVES The purpose of the Video Study is to monitor the level of success of the initiatives implemented through the Teacher Law and the teacher reform effort. The data collected is intended to enable stakeholders to gain a better understanding of teaching practices in the Indonesian classroom, to see how these practices have changed over time, and how this change relates to changes in policy. In addition, it is also intended to generate insights into how the changes impact levels of student achievement. A secondary purpose of the project is to build the capacities of relevant personnel in Indonesia. BERMUTU is a World Bank project intended to support the Indonesian Government’s initiatives under the Teacher Law. As such, the expectation is that the Video Study should be conducted mainly by personnel in Indonesian Government agencies, with the World Bank primarily playing an advisory and supportive role. This will not only enhance ownership of the project by local officials and educators, it will hopefully also enable them to build their capacities to use videos as a research tool. A third purpose of the study is to document teaching practices over time and to integrate the use of video into teacher professional development activities. As a by-product of the Video Study, the collected video data may be used for studying aspects of Indonesian classrooms other than those on which this Video Study focuses. The classroom videos may also be used to produce an archive for future use, for the development of teacher training materials and other purposes. It has also been suggested that the Video Study could be expanded for use in teacher professional development activities. The BERMUTU project supports teacher working groups, which consist of teachers from 6-10 neighboring schools, with members of these groups meeting regularly to conduct professional development activities. These teacher working groups could provide an ideal environment for the use of video as a means for self-assessment and for improving teacher quality. A fourth purpose of the study is to go beyond what takes place in the classroom to address the issue of why it takes place. The analysis from the initial phase focused mainly on what takes place. However, it also raised new questions related to why certain practices are used. An understanding of why certain practices are used in the classroom is critical for the development of programs and policies intended to facilitate changes in teacher behavior 8 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Overview of Design, Methods and Analytical Approach and to improve teacher quality. A key additional component of the study in the 2011 phase is the incorporation of components to explore teacher mathematical beliefs and teacher mathematical knowledge in teaching (MKiT). In addition, the second phase included in-depth qualitative components, including the in-depth case studies. To summarize, the objectives of the study are as follows: 1. To characterize classroom teaching-learning behavior in the context of the stated intention of the curriculum and with reference to classroom characteristics in other countries; 2. To compare baseline data collected in 2007 with data collected in 2011 to identify emerging trends in teaching practices; 3. To understand why teachers use specific practices, particularly in relation to key influencing factors such a teacher’s individual subject and pedagogical knowledge and teacher mathematical beliefs on teaching and learning, as well as outside influencing cultural and system factors; 4. To understand the relationship between classroom teaching-learning behavior and student learning outcomes and to determine which teaching methods are more effective; 5. To produce an archive of classroom videos for the purposes of research and to facilitate teacher development; 6. To develop the capacities of relevant personnel. 2.3 RESEARCH QUESTIONS The overarching aim of the study was to establish a foundational understanding of mathematics teaching in the Indonesian context. On the basis of this foundation, the study aimed to explore how teaching practices relate to student learning outcomes. The study was intended to determine not only what takes place, but also why it takes place. While the teaching-learning process is the focal point of the study, the analysis also attempts to determine what shapes teachers’ strategies and practices. Characteristics of the teaching-learning process 1. How well are teachers prepared for their teaching? 2. What mathematics contents are covered in the lessons? 3. How competent are teachers in teaching mathematics? 4. How are the lessons structured and how is time managed during the lessons? 5. What types of mathematics problems do students solve? 6. How are mathematics problems solved? 7. What teaching strategies do the teachers use? 8. What are types of questions do teachers ask? 9. How do teachers assess students’ learning? 10. Are teachers motivated to improve their teaching skills? 11. What learning resources are used for supporting teaching and learning? 12. What are the profiles of the teachers? 13. What are students’ attitudes toward mathematics? 14. Which of the characteristics above have positive or negative relationships with high student achievement? Main Report 9 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms Characteristics of the Beliefs-Knowledge-Practices relationship 1. What is the relationship between teaching practices, teacher mathematical beliefs, and mathematical knowledge in teaching? 2. How does the teaching-learning process ensue within Indonesian eighth grade mathematics classrooms in terms of teaching practices and student responses? 3. What are the mathematical beliefs and orientations of Indonesia’s eighth grade mathematics teachers? 4. What is the level and variation of Mathematical Knowledge in Teaching (MKiT) of Indonesia’s eight grade mathematics teachers? 2.4 WHY A VIDEO STUDY? In recent years, the use of videos has been shown to be a highly effective means of studying classroom activities and the classroom practices of teachers (Hiebert et al, 2003). In parallel with the implementation of the BERMUTU project, the World Bank and the MOEC decided to conduct a Video Study4 to gain a better understanding of the level of competence of teachers in the classroom. By deliberate design, the Video Study is linked with Indonesia’s participation in TIMSS 2007 and 2011. Indonesia’s participation in these two rounds of the TIMSS provided a golden opportunity to monitor the effect of the Teacher Law by exploring the relationship between teachers’ classroom performance and their learning outcomes, as measured by the results of the TIMSS process in 2007 and 2011. In this Video Study, to determine the effectiveness of the initiatives implemented through the Teacher Law, the subject of mathematics at the eighth grade level was chosen as an example of teaching practices more generally. The process of teaching mathematics in a selected sub-sample of classes that participated in 2007 was studied by extensive videotaping of lessons conducted in the classroom. The process was repeated in 2011, with the results of the study being linked students level of achievement as measured by TIMSS in both cases. Since the study attempted to track changes over time, exact replication was an essential feature, with the core activities conducted in 2007 being reiterated exactly in 2011. 2.5 CAPTURING TRENDS OVER TIME A central aim of the study was to analyze changes in teaching practice over time and to determine the effect of these changes on levels of student achievement. As stated previously, the Teacher Law establishes minimum academic and professional standards for teachers. The assumption is that teachers with a higher level of qualifications and who participate in various professional development activities mandated by the Teacher Law will provide a higher quality of teaching, which will lead to improved student learning outcomes. Accordingly, this Video Study is based on the conceptual framework represented in the diagram below: 4 Arguments for the use of videos in studying classroom teaching over the use of other means such as questionnaires and live observation in studying classroom teaching were presented in the report of the Pilot of this Video Study by Asrijanty et al (August 2006) and are not repeated here. 10 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Overview of Design, Methods and Analytical Approach Figure 2: Concept of Measuring Trends in Teaching Over Time 2007 2011 Teacher Law: More quali ed sets minimum requirements teaching force for teachers Video Study Video Study (Phase 1) (Phase 2) Teachers’ Classroom Improved Teachers’ Instructional Practices Classroom (base-line) Instructional Practices Student Achievement Improved Student (base-line: TIMSS 2007) Achievement (TIMSS 2011) As this diagram makes clear, the Video Study was conducted in the context of the implementation of the Teacher Law to determine its level of effectiveness. Thus, phase one of the study documents and assesses teachers’ classroom instruction practices in 2007. This was the first time that a representative documentation of teaching in the Indonesian classroom had been conducted. A comparison of the output of this study with that of similar studies in other countries will facilitate an understanding of instructional practices in the current Indonesian classroom and enable a determination of how they differ from practices elsewhere. More importantly, the output of this study provides base-line data to determine the extent to which practices had changed by 2011, when the second phase of the study was conducted. By 2011, it could be assumed that the measures mandated by the Teacher Law will have impacted a portion of the teachers in the sample. A comparison of the teaching practices of these teachers with those of: (i) similar teachers in 2007; and (ii) teachers in 2011 who have not yet met the new criteria established by the Teacher Law, will provide a very good indication of the effects of the Teacher Law. The framework is based upon the hypothesis that both in 2007 and 2011, there would be a relationship between teachers’ classroom practices and student learning outcomes, as measured by the TIMSS scores. For the 2007 phase of the study, this will provide valuable input regarding the appropriateness of the curriculum and the kind of instructional practices necessary to enhance students’ level of achievement. A similar statistical analysis in 2011 would provide a means to determine the effectiveness of the changes mandated by the Teacher Law. Main Report 11 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms 2.6 CONCEPTUAL FRAMEWORK A conceptual framework for analysis was developed to examine what takes place in Indonesia’s mathematics classrooms, why it takes place and how it relates to student learning. This framework was inspired primarily by the work of Askew et al (1997), although it also incorporates many of the concepts discussed in the literature review included in the technical report. While the framework is teacher-centric, this does not imply that a student-centered framework, which has been used by a number of researchers, including Clarke et al. (2006) is not relevant. However, for scoping purposes, this research focuses specifically on teachers and teacher practices. Figure 3: Framework for Analysis of Teacher Practices, Beliefs and Knowledge Classroom Environment In uences Lesson Segment Purpose System Cultural Teacher’s and Policy In uences Practices In uences Mathematical Mathematical Beliefs Knowledge in and Orientation Teaching (MKiT) Knowledge of How to teach How students Subject learn to do knowledge teaching mathematics approaches mathematics What it is a Knowledge mathematics- of capable students student Sources: The model is an adaptation of various models on teaching practices, teachers’ beliefs and knowledge, mainly from Askew et al. (1997), but also including: Aubrey (1993); Bennett, Summers and Askew (1994); Lerman (1990); Thompson (1984), Lindmeier and Ufer (2010) and Schoenfeld (1989, 2010). Contextual factors are influenced by multiple sources, including: Skott (2001, 2009), Beswick (2005, 2007), Andrews (2007b, 2010), Sullivan and Mousley (2001), and Correa et al. (2008). The entities within the above framework can be separated into two distinct types: (i) the triad formed through the connection of beliefs, knowledge and practices;5 and (ii) the contextual influencing factors. In the above diagram, the triad is depicted in the foreground. This triad is the direct focal point for analysis, while the contextual influencing factors can be seen in the background. These contextual influencing factors play an indirect role in how the beliefs, knowledge and practices are formed and ultimately enacted in the classroom. Each of the entities within the framework is discussed below. 5 The terms “beliefs”, “knowledge” and “practices” are shortened terms used throughout this text for “mathematical beliefs”, “mathematical knowledge in teaching” and “teaching practices” respectively. 12 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Overview of Design, Methods and Analytical Approach The framework revolves around Teachers’ practices. According to this framework, teachers’ practices drive what takes place in the classroom and how student learning takes place. But this begs the question: How do teachers determine what practices to use? The hypothesis is that two key background dimensions influence teaching practices, these being: (i) teachers’ mathematical beliefs and orientation; and (ii) mathematical knowledge in teaching. An important aspect of the research design is its recognition that beliefs, knowledge and practices do not exist in a vacuum, but rather develop and evolve due to contextual influencing factors. The factors seen as being most influential are represented in the framework and range from very event-specific to much more general and longer-term. Teaching practices in particular are not static, but are instead dynamically shaped by the situation of the moment. The teacher may plan a lesson and have ideas of the teaching practices to employ, but each lesson contains instances where the teacher must evaluate the context and make teaching decisions in the moment (Schoenfeld, 1989; Thompson, 1985; Handel, 2003). The context of events is therefore considered to be of crucial importance in the analysis. The background bubbles of lesson segment purpose, classroom environment, cultural influences and system and policy influences are intended to illustrate their important role in shaping what ultimately takes place in the classroom. The four factors include: • Lesson Segment Purpose: A teacher’s practices are likely to be tailored to the lesson segment purpose. For example, one layer of the overall video study coding structure divides lessons into review of previous material, introduction of new material, practice and assessment. How might practices such as questioning, handling of misconceptions, use of mathematical problems and teaching strategies differ based on these segments? A teacher may choose to use a different type of questioning in each of these instances. • Classroom Environment: The classroom context is also likely to influence the practices a teacher chooses. A teacher’s approach is not static and is likely be adjusted to fit student needs. Factors like student background, ability level, willingness to participate and others are likely to be taken into account by teachers. A classroom in, for example, a rural, high poverty community presents a very different environment from an urban, wealthy community. Understanding how teachers see their environment and context may provide insights into why certain teaching practices are utilized. • System and Policy Influence: Policies may also directly or indirectly play a role in classroom activity. Indonesia has promoted certain policies relating to mathematics education and education in general. The goals and policy lingo used at times emerged in teacher interviews and at times surfaced in classroom activities. It is certainly not always clear whether their appearance is directly due to the policies themselves, so drawing a link between policy and practice must be approached with extreme caution. At the same time it is important to consider how the teachers internalize the policies in terms of interpretation and their willingness to follow them. Some of the more prominent recent policy goals in Indonesia relate to: (1) student-centred learning; (2) “active and joyful learning” which encourages participation and creation of an enjoyable, fun atmosphere; and (3) contextual teaching and learning (CTL), which encourages the use of real-life examples to help students see how mathematics applies to their everyday life. How might the teachers interpret the policies and how might the teacher choose to incorporate them into the classroom or not? Main Report 13 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms One aspect that is not explicitly stated in this framework but is found in other important frameworks such as that of Bruns and Luque (2014) is teacher motivation. In their education results chain and reduced form education production function, teacher skill and motivation along with teacher content knowledge drive teacher classroom practice. Vegas and Umansky (2005) identify with three broad classes of incentives to motivate teachers: (1) professional rewards; (2) accountability pressure; and (3) financial incentives. The concept of motivation is particularly relevant for this study in considering whether teacher certification and doubling of salary may have an influence on teaching practices. The motivation and influence of the teacher reforms are explored in detail within this report. • Cultural Influence: Indonesia is a large and culturally diverse country, so any attempt to use blanket descriptions of cultural traits would be dangerous. Still, there are certain cultural tendencies that may be taken into account when attempting to understand what takes place in the classroom. Aspects like authority and hierarchy, individual vs. societal balance, when and how it is appropriate to speak in public situations, and socio-economic factors may shape teaching practices. Also, culture is something that can vary from community to community. For example, an education expert stated that rural communities in Indonesia tend to hold what might be seen as more traditional role of a student within the classroom as quiet, disciplined and listening to the teacher (with one rural school having a sign with the phrase “Quiet students, happy teacher” possibly providing an insight into what is expected of students), while in urban environments many parents push the schools to develop communication, debate, and leadership skills for their children and there is an expectation from the principal, teachers, parents and students themselves that students should speak up in class and make persuasive arguments rather than following the teacher’s lead (Jalil, 2011). Again, blanket statements should be avoided, but these cultural aspects can help explain what takes place in the classroom. The positioning and layering of these bubbles within the framework are intentional. The layers are intended to illustrate how direct the influencing factor is, starting with most direct and moving to successively more and more indirect. It begins with the lesson segment purpose, followed in order by classroom environment, system and policy influences and finally cultural influences. Regarding the positioning, it is hypothesized that cultural influences, while touching all aspects of the model, tend to most influence teacher’s beliefs relatively more than other components, while system and policy influences, also touching all aspects, tend to play more of a role in teacher’s MKiT. While the influences are important to keep in mind, they will not be examined explicitly on their own, but rather incorporated into the examination of beliefs-knowledge-practices. The model recognizes that there are important factors that operate in the background to influence a teacher’s decision on what practices to use. These come from system and policy influences such as the curriculum. The model is based on the assumption that practices in classroom lessons are the major factor influencing student learning outcomes.6 6 It is possible that students also learn from influences outside the classroom, including tutoring, parental involvement and opportunities to use mathematics in everyday situations. Data related to these aspects were captured in the student questionnaire and controlled for when performing analysis. 14 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Overview of Design, Methods and Analytical Approach The study is about interactions and relationships, with the model highlighting the interactions between beliefs, knowledge and practices. In Figure 3, the relationships are illustrated through the arrows connecting the various components. It is hypothesized that the direction of the flow is not one way, but that there is typically a primary direction. As can be seen with the Teachers’ practices component, the arrows indicate that the relationship between this component and the other components are reciprocal. However, the thicker arrows indicate the hypothesized primary direction of flow. For example, much of the research on teachers’ beliefs tends to emphasize that a change in beliefs leads to a change in practices (e.g. Swann, 2007; Arvold and Albright, 1995). However, there have also been findings to indicate that the flow can go in the opposite direction, with a change in practices leading to a change in beliefs (e.g. Hoyles, 1992; Sullivan and Mousley, 2001). 2.6.1 INCORPORATION OF STUDENT LEARNING INTO THE ANALYSIS Beliefs, knowledge and practices become more relevant if they can be examined in the context of student learning outcomes. A measure of student learning outcomes was incorporated into the design of the research by using the results of mathematical assessments that were conducted for all students of the teachers participating in the study. The tests were conducted both at the beginning and the end of the eighth grade year. Student learning outcomes are often used in studies to determine the effectiveness of aspects of the teaching process. In the case of this study, these outcomes were used not only to gain a sense of the effectiveness of teaching processes, but primarily as a contextual factor. For example, the student entry achievement level, as determined by the pre-test, is useful for understanding the nature of the environment in which teachers are operating. A teacher of a class of students with a high average entry achievement level may use a very different approach than a teacher with students with a low average entry achievement level. It is also useful to understand how students progressed over the course of the year, as it provides an indicator of how the various factors studied might have played a role in student learning. However, this study does not have the goal of identifying a specific list of practices that teachers should use or a set of beliefs they should have. Full descriptions of each component in the model can be found in the Technical Report. 2.7 GENERAL METHODOLOGICAL ASPECTS 2.7.1 MIXED METHODS APPROACH While the first phase of the study had some qualitative elements, it mainly involved quantitative methods. In an effort for the second phase of the study to go beyond the coding of videos to gain a deeper understanding of specific practices and to determine why teachers use the practices, it became clear that a greater emphasis on qualitative methods was required. The study used a mixed methods approach involving both quantitative and qualitative elements. The aim was for these elements to complement one another and to create positive synergies. Denscombe (2013, page 270) notes that the mixed methods approach has emerged as a “third paradigm” for social research and that: “It has developed a platform of ideas and practices that are credible and distinctive and that mark the approach out as a viable alternative to quantitative and qualitative paradigms.” Johnson and Onwuegbuzie (2004, page 14) argue that: “A key feature of mixed methods research is its methodological pluralism or eclecticism, which frequently results in superior research (compared to monomethod research).” Main Report 15 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms For the second phase, the research was designed on the assumption that a mixed methods approach involving both qualitative and quantitative approaches would enable a better understanding of the teaching practices of eighth grade mathematics teachers in Indonesia. Thus, in addition to the full sample of 104 teachers (with a combined total of more than 3,500 students), case studies of eight teachers were conducted. A study based on the full sample can help to create a somewhat generalizable7 picture of teaching practices (with recognized limitations). However, practices that can feasibly be examined for such a large data set has many inherent limitations, including the need to use low inference data and the inability to generate critical insights into why teachers use the practices and how they make their decisions. The case studies of eight teachers allows for a more in-depth exploration of teacher thinking and an examination of the rationale for the decisions they make related to teaching practices. It also facilitates a more in-depth and higher-inference analysis of the practices. The case studies have greater value if what emerges from these studies can be related to the larger sample in order to understand the results in a broader context. 2.7.2 UNIT OF STUDY AND ANALYSIS The most applicable unit of analysis for the purposes of this study is the classroom. The objective is to characterize mathematics teaching practices in Indonesia’s classrooms and to identify which of the classroom teaching practices are related to high student achievement. The purpose is not to characterize or evaluate the performance of individual teachers. Thus, individual teachers are not the unit of analysis in this study. Of course, using individual teachers as the unit of analysis would have the advantage of enabling a more sensitive analysis. However, to do this, it would be necessary to videotape multiple lessons by the same teachers to obtain a reliable measure of the performance of these individual teachers. In addition, to study changes in performance over two points in time, from 2007 to 2011, it would be necessary to trace the teachers so that they could be videotaped for a second time, four years later. This would obviously be logistically impractical. As an alternative, this study utilized the same methodology as that adopted for the TIMSS 1999 Video Study, in which a representative sample of teachers was chosen, with one lesson per teacher being videotaped. Following this methodology, no conclusions can be drawn regarding the level of performance of individual teachers. Rather, the aim is to draw conclusions regarding the level of performance of teachers as a whole. This method also enables a comparison between groups of teachers with particular characteristics, such as teachers with undergraduate qualifications compared to those without such qualifications, over the two points in time. 2.7.3 SAMPLING The Video Study used a random sub-sample of the schools selected for the TIMSS sample, which itself was selected on the basis of a rigorous sampling methodology. A representative sample is necessary in order to be able to generalize the findings to the population under study. There are at least two aspects of representativeness in a Video Study: (i) whether the teachers chosen are representative of teachers in the general population; and (ii) whether the lessons videotaped are representative of the teaching of the teachers concerned. 7 The sample is based on the TIMSS sample, which is intended to provide a generalizable view for Indonesia, but limitations are recognized and will be treated with care. 16 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Overview of Design, Methods and Analytical Approach Sample of Video Study teachers The selection of a random sub-sample of the TIMSS schools is intended to ensure that the selected teachers are representative of teachers in the population. Since the goal is to establish a relationship between the characteristics of classroom teaching with student learning outcomes, as measured by TIMSS, the sample for the Video Study should be linked to the sample of the eighth grade classrooms in the TIMSS study. To keep the scale of the study manageable while ensuring representativeness, 100 classrooms and their teachers were randomly selected from the TIMSS sample. Following the practice of the TIMSS 1999 Video Study, one lesson per classroom was videotaped for the purposes of coding and analysis. By randomly selecting a subset from this overall sample, the Video Study sample was intended to be as nationally representative as possible. The Indonesian TIMSS sample was itself selected in two stages. First, a random sample of 150 schools was drawn from the target population of schools in the country, using a PPS method.8 Two stratifications were used in the sample: type of school (public or private) and quality of school (based on the national test scores: high, average, and low). One eighth grade class from each selected school was randomly drawn for the purposes of the study, with all students in the chosen classes being asked to take the test and to complete the student questionnaire. Following the practice of TIMSS, the sub-sample for the Video Study was also drawn from the main sample set using a PPS method. This method was utilized at two points in time, both in 2007 and in 2011. In each case, 100 teachers were selected. In 2007, there were 150 classrooms in the sample set, while in 2011 there were 174. After the 100 schools were selected, the teachers of the eighth grade classes selected for the TIMSS were invited to participate in the Video Study. The sample of teachers chosen to participate in the Video Study was randomly selected from the 174 classrooms in 154 schools that participated in the 2011 TIMSS.9 The TIMSS sample followed a rigorous methodology established and was conducted by IEA to ensure national representativeness. By randomly selecting a subset from this overall sample, the Video Study sample was also intended to be as nationally representative as possible.10 8 PPS stands for “Probability Proportional to Size”. The sample is chosen in such a way that the probability of a school being chosen is proportional to the number of grade 8 students in the school. 9 There are 19 large schools that have two classrooms included in the sample. 10 One important note on the approach is that the study favored breadth over depth in its approach of including a large sample. Whereas some studies have a smaller sample but examine a large number of lessons, this study aimed to get a large number of teachers, but filmed only two lessons and coded one. This has methodological limitations that are discussed later. Additional depth was added through multiple filmings of ten case study teachers. Main Report 17 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms Sample of case study teachers The eight case study teachers were selected through a targeted approach, with an attempt to include teachers who were most likely to provide insights into which teaching practices might be related with positive student learning outcomes. A rich set of data was available for teachers that participated in the 2007 Video Study phase, including classroom, school, student, community and teacher background variables, as well as TIMSS and school- level national assessment results. This data allowed for the selection of a sample of teachers that the data indicated were effective teachers. This selection was made purely on the basis of objective data and results, with the data from the videos not being used. This use of this data was avoided in order to ensure that teachers were selected on the basis of their results, rather than on any preconceived notions of what teaching practices might be effective. The selection methodology was based on the actual versus predicted performance of each teacher’s students, after controlling for key variables such as community poverty level, parent education level, etc. The top teachers, whose students performed higher than predicted, were selected. It is important to note that it was relative rather than absolute performance that mattered. Thus, the final sample selection included teachers from poor, rural, under-resourced schools whose students did not score as well as those in other schools, but who performed better than could be predicted on the basis of their circumstances. In the selection of teachers, it was decided that five teachers would be selected from the 2007 sample based on their relative effectiveness. However, another five teachers were selected from the 2011 sample on the basis of variables of interest to MOEC and World Bank. Schools were chosen based on community poverty levels, urban- rural mix, student pre-test results and teacher assessment results. Initial analysis11 of the pre- and post-test results indicated that four of the teachers in the sample recorded significantly higher than average student learning outcomes. However, four teachers recorded only average or slightly above average levels of performance, while two teachers recorded lower than average levels of performance. This altered the analysis approach somewhat, as the case studies could not be utilized to the extent originally intended to examine common practices amongst the relatively more effective teachers. Still, the mix of high, average and low-performing teachers enabled many useful comparisons, as will be discussed in later sections of this report. 11 Only part of the teacher, student, school, and community background information was available for the initial analysis. This data is important for more concise and accurate comparisons so student results are compared to others in similar contexts. This will be done, but the initial results do indicate a mixed group of teachers and this is unlikely to change. 18 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Overview of Design, Methods and Analytical Approach 2.8 INSTRUMENTS The study was highly complex, with multiple components and a range of different types of data. The following section provides a brief overview of the multiple instruments used in the research. Table 1: Research Data, Sources and Sample Grouping Full Case Studies Data 2007 2011 Source Type Sample Only LESSON OBSERVATION Video of classroom sessions X X Study team Videotaping X Transcripts of videotaped lessons X X Study team Transcripts X Observation data sheets X X Study team Lesson data X Coding of activities/practices following 1999 TIMSS X X Study team Coded data X coding scheme Video of case study teachers X Study team Videotaping X Stimulated recall of case study X Study team Recorded X Interview TEACHER MATHEMATICAL KNOWLEDGE IN TEACHING Subject knowledge competency assessment X Puspendik Assessment X Pedagogic knowledge competency assessment X Puspendik Assessment X Mathematical Scenarios X Study team Recorded X Interview TEACHER MATHEMATICAL BELIEFS AND ORIENTATION Beliefs survey X Study team Questionnaire X Beliefs interviews X Study team Recorded X Interview BACKGROUND SURVEYS School survey X X IEA**, Study Questionnaire X team Teacher survey X X IEA**, Study Questionnaire X team Student survey X X IEA**, Study Questionnaire X team STUDENT LEARNING TIMSS Mathematics test X X IEA Assessment X Student pre- and post-tests X Puspendik Assessment X ** The original instruments from IEA/TIMSS were modified with additional questions specifically for the Video Study. These surveys were given separately from the IEA visit that took place in May 2011. For the final analysis, data was available from both the IEA and Video Study surveys. Main Report 19 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms The figure below maps the instruments from a participant-specific point of view. All instruments are at least indirectly connected, but in order to ensure clarity, only the primary links are displayed. The diagram highlights the fact that for every set of teacher data at the full-study level of 104 teachers, there is related data at the case-study level. Each of these instruments is elaborated in this section, providing details on the purpose, design, the analytical approach and links to other data, together with milestones and issues related to the design and implementation of the study. Figure 4: Mapping of Instruments by Full Sample vs. Case Study Sample Student Assessment (pre- and post-tests) and Students All Teachers Mathematical Beliefs Teaching Practices MKiT Context/Background Teacher Subject Teaching Practices Teacher, student Beliefs Survey and Pedagogy (video analysis) and school surveys Assessment Studies Case Teacher General Beliefs Interview Stimulated Recall Teacher MKiT Background Interviews Scenarios Interview 2.9 CASE STUDIES The case studies were intended to go beyond the what of teaching practices to explore the why. The full-sample portion of the study provided extremely useful data for quantitative analysis and for painting a picture of teaching practices in Indonesia’s eighth grade mathematics classrooms. However, while the overall study could help answer what takes place, it could not answer why it takes place. In addition, the quantitative data from surveys could only provide simple and somewhat superficial information, without enabling much deep insight. The case study component was intended to address this gap by not only examining what takes place, but extending the analysis to examine why certain teaching practices are used, including motivations and factors that influence choices made by teachers in the teaching process. The case studies are by no means simply a supplement to the larger study. Rather, it is the interactions between the quantitative and qualitative aspects of the study that enable us to draw a more complete picture of mathematics teaching practices. The case studies involved week-long visits to allow for immersion into each individual teacher’s environment, with ample opportunity for interviews and observations. Although it is an exaggeration to consider the week-long field visits to the case study teachers as ethnographies, the concept captures the spirit of the case study approach. A foundational concept was that the visits should not simply be treated as a series of interviews to gather information, 20 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Overview of Design, Methods and Analytical Approach but rather an immersion in the teacher’s environment to the fullest extent possible over the week spent at the school. Thus, the extent of this immersion varied at each school, depending on the schedule and willingness to participate of the teacher and other members at the school, with some teachers fully dedicating themselves to the week-long visit, while others preferred to focus only on the specific activities related to the case study. In the cases where the teachers were less accessible, an attempt was made to interact with other teachers and the headmaster. Design Based on Conceptual Framework The case study design is based on the framework described in Section 2.6. The conceptual framework relies heavily on the case studies to provide details for all examined components. In particular, it relies on these case studies to provide insights into why teachers use their chosen teaching practices. The case study visits were planned with the purpose of obtaining a detailed account of the teacher’s perspectives and skills from a variety of angles, with the focus areas being: (i) teaching practices; (ii) teachers’ Mathematical Knowledge in Teaching (MKiT); and (iii) teachers’ mathematical beliefs. Based on these dimensions, the formal activities listed in Table 2 below were created for the case study visits. Table 2: Case Study Activities and Their Purpose Activity Description/Purpose General Interview with A relatively unstructured interview designed to obtain important background information on Headmaster the school, parents, community and the headmaster him/herself. This interview also served the purpose of making the headmaster comfortable with presence of the researchers. General Interview with Teacher This interview was intended to gather important background information on the teacher, but its primary goal was to get the teacher comfortable with the method of being recorded in an interview and to establish a cordial relationship with the teacher. Questions were related to the teacher’s experiences with mathematics from his/her time as a student up through becoming a teacher. Observation and video recording All lessons during the week that the teacher had for the given class involved in the study (e.g. of lessons class 8.C) were observed and videotaped with a single digital video recorder at the back of the classroom. Teachers typically had other eighth grade classes, but only the sampled class was videotaped. Stimulated Recall Interviews In these interviews, the teacher watched videos of his/her class and provided insights on what took place in up to three classes. The first interview was designed to be teacher-driven, with the researchers asking the teacher to identify any key moments or interesting insights. The researchers remained relatively quiet in an attempt to avoid influencing the teacher regarding what to focus on. The second and third interviews involved the researchers first analyzing the videos to identify events of interest and then developing specific questions for the teachers. Mathematical Scenarios These interviews involved the construction of five different classroom scenarios designed to elicit Interview responses to enable the teacher to demonstrate aspects of their MKiT. Mathematical Beliefs Interview These interviews consisted of two stages. The first stage involved a general discussion regarding mathematical beliefs, including where the teacher developed his/her beliefs about mathematics and how they may have evolved over time, as well as specific topics relating to (i) the origin and essence of mathematics; (ii) how students learn mathematics; and (iii) how to best teach mathematics. The second stage of the interview involved the teacher responding to descriptions of different mathematical orientations. Main Report 21 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms 2.9.1 LENGTH AND SCHEDULING OF VISIT The case studies involved visits of one week in each case. Various factors were taken into account when determining the length of the case study visits, with the primary consideration being how to ensure the core information desired could be gathered in the time allowed. However, other factors, including both fixed and variable cost, the burden on the teacher and school, and the value of time, were taken into consideration. It was determined that a shorter visit would be ineffective, because teachers tend to be less open and possibly even less honest during initial contact with researchers. The researchers needed to build a relationship with the teachers before putting them through challenging assessment-oriented tasks and discussing sensitive topics. Therefore, the visit was structured specifically to enable first building a relationship with the teacher and school, followed by increasingly difficult or sensitive tasks. Every attempt was made to ensure the visit was as unobtrusive as possible. It was recognized that a week-long visit with multiple recording sessions and interviews is a burden on both the teachers and their students. While disruption was inevitable, the scheduling of activities was extremely flexible and always built around the schedule and needs of the teacher. 2.10 LINKING COMPONENTS An overarching aim of this research was to explore the relationships between interactions, practices, mathematical beliefs and knowledge, as well as student learning outcomes. The following is a description of the analytical approach for establishing these connections and exploring the relationships between them. 2.10.1 OVERVIEW OF RELATIONSHIP MAPPING The multiple sources of data can be seen as providing slices of perspective. The diagram in Figure 5 below illustrates that data for both MKiT and mathematical beliefs was gathered at four different levels, with each of these levels providing a different perspective through which to inspect the component. This diagram is somewhat hierarchical in the sense that it begins with the most concrete possible situation (the actual practices used in lessons) and moves a step away to stimulated recall, in which the teacher gives a description or explanation of specific events within the concrete situation. While this is “actual” in the sense that it relates to actual situations experienced by the teacher, it is also a post-hoc construction by the teacher at the time of the interview. It is possible that the teacher may intentionally or unintentionally craft an explanation that does not fully reflect the understanding or thought process of the teacher at the time of the event. The next step away from the concrete comes through focused explorations. The MKiT scenario uses hypothetical situations, while the beliefs interview explores the mathematical beliefs in a more abstract, general sense. The survey and assessment do not necessarily represent a further step away in the hierarchy, but rather provide information to complement the interviews in a standardized fashion that can also provide a more generalized context, given that the full set of 100 teachers also participated in these activities. 22 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Overview of Design, Methods and Analytical Approach Figure 5: Data Shown as Levels of Perspective for MKiT and Mathematical Beliefs MKiT Beliefs Practices used Video Video Enacted/Actual in lessons Analysis Analysis Teacher explanation Stimulated Stimulated of the practices Recall Recall Evoked/Espoused Focused explorations Scenarios Beliefs (general/hypothetical) Interview Interview Standardized Teacher Beliefs measures Assessment Survey • Actual practices: This level examines what teachers actually do in the classroom in search of aspects related to MKiT and beliefs. From a mathematical beliefs perspective, this is often referred to as “enacted beliefs.” For MKiT, this represents the knowledge the teacher evokes and applies in the classroom for the given situation. There are many environmental and circumstantial factors that may influence the teacher’s actions. These may not be visible or understood by the observer. What actually takes place should be interpreted with caution, given the limitations that the researcher has in interpreting MKiT and beliefs of a teacher through observed practices. • Teacher explanation of practices: Through stimulated recall interviews, certain moments in the lesson are discussed with the teacher, to enable the teacher to provide his or her perspective on the event. The researchers typically did not ask questions directly related to the teacher’s beliefs or MKiT, but statements made by the teachers often gave indications related to their beliefs and, to a lesser extent, provided insights into their MKiT. • Focused explorations: The MKiT Scenario and Mathematical Beliefs interviews explored the concepts much more directly than the stimulated recall interviews. However, they did so with a certain degree of separation from the actual situations recorded in the videotaped lessons. Beliefs were discussed in general terms, although teachers would often contextualize their statements or relate them to situations they experienced. The Mathematical Scenarios interview presented hypothetical classroom situations to demonstrate aspects of MKiT through their identification of issues, their thought process and strategies they developed. This demonstration of MKiT may or may not capture what teachers would do in an actual classroom situation, where many other complexities and contextual considerations may be involved. • Standardized measures: The Teacher Assessment and beliefs survey can be seen as scaled and more easily cross-comparable aspects of MKiT and mathematical beliefs. Main Report 23 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms Each level provides a different perspective. It is through the triangulation (or in this case, “quad-angulation”) of the angles that a well-rounded picture of the teachers’ MKiT and beliefs are formed and the role they play in the classroom. The design of the conceptual framework is based on the theory that a teacher’s MKiT and beliefs are important influencing factors in their teaching practices. The Standardized measures and Focused explorations levels provide a picture through espoused statements and evoked knowledge; the video level shows actual practice; and the analysis is a constant back-and-forth between actual and espoused/invoked aspects in more hypothetical situations. The stimulated recall not only provides detail through elaborated explanations, but also enables the teacher to provide important insights into why there may be discrepancies between the actual and hypothetical/external. 2.10.2 ANALYTICAL STEPS FRAMEWORK The analysis can be seen as a four-stage process, with each subsequent stage building upon the results of the previous stage. This progressive analysis is depicted in Figure 6 below. Figure 6: Four Stages of Analysis: (1) Individual; (2) Component; (3) Cross-component; and (4) Full-component PRACTICES Video Teaching practices used in the lessons A Stimulated recall D How teacher describes events within the E lessons and explains decision/actions G MKiT MATHEMATICAL BELIEFS Scenarios Beliefs Interview Demonstrated knowledge of students, Espoused beliefs through general knowledge of practices and subject knowledge discussion and beliefs situations F B C Teacher Assessment Beliefs Survey Subject and pedagogical knowledge test scores Questions covering nature of mathematics, how students learn, how to teach and teacher’s own practices Stage 1: Individual Instrument Analysis The data collected for each instrument is first analyzed on its own, with the method of analysis being dependent on the type of data collected. The results are summarized with the aim of being to enable a comparison with the results of the other instrument within in the component. 24 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Overview of Design, Methods and Analytical Approach Stage 2: Component Analysis (A, B and C) The Points A, B and C indicate comparative analysis that takes place within components. Each component has two data sources and the cross-instrument analysis is mainly concerned with identifying consistencies and discrepancies between the two sources of data and with combining the different aspects provided in order to generate a more complete picture of Beliefs, Knowledge and Practices. Stage 3: Cross-Component Analysis (D, E and F) The next stage was to relate the three components to one another. The Points D, E and F indicate analysis across components. The questions that drive this relationship analysis relate to consistency, flow of influence and effectiveness, as follows: • Are there consistencies in what is espoused or demonstrated outside the classroom (hypothetical scenarios) and what is enacted or evoked in the classroom? • In the case of identified inconsistencies, what might explain those inconsistencies? –– Are there common factors that may explain why some teachers are more or less consistent than others? –– Are the inconsistencies isolated incidents or are they part of a larger pattern? –– To what extent could the inconsistencies be due to researcher interpretation? • How might the components be influencing one another and what is the dynamic of the influence? –– The clearest hypothesis, as demonstrated in previous studies, is that Beliefs and MKiT influence Practices, but is the flow primarily one-way or is it bi-directional and/or cyclical? • Might the inconsistencies in any way impact the effectiveness of teaching practices? Stage 4: Full-Component Analysis (G) This final stage, depicted at Point G, could be considered part of Stage 3, in that it is a form of cross-component analysis. However, it involves taking a summary view of the cross-component results to determine how all three components interact as a whole. 2.11 MACRO TO MEZZO TO MICRO ANALYSIS The original analytical approach anticipated using the full set of 104 teachers to generate a general, high-level perspective and case study teachers to generate a more in-depth perspective. The sample size of eight case study teachers was chosen to ensure the inclusion of a good range of situations to enable meaningful comparisons. It also included a buffer in case some case study teachers dropped out of the study. All ten teachers did end up making themselves available for the field visits. However, various complications arose in cases CS1 and CS8. In the case of CS1, the beliefs interview was not recorded due to a technical issue. In the case of CS8, the teacher was busy at the time of the visit and unable to allocate sufficient time for some of the interviews. While she had agreed to participate, she was clearly uncomfortable with the interviews and provided short and somewhat incomplete responses. It was decided that these case studies would be dropped from the analysis. Main Report 25 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms Initial analysis of the eight remaining case studies provided a rich range of data for the purposes of comparison. However, when attempting to formulate detailed descriptive accounts for each case study, it became clear that the large number of cases made the descriptions unwieldy and difficult to follow when moving from stage to stage of the analysis steps framework (see Section 2.10.2). In order to address this issue, a Macro-Mezzo-Micro approach was chosen. The eight cases provided a rich range of data and it certainly made sense to keep the analysis. However, in order to make the detailed descriptions more meaningful and to ensure that the additional in-depth analysis remained focused, a subset of three case study teachers was selected. The number of teachers and the general presentation approach used for each level is shown in Figure 7. Figure 7: Breakdown of Macro-Mezzo-Micro Approach to Analysis MACRO MEZZO MICRO Full set of 104 teachers 8 Case Study Teachers 3 Case Study Teachers Survey, coding and Initial analysis for all More in-depth assessment data only components descriptions No individual teacher Summary results Cases threaded through descriptions of non- presented each stage of analysis case study teachers Some speci c examples drawn upon The Micro cases were selected on the basis of the results of the initial Mezzo analysis. All the Micro cases included interesting material for analysis and important lessons could have been learned from each through deeper focused analysis, making it difficult to reduce the number. A primary criterion at the initial research design stage was to attempt to determine what makes for effective teaching in Indonesia. Thus, student learning outcomes, as determined by pre-test and post-test results, were a leading factor in the selection process. A clear choice was CS9, because this teacher’s students recorded the largest learning gains out of the full 104 sample of teachers. Therefore, to gain an understanding of what might make teachers effective, CS9 could provide important insights. She was also one of the highest scorers in the MKiT assessment, in both the subject and pedagogy sections. This case study had the potential to enable an investigation into the relationship between knowledge and teacher effectiveness. CS9 had similar characteristics to CS6 and CS4 in terms of the school environment, relatively high MKiT scores, and strong student learning outcomes. Thus, these cases were dropped from the Micro analysis in order to ensure a wide range of teacher types. CS3 had some interesting characteristics, in that his subject score was the highest of the full sample of teachers, but the pedagogy score was closer to average. In this case, the students’ learning outcomes were slightly below average, providing a contrast to CS9. Finally, CS7 had a relatively high pedagogy score, but a below average subject score. This teacher also taught in a poor rural area environment, but recorded above average student learning outcomes. It was decided that this case study had the potential to provide insights into effective teaching and learning in this context. 26 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Overview of Design, Methods and Analytical Approach As the Micro analysis progressed, it became clear that at times the other Mezzo teachers needed to be brought in for certain pieces of the analysis. A gap that emerged was that none of the three selected Micro teachers had a strong transmissionist beliefs orientation. As this became an important focus of the analysis, CS2 was often brought in to provide additional insights. 2.12 REGRESSION MODELLING APPROACH An overarching goal of the study was to understand how teaching practices, teacher mathematical beliefs and mathematical knowledge in teaching relate to one another and how they in turn relate to student learning outcomes. For the quantitative analysis, a regression modeling approach was used to analyze the data from the surveys, assessments and video coding produced for the full sample of 104 teachers. The analysis was conducted to examine patterns from the regressions. In order to ensure consistency throughout the study and to make it easier to interpret the information, the results of the regression are all presented in a standardized manner. The logic behind the results presentation and interpretation are provided below. 2.12.1 SELECTION OF CONTROL VARIABLES The results of regressions are presented in terms of relationships of multiple two-variable (e.g. variable x and variable y) combinations. For example, the regression would show the relationship between (x) teachers’ use of expositional teaching in relation to (y) teachers’ score on the MKiT assessment. However, when producing those results, many student, family, classroom, teacher, school and community variables are controlled for. In the selection of variables, a reference point had to be used. Thus, student learning outcomes, based on the post-test score of each student with their pre-test score being controlled for, were used as the reference point. The selection of variables was implemented by asking the question: “Which student, home, classroom, school, teacher background and community variables are related to student learning?” The first step was to determine basic correlations for each variable from the student, teacher and school surveys. For example, simple correlations indicated that student learning outcomes were related to the level of poverty of the community; the educational level of the student’s parents; whether the teacher was a civil servant; the frequency of bullying in school; and many other variables. In the case of all of these factors, there was a statistically significant correlation. These variables were then combined using an iterative process of stepwise regression analysis until a set was developed of the most important variables to control for when comparing practices, beliefs and knowledge variables. These variables are listed in Table 3 below. Main Report 27 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms Table 3: Variables Used in Various Layers for in the Regression Analysis Groupings Variables Student Student pre-test score Student age Hours of homework per week Extent to which homework done with peers Hours spent reading for enjoyment per week Hours of tutoring in mathematics outside the classroom per week Student’s perception of mathematics as relevant for future work Student home Whether the home has a computer Extent to which Indonesian is spoken at home (vs. a local dialect) Average education level of parents Classroom and School School size School type (public or private) Whether the school is a madrasah (religious school) Resource measure –materials Resource measure – calculator Level of involvement of the principal (as specified by the teacher) Level of involvement of visiting supervisor (as specified by the teacher) Frequency of students skipping class (as specified by the principal) Extent to which vandalism is a problem (as specified by the principal) Community Community poverty level (percent poor) Community size Teacher background Years of experience teaching Whether teacher is a civil servant Whether teacher is female Whether teacher is school-hired Whether teacher participates in a teacher working group Amount of training received in the last year Teacher quality Teacher degree type Teacher education level Whether teacher is certified Teacher MKiT assessment score (not included in all regressions) For the purposes of scope, these variables are treated as control variables only, with no discussion on how these variables change on the basis of the practices, beliefs and knowledge variables used. 2.12.2 PROGRESSIVE BUILDING-UP OF THE MODEL The regressions began with simple correlations. They then progressed with a small number of variables, which were progressively built up with more variables. At first, no control variables were used; then student, school and community variables were added; then teacher background variables were added; and finally teacher quality variables were added. Such analysis may be useful for probing how relationships might change based on aspects of the teacher. However, for the purposes of scope in this study, it was decided that the results from only the final, fully built model would be used. 28 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Overview of Design, Methods and Analytical Approach 2.12.3 PRESENTATION OF RESULTS IN RELATIONSHIP MATRICES The results from the regressions analysis provide an extensive amount of information, such as coefficient size. However, for the purposes of this analysis, a single result was the focus of attention, this being the statistical significance level of the relationship between the two variables of interest. In the case of these regressions, it is the t-statistic. It provides a simple answer to the question: “Is there a statistically significant relationship between variable x and variable y?” This could be applied for multiple combinations of variables for beliefs, knowledge and practices. The benefit of focusing on the statistical significance is that it allows for pattern analysis. However, it is limited in its interpretation, as this study recognises. It is certainly true that the coefficient, which is the constant (a) in the formula y = ax + b, is of particular importance because it represents the rate of change of one variable (y) as a function of changes in the other (x). A particularly small coefficient would indicate that even if the relationship is statistically significant, the change itself is minor. However, for the purposes of this analysis, what is of interest is the determination of whether relationships exist and detection of patterns over multiple combinations. No claims are made regarding terms of magnitudes or how much of a difference these relationships would make in terms of dimensions such as student learning outcomes. A simplified illustration of a matrix is provided in Figure 8 below. It shows two beliefs statement variables and three MKiT assessment score variables. The first statement: “Mathematics is a creative subject like arts / music” shows that there is a positive relationship at the 5% level (95% confidence) with the teacher MKiT score. This means both move in the same direction and that as teachers’ MKiT score increases, there is a tendency towards an increased level of agreement with the statement. When breaking town the MKiT score into subject and pedagogy sections, there is no relationship between subject score and the statement. However, there is a positive relationship at the 1% level, indicating a very strong relationship between the statement and the pedagogy score. On the other hand, the statement: “Telling the children the answer is an efficient way of facilitating mathematics learning” has no relationship with MKiT and subject scores. However, it has a negative relationship with the pedagogy score at the 1% level. This indiates that as the teacher pedagogy score increases, there is a tendency for the respondent to disagree with the statement. Figure 8: Example of Coding of Regression Results to Indicate Relationships Pedagogy Subject Beliefs Statements and Stated Use of Practices MKIT Mathematics is a creative subject like arts/music 2 Telling the children the answer is an e cient way of facilitating their mathematics learning -1 -1 POSITIVE relationship and statistically signi cant at 1% level (99% con dence level) POSITIVE relationship and statistically signi cant at 5% level (99% con dence level) POSITIVE relationship and statistically signi cant at 10% level (99% con dence level) NEGATIVE relationship and statistically signi cant at 10% level (99% con dence level) NEGATIVE relationship and statistically signi cant at 5% level (99% con dence level) NEGATIVE relationship and statistically signi cant at 1% level (99% con dence level) Main Report 29 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms The color coding for the cells is based on the t-statistic.12 While initially the numbers themselves were displayed in the cells, this resulted in a lack of clarity. Since the approach to the analysis is to identify relationships and patterns, it was determined that using color coding, without providing numbers, was more appropriate. An example of the regression that created the above result for the combination of the first statement and MKiT can be found in the technical report. 2.12.4 DATA MINING FOR PATTERNS The data mining concept posits that within large datasets made up of many variables, there often are hidden gems of data that cannot easily be uncovered using simple analysis or by approaching the data with specific hypotheses.13 Hand (2007) defines data mining as the discovery of interesting, unexpected or valuable structures in large datasets and notes two different aspects to this definition. One concerns large-scale, global structures, with the aim being to model the shapes, or features of the shapes, of distributions. The other concerns small-scale, local structures, with the aim being to detect these anomalies and to determine if they are real or chance occurrences. Data mining uses algorithms designed to exploit relationships in the data, searching for unexpected but important pieces of information. A classic example used to describe the method and benefits of data mining is one in which Walmart analyzed data of purchases to uncover an unexpected relationship between beer and diapers (Chan and Wong, 1991). This connection was surprising, but through additional analysis it was discovered that young fathers tended to be the ones making the beer and diaper purchase combination. The explaination goes that when young fathers are sent out on an errand to buy diapers, often on weekends, they would purchase a six-pack of beer to reward themselves for their heroic efforts. This information could then be used to place beer next to diapers in their stores, resulting in a dramatic sales increase.14 While important results may emerge by mining the data in search of relationships and patterns, there are also inherent deficiencies in this approach. The approach is weak if the analysis stops at the stage of identifying relationships and blindly draws conclusions on the basis of the correlation. Rather, data mining is an initial step that must be followed by posing conjectures and conducting additional analysis to understand why the relationship might exist. 12 Although the 10% level (90% confidence) is not considered sufficient to meet the threshold of being considered statistically significant, these instances are still identified in with very light colorings in order to identify those that are approaching but do not quite reach statistical significance. 13 As a preface, I previously worked as a consultant designing databases for corporations. In 1998 the concept data mining became was emerging as new analytical tools and methods allowed for more powerful and accurate analysis. It continues to be a common analytical method, particularly in large businesses and government organizations. 14 It should be noted that there are many variations on this story and there is also a question as to whether it is true or a legend. 30 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Overview of Design, Methods and Analytical Approach The analysis performed in this study uses the data mining concept to analyse thousands of variable combinations through regressions, with a common set of control variables to identify statistically significant relationships. These relationships are then analyzed for common patterns and groupings. For example, there may be many statistically significant positive relationships between combinations of beliefs-practices regarding beliefs related to the teacher’s role as a guide and practices related to teachers’ use of the investigative method. This combination might be expected and could be used as evidence to strengthen the hypothesis. However, there may also be cases where unexpected relationships emerge, either with combinations where there is no hypothesized relationship or, even more interesting, where the hypothesized relationship goes in the opposite direction to that expected. 2.12.5 CONFOUNDING FACTORS An important limitation to recognize in this regression approach is that of confounding factors. The regressions may produce relationships between the independent variable and a dependent variables that are estimated incorrectly because the estimates fail to account for confounding factors. The incorrect estimations could suffer from omitted- variable bias. As an example that will be discussed later, there is a strong relationship between teacher subject knowledge and student learning outcomes. While it is logical and likely that the teacher’s knowledge would play a role in making him or her be more effective, it may also be the case that teachers with higher knowledge tend to be more motivated, driving them to study harder in pre-service training and seeking out in-service opportunities and other information to build their skills. This motivation would also play a role in the effort they put into planning and executing their lessons. The higher learning outcomes could therefore be in part due this motivation, but it is being captured through the teacher’s higher knowledge scores. This limitation is recognized and attempts were made to not simply take the regression results as given. A great deal of effort was made to question the regression results and combine them with more qualitative analysis in order to provide a more complete picture. With the teacher knowledge example videos were examined to understand how knowledge is enacted in the classroom. Case studies were used to probe knowledge from various angles in order to probe connections between knowledge, beliefs, practices and learning. While the confounding factors cannot be eliminated, it was hoped that this process would get closer to understanding relationships. Main Report 31 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms 32 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning CHAPTER 3 TEACHING PRACTICES A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms This chapter presents a brief summary of the study’s findings on proportions, frequencies, trends, cross- country comparisons and relation to student learning outcomes. The practices are grouped as follows: (i) lesson structure; (ii) lesson content; and (iii) instructional practices. The Lesson Structure focuses on the length of the lessons; how much time is dedicated to mathematics, non-mathematics and mathematics organization activities; the purposes of various segments; and the type of interaction that takes place. The Lesson Content section focuses on the mathematics content of the lessons and assesses the complexity of problems and whether the problems involved applications or proofs. The Instructional Practices section focuses on how the mathematics problems were presented and addressed; the opportunities of teachers and students to talk; and resources used during the lesson. Complete analysis of each teaching practice, including full definitions; graphical representations of proportions of time dedicated to each practice; and comparisons with the seven other countries that participated in the 1999 TIMSS Video Study, can be found in the full Technical Report. This Main Report summarizes the key findings, with more in-depth discussion in relation to practice and policy themes in chapter 8. 3.1 PROPORTIONS, TRENDS AND CROSS-COUNTRY COMPARISONS Lesson Structure There was a large variation in the length of the mathematics classes videotaped in Indonesia. The average duration of classes in Indonesia ranged from 39 to 97 minutes, with an average class length of 70 minutes. This period of duration is much longer than the seven comparator countries. However, the actual total time dedicated to mathematics classes each week is estimated to be an average of 140 minutes, which is lower than the other countries, which tend to have more lessons each week. Indonesia’s classes tended to spend a relatively larger portion of class time on non-mathematical and mathematical organization activities. The proportion of time spent on mathematical activities was on average 89% in 2007, declining to 86% in 2011. By comparison, the average proportion of time spent on mathematical activities amongst the comparator countries ranged from 95% to 98%. Figure 9: Proportional Breakdown of Lesson Time Indonesia 2007 86 10 4 Indonesia 2011 89 8 3 Japan 98 11 Czech Republic 98 11 Switzerland 97 21 Hong Kong 97 21 United States 96 3 1 Netherlands 96 2 2 Australia 95 4 1 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Mathematics Mathematics Organization Non-Mathematics 34 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Teaching Practices In terms of lesson segment purpose, the most common use of mathematics time was to introduce new content. On average in 2011, 59% of time was dedicated to new content, 26% to practice, and 15% to review. The trend indicates that there was a large increase in time dedicated to new content and a corresponding decrease in practice. In comparison to other countries, the most striking difference is how much more time is dedicated to review, with Indonesia are spent relatively less time on review and more time in practice than did classes in the comparator countries. Figure 10: Duration for Different Activities in Indonesia and Other Countries Indonesia 2007 11 48 41 Indonesia 2011 15 59 26 United States 53 23 25 Switzerland 35 40 25 Netherlands 39 34 27 Japan 24 60 16 Hong Kong SAR 24 39 37 Czech Republic 58 22 20 Australia 39 33 28 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Review New Content Practice Instructional Practices By far the most common teaching strategy utilized in Indonesian classrooms was exposition. Indonesia’s lessons tend to be teacher-centered, with 61% of time on average being dedicated to exposition or lecturing; 22% to problem solving; 9% to discussion; and 7% to practical work. Only 1% of the time was dedicated to investigation. In terms of trends, the biggest change was the increase in exposition from 53% to 61%, while discussion, investigation and practical work all saw slight decreases. Figure 11: Time for Different Learning Activities in Mathematics Lessons 2007 13 53 3 10 21 2011 9 61 1 7 22 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Discussion Exposition Investigation Practical work Problem solving Note: The instructional practices layer was not part of the 1999 TIMSS video study, so no cross-country comparison can be made. Main Report 35 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms Classroom Interaction Indonesian classes fell in the middle of the range compared to other countries in terms of whole-class (public) versus group and seatwork (private) interaction, with 64% of class time involving whole class interaction and 36% involving private interaction. The proportion of time spent on whole class interaction increased in 2011 increased from the figure of 57% recorded in 2007. By contrast, teachers in Hong Kong SAR were at one extreme, with 80% of class time being spent on public interaction, while the Netherlands was at the opposite extreme, with only 44% of time being allocated for this purpose. Figure 12: Percentage of Time for Public Interaction and Private Interaction Indonesia 2007 59 41 Indonesia 2011 64 36 Hong Kong SAR 80 20 Czech Republic 79 22 United States 68 32 Japan 66 34 Switzerland 55 45 Australia 52 48 Netherlands 44 55 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Public (whole-class) interaction Private (group work and seatwork) interaction Of whole-class interaction time, in Indonesian classrooms, most of the time was spent on teacher-only interaction, with these interactions typically involving lecture type presentations. The teacher-only (lecture) was the most common form of public interaction, making up 74% of all public interaction in 2011. This was also a large increase from the proportion of time spent on this type of activity in 2007, when the proportion stood at 61%. Figure 13: Public (Whole Class) Interaction Breakdown: Teacher and Student Involvement 2007 61 21 18 2011 74 11 15 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Teacher only Teacher and student Student only On the other hand, teacher-student interaction made up a small proportion of whole-class time, with this proportion declining over the two points in time. In 2011, the proportion of time spent on this type of interaction was only 11%, down from the figure of 21% recorded in 2007. 36 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Teaching Practices The number of words spoken by both teachers and students in Indonesia’s classrooms was significantly less than in other countries. The average number of words spoken by the teacher in a lesson (standardized to 50 minutes) was 3,243. At the extremes, some teachers spoke only 813 words, while others spoke up to 5,687 words. By contrast, the average number of words spoken by students in a lesson was 509. Figure 14: Average Number of Words Spoken by Teacher and Students During the Lesson 7,000 6,000 5,798 5,902 5,360 5,452 5,536 5,148 5,000 4,000 3,243 3,000 2,000 1,016 1,018 1,000 766 824 810 509 640 0 Indonesia Netherlands Switzerland Czech Australia Hong Kong United Republic States Average Teacher Words Average Student Words Indonesian teachers rarely spoke in long sentences. Fifty-six percent of all sentences spoken by teachers consisted of 1–4 words, which was a much higher proportion than that recorded in other countries, which ranged from 18% to 29%. The proportion of sentences consisting of 24 words or more was very low in Indonesia, at 3%. In other countries, this proportion ranged from 25% to 41%. Indonesian Students also rarely spoke in long sentences. Seventy-nine percent of all sentences spoken by Indonesian students consisted of 1–4 words In Indonesia, which was significantly higher proportion than that recorded in other countries, which ranged from 66% to 77%. The number of sentences consisting of 5–9 words in Indonesia was 12%, significantly lower than in other countries, where the figure ranged from 23% to 34%. The proportion of sentences consisting of 9 words or more in Indonesia was 9%, compared to a range of from 4% to 9% in other countries. The Role of Mathematical Problems As in other countries, most of the lesson time in Indonesia was spent on solving mathematical problems, either by the teacher or by the students. But in Indonesian classrooms, a significantly larger proportion of time was spent on non-problem segments compared to other countries. Indonesia spent much more time on non- problem segments than in other countries, with 76% dedicated to problem work in 2007, with this figure declining to 64% in 2011. Of the seven comparator countries, the next lowest proportion was 82% in Japan, while the highest was 96% in the Netherlands. Main Report 37 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms Figure 15: Mathematical Time Divided into Problem vs. Non-Problem Segments (Percent of total) Indonesia 2007 76 24 Indonesia 2011 64 36 Japan 82 18 Hong Kong SAR 85 15 Czech Republic 85 15 Australia 85 15 Switzerland 87 13 United States 89 11 Netherlands 96 4 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Problem Non-problem In comparison with other countries, Indonesian classes addressed fewer independent problems, but spent more time on solving them. In Indonesia, most classes addressed only two independent problems per lesson, compared to an international average of 3.3 problems. The number of independent problems solved over the course of each videotaped lesson varied from one to 10. At one extreme, Japanese classes addressed an average of three independent problems per lesson, but spent 15 minutes on each problem. At the other extreme, classes in the Czech Republic addressed an average of 13 independent problems, but spent only four minutes per problem. Only in Japanese classrooms was more time spent per independent problem. On average, Indonesian classes addressed very few problems of high complexity. However, there were a larger proportion of problems of medium complexity than in other countries. The proportion of problems that were of low complexity ranged from 17% to 77%, with problems of average or medium complexity ranging from 22% to 45% of the total, while high complexity problems ranged between 6% and 39% of the total. The problems solved in Indonesian classrooms were generally of a lower level of complexity, with the average percentage of low level problems being 57% and of medium complexity being 40%. Only 3% of the problems were of a high level complexity. Indonesia had very few problems that involved applications. Applications are often presented using verbal descriptions, graphs or diagrams, rather than just mathematical symbols. Applications are important because they require students to make decisions about how and when to use procedures they may have already learned and practiced. In this sense, applications are, by definition, more conceptually demanding than routine exercises. The average proportion of problems which included applications in Indonesia was 16%, compared to a range of 34% to 74% in other countries. Relative to other countries, Indonesian teachers stated the concepts for problems to a greater extent, while they used procedures to a lesser extent. When comparing Indonesia to other countries, it can be seen that Indonesian teachers stated concepts much more frequently than did their counterparts in other countries, with a rate of 35% compared to a range of 5% to 24%. Indonesian teachers made relatively less use of procedures, with a rate of only 41%. They made connections relatively frequently, with a rate of 24%.15 15 Note: The number of problems differs significantly from the amount of time for each type. Make connections was only 3.6% of the total time while state concepts was 60% and use procedures was 35% 38 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Teaching Practices Indonesian teachers tended to use the problem context of real-life situations relatively less frequently than did teachers in other countries. In Indonesia, the average proportion of problems involving a real world context was 12%, while in other countries, the figure ranged from 9% to 42%. On the other hand, 88% of the problems in the Indonesian classroom involved mathematical language and symbols only, compared to a range of 40% to 89% in other countries. Tools and resources used The use of physical tools in Indonesia was noted in more than half the classrooms. Physical tools in the mathematics learning process include tools for measuring (e.g., rulers, protractors), compasses, visual aids tools, tiles, three-dimensional and two-dimensional geometry models, etc. Essentially, physical tools are tools that can be manipulated by teachers and students. Sixty three percent of the lessons in the sample used physical tools. The average number of problems which required the use of physical tools was 3.7. The use of real objects was relatively more prevalent in Indonesia than in other countries. Approximately 28% of the lessons in the sample used real objects compared to a range of 4% to 21% in other countries. The percentage of lessons in Indonesia in which a calculator was used was very small, occurring in only 13% of schools in the sample. This was in great contrast to the practice in other countries, where the rate of usage could be up to 91%. This situation in Indonesia was due to policies set by schools or teachers of not permitting the use of a calculator in the mathematics learning process, the intention being to prepare students for the national mathematics examination, during which students are not permitted to use calculators. 3.2 RELATIONSHIPS BETWEEN TEACHING PRACTICES AND LEARNING OUTCOMES Regression analysis allows for the detection of relationships between various teaching techniques and student math scores. For the 2011 phase, tests were given to students at the beginning and end of the 8th grade year to measure student learning outcomes for the year. Relationships cannot be interpreted to show cause and effect. Still, after controlling for key student, household, class, school, and demographic characteristics, statistically significant relationships emerge between teaching practice and student learning outcomes and provide useful initial insights into what may be relatively more effective teaching practices in Indonesia. It should be emphasized upfront that these relationships were seen as simply a first step in understanding how teaching practices relate to student learning; the relationships tell only part of the story. The effectiveness of practices is not simply the proportion of time a teacher spends using a specific practice, but also how the teacher uses it that will determine its effectiveness. The practices were also examined in a qualitative manner and considered in the context of the classroom environment and teacher characteristics. Main Report 39 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms The following results emerged: • A constant theme that emerges from the regression results is that classes with higher student-teacher involvement (e.g. student-teacher interaction in whole class settings and teacher monitoring and participation in group work and individual seatwork) had higher test scores. Traditional teacher lecturing, on the other hand, tended to have a negative relationship. • Teacher involvement in activities, either through monitoring or direct teacher-student interaction, tended to have a positive relationship compared to student-only interaction (students working alone during seat work and group work and whole class time where the teacher was not actively involved). • For teaching approaches, classes where investigative learning and practical activities were used more often tended to have higher learning outcomes whereas classes with more expositional teaching tended to be negative. • The proportion of time on task tended to be important, where classes spending a larger proportion of time on mathematics tended to have higher learning outcomes, while classes with a higher proportion on non- mathematics (off-task) tended to have lower learning outcomes. • Although it was rarely used, assessment time and assessment-related activities such as quizzes had a positive relationship to test score. • The percent of time spent on problem math (as opposed to non-problem math) had a positive relationship with student learning outcomes. • The process of setting up a problem with use of a procedure had a negative relationship with student achievement. Making a connection, on the other hand, is not as common, but it has a positive relationship with student learning outcomes. • Students in classes where a larger number of proofs were introduced tended to have higher student learning outcomes. • Classes where the set-up of problems used math language tended to have higher student learning outcomes than those that tended to use setup with real life context. • Classes where open-ended questioning rather than closed questioning tended to have higher learning outcomes, but the practice is rare. • Classes where non-routine problems were used tended to have higher learning outcomes, but the practice is rare. • Use of tools and resources could play a role, where the use of projectors tended to have a positive relationship with math scores, while use of textbooks tended to have a negative relationship. The results in this brief overview on proportions, trends and relationships to student learning outcomes will be explored in more detail through later sections and presented in a more thematic approach in section 8 where conclusions are drawn and policy implications are considered. 40 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning CHAPTER 4 MATHEMATICAL KNOWLEDGE IN TEACHING A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms Across multiple studies, the level of teachers’ knowledge has been found to be one of the most significant components in relation to student learning outcomes. Although there is no strong, consistent correlation between observable teacher characteristics such as age, degrees, certification status and years of experience, with effectiveness, teacher knowledge is consistently correlated to student learning outcomes. Bruns and Luque (2014) note that while relatively few studies have been able to examine the academic ability of teachers in service by testing them directly, the studies that have done so find positive correlations between teachers’ test scores and those of their students (Marshall and Sorto 2012; Hanushek and Rivkin 2006; Hill, Rowan, and Ball 2005; Eide, Goldhaber, and Brewer 2004). The significance of levels of teacher knowledge as a factor in determining student learning outcomes appears to be even stronger in developing countries. A review conducted by Bruns and Luque (2014) of the evidence from developing countries on how different educational inputs affect educational outcomes found that teachers’ knowledge of their subjects is one of the few variables consistently correlated with student learning outcomes. The review involved 20 different studies of the impact of teachers’ knowledge, as measured by test scores, on their students’ learning performance. All of these studies found a positive correlation, with 13 finding a significantly positive correlation. The researchers concluded that of all the education inputs that have been evaluated, including school infrastructure, learning materials, teacher and principal characteristics, and school organization, only three are consistently and significantly correlated with learning outcomes. These are (i) teacher knowledge; (ii) teacher absence rates; and (iii) the availability of desks. (Glewwe et al. 2011) The importance of the teacher’s level of knowledge in the teaching-learning process has long been recognized. Recent research in mathematics teaching has begun to uncover the complexity of this knowledge. Since the mid- 1980s, attempts have been made to investigate the concept of what, for the purposes of this research, will be referred to as Mathematical Knowledge in Teaching (MKiT). Various theoretical models have been developed to describe the domains of mathematical knowledge needed for teaching. Many studies have also explored the interactions between various knowledge domains, with some being topic-specific (e.g. geometry, algebra, etc.), while others focus on the classroom and cultural contexts within which knowledge may play different roles. With a few notable exceptions,16 research on teacher knowledge before the mid-1980s tended to focus on general aspects of teaching, with subject matter being seen as little more than context. Shulman noted a gap in the examination of content and its role in instruction, labeling this the “missing paradigm” in research on teaching and teacher knowledge (Shulman, 1986). Shulman is often credited with opening the doors to a new area of research on teacher knowledge through his proposed special knowledge domain that he and his colleagues termed Pedagogical Content Knowledge (PCK). Shulman noted that that there is content knowledge unique to teaching, a kind of professional knowledge that is specific to a particular subject matter. Shulman argued that: “The currently incomplete and trivial definitions of teaching held by the policy community comprise a far greater danger to good education than does a more serious attempt to formulate the knowledge base.” (Shulman, 1987, p. 20) added that: “Pedagogical content knowledge is the category most likely to distinguish the understanding of the content specialist from the pedagogue” (Shulman, 1987, p. 8). 16 For example, two notable early theories that began to distinguish different types of mathematical knowledge include Schwab (1978) and Skemp (1976). Schwab made a distinction between substantive and syntactic knowledge, arguing that teachers need to understand both the content of their subject discipline and how that content is organized (Schwab, 1978). Skemp made a distinction of a relational (knowing both what to do and why) vs. instrumental (rules without reasons) understanding of mathematical ideas (Skemp, 1976). 42 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Mathematical Knowledge in Teaching Ball, Thames and Phelps (2008) credit Shulman and his colleagues with another important contribution for representing content understanding as a special kind of technical knowledge that is critically important to the teaching profession. Shulman proposed that knowing a subject for teaching requires more than knowing its facts and concepts. It also requires an understanding of the organizing principles and structures and the rules for establishing what is legitimate to do and say in a field, stating that: “The teacher need not only understand that something is so; the teacher must further understand why it is so, on what grounds its warrant can be asserted, and under what circumstances our belief in its justification can be weakened or denied.” (Shulman, 1986, p. 9) Among the challenges related to the concept of PCK is whether lines can be drawn to distinguish content knowledge and pedagogical content knowledge and, if so, how. Goulding, Rowland and Barber (2002) pointed out that the boundary between knowledge of mathematics content and PCK is not clear-cut. Like Ball (1990) and Ma (1999), they emphasized that it is the way in which mathematics is known that is crucial (cf. Beswick, 2008). The MKiT assessment used for the current study consisted of both subject matter and pedagogy questions. The test itself is proprietary, so actual test items cannot be shown in this study. Askew et al. (1997) proposed different categories of knowledge: 1. Numeracy subject knowledge: This involves an understanding of mathematics and numeracy appropriate to what is being taught; 2. Knowledge of teaching approaches: This involves an understanding of teaching styles and different ways of presenting mathematical and numerical concepts to pupils, including a range of diagrammatic and verbal representations, so that they gain access to the subject knowledge; 3. Knowledge of students: This involves a knowledge of what particular students currently being taught understand as well as the level of knowledge of students more generally. For example, it involves a knowledge of the aspects of the numeracy curriculum that are generally found difficult, common misconceptions and models of progression. This study attempts to address the multi-faceted nature of MKiT by viewing it from various perspectives in order to create a well-rounded picture of each teacher’s MKiT. All teachers in the MOEC-WB study were asked to complete a multiple choice teacher assessment tool developed by MOEC. The assessment provided a comparable measure of some aspects of MKiT, containing both subject matter and pedagogy sections. The limitations of measuring pedagogical knowledge through a multiple-choice assessment were recognized at the time of design of the study. Thus, other mechanisms were developed to compensate for this limitation. For example, the scenarios interview was designed to elicit aspects of MKiT through discussion of hypothetical classroom cases. The situations that arose naturally in the recorded lessons of the case study teachers were also taken into account and investigated through subsequent stimulated recall interviews. The triangulation of the three sources of pedagogical content knowledge was used to provide a better-rounded picture of the case study teachers. Main Report 43 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms Table 4: Sources of Data for MKiT Analysis Data Source Main Strengths Main Shortcomings MKiT: • Can be given to all teachers; not just case study • Testing theory rather than actual situations Pedagogical portion of the teachers • it is questionable whether pedagogical MKiT Teacher assessment • Easily comparable and objective from a scoring can be measured through a multiple-choice standpoint. assessment • Does not capture whether teachers can call upon their MKiT in real life situations MKiT: • All teachers work on the same scenarios, • Hypothetical situations are not able to fully Scenarios interview allowing for comparability capture knowledge in action • Scenarios are designed to target various • The scenarios can only partially bring pedagogical aspects of knowledge, including pedagogical MKiT to the surface the handling of student misconceptions. Practices: • Captures the teachers’ handling of actual • Each teacher had unique situations, not situations and their knowledge in action. allowing for comparability Situations from actual • The stimulated recall allows for a better • The teacher’s MKiT is almost impossible to lessons and subsequent understanding of the practices from the identify and assess since the researcher cannot stimulated recall interviews teacher’s perspective get into the mind of the teacher and, to a certain extent, can only conjecture 4.1 TEACHER SUBJECT AND PEDAGOGY COMPETENCY ASSESSMENT The results of the MKiT assessment, as shown in Figure 16 below, indicate that there was a wide range of scores among the teachers. Figure 16: Scatter Plot of Teacher Subject and Pedagogy Scores on the Teacher Assessment 100% 90% 80% 70% CS9 Pedagogy Score 60% CS5 CS7 CS6 50% CS4 CS3 CS2 CS10 40% 30% 20% 10% 0% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Mathematics Score 44 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Mathematical Knowledge in Teaching There is a clear correlation between the scores teachers received for teacher subject knowledge and pedagogical knowledge, with teachers who scored higher on the subject matter portion also tending to score higher on the pedagogical section. However, there are also many cases of teachers scoring relatively highly in one area, but low in the other. This provides a good variation when relating the scores to practices and beliefs. It is also important to note that the subject items were related to the junior secondary school curriculum topics and were not considered to be particularly advanced. The fact that the majority of teachers obtained scores below 60% and 16 teachers obtained scores below 40% appears to indicate that in the case of some teachers, the level of subject knowledge is not much more advanced than that of the students whom they are teaching. This is an important contextual aspect of Indonesia’s situation compared to relatively more developed countries, where teachers tend to have much higher levels of subject knowledge. It also ties into the results of MOEC’s national teacher competency testing (Uji Kompetensi Guru, or UKG), which showed similarly low competency levels. This will be discussed in detail in section 8.1. Interestingly, for the case study teachers, all but two are placed well above the trend line shown in Figure 16, indicating that their pedagogy scores are high relative to their subject scores. Case Study 9 (CS9) obtained the second highest score in the pedagogy portion and was among the top 10 in the subject portion. CS3 had the highest subject score of all 104 teachers. Although his pedagogy score is slightly above average, it is very low relative to his subject score, placing him well below the trend line and indicating a subject-heavy slant. CS5 and CS4 obtained relatively high scores in both sections. CS6 and CS7 obtained very strong pedagogy scores, but scores that were near the average in the subject portion. CS2 and CS10, on the other hand, obtained scores that were well below average in the subject portion, while their pedagogy scores were the lowest among the case study teachers (although they were only slightly below average compared to the full sample). These case study results will be of importance in section 4.3 when comparing the competency scores to how the teachers were able to address hypothetical classroom scenarios. It will also be of importance when examining their videotaped classroom practices when exploring the knowledge-practices link in section 6.3. 4.2 TEACHER MATHEMATICAL SCENARIOS INTERVIEW This section discusses the approach and findings of the mathematical scenarios interviews conducted with the case study teachers. It examines both individual case responses at the Micro case level and providing a general summary at the Mezzo case level. The scenarios interview consisted of five hypothetical mathematics lesson situations where teachers were asked to identify issues, make conjectures on students’ thoughts (e.g. misconceptions held by students) and strategies for how they would address the situations in their lessons. Main Report 45 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms The following is one example of the types of scenarios used in the interview: Mrs. Dian’s class was working on an assignment where they had to find the measures of unknown angles in triangles. One student consistently found the measures of the unknown angles in right triangles by subtracting the known angle from 90. For example: 90 - 62 = 28 X = 28 62° X Mrs. Dian was concerned that this student might run into difficulty when trying to find the measures of unknown angles in more general triangles. Intended Explorations • Recognizing that the student has the correct answer, but that by the student’s solution method, there may be a misconception. • Probing student to see if there is, in fact, a misconception or if the student understands and just came up with what might be considered a shortcut. • Developing a method for ensuring the concept is understood. Guideline of questions to ask in the interview • Why do you think Mrs. Dian is concerned? • (Teacher should identify misconception on his/her own, but if prompting is needed or if the teacher mentions a misconception exists, but does not specify what it is, then prompt.) What is the possible misconception that the student has? • Why might the student have come up with this method of solving the problem? • If you were the teacher, how would you approach this situation? All mathematical scenarios and more detailed analysis can be found in the Technical Report. 4.2.1 SUMMARY OF CASE STUDIES ON THE MATHEMATICAL SCENARIOS The table in Appendix 12.4 provides an overview of all case study teachers, based on distinguishing categories that emerged amongst teacher responses. The following are summaries of the sections analyzed. Of importance in this scenario analysis was to understand each case study teacher’s mathematical knowledge in hypothetical situations so that this could then be compared to their actual assessment tests. 46 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Mathematical Knowledge in Teaching Mathematics In general, the level of complexity of the mathematical concepts contained within the scenarios did not create major difficulties for the teachers. Often teachers would appear to understand the mathematical concepts raised in the scenario, but then chose an incorrect answer. In the case of Scenario 2 and Scenario 3, there is no disputing which of the responses was correct. However, Scenario 4 in particular was vague and open to interpretation. Sometimes a teacher could give a good justification for why an “incorrect” option would work. The vague nature of this scenario was useful in that it led to interesting discussions. However, it also left the level of MKiT demonstrated through the scenario open to interpretation. Scenario 5 generated many errors from the teachers, but this might have been due to the question itself being slightly unclear. Still, the teacher with the strongest subject matter knowledge (as determined through the teacher assessment) did a particularly admirable job of discussing inequality and identifying the correct answer, indicating that the scenario still serves an important function in the evaluation of teachers’ MKiT. The teachers who stood out as being particularly strong in their discussions of mathematical concepts were CS3, CS5 and CS9. Identify Issues Most teachers tended to spot the issues quickly, although the speed of recognizing the issues and the ability to describe it varied. One particularly distinguishing aspect related to the identification of issues was the speed with which teachers noted the issues in questions and the extent to which they elaborated on the issue itself. In particular, CS4 and CS7 would elaborate upon and discuss the issue, together with its root causes. It cannot be determined whether they truly grasped the issues better the other subjects or whether talking through the issue might have been their method for coming to terms with the situation in their own minds. Conjectures of Students’ Thoughts Another interesting distinction was that some teachers placed a greater emphasis on the students’ thinking processes, while others placed greater emphasis on the mathematical issues. CS3, who placed greater emphasis on the mathematical issues, was particularly interesting in this regard. The fact that he obtained the highest score on the subject portion of the MKiT assessment but a relatively lower score in the pedagogy portion probably helps to explain his focus on the mathematical aspects. By contrast, CS7 was much more focused on the student’s thinking process, continually placing herself in the mind of the student, although at times she struggled with the mathematical concepts. Her relatively high pedagogy score and her average subject score may help explain this. Develop a Strategy The development of strategies for the scenarios was revealing in terms of the level of knowledge of teachers, but it actually provided even greater insights into aspects of their beliefs. CS4 and CS5 came up with strategies that would enable the students to come up with the solution themselves, while CS2 would use phrases such as: “I would explain…” and “I would repeat…” In their formulation of a strategy, CS3 and CS6 emphasized mathematical aspects, while CS7 and CS9 came up with creative alternatives. Main Report 47 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms 4.3 LINKING MKIT ASSESSMENT AND SCENARIOS RESULTS As might be expected, the case study teachers with higher MKiT assessment scores were generally able to handle the scenarios with greater ease. They tended to identify the issues quickly and provided more complete and more elaborate responses when discussing the mathematics involved and the strategies they would use. The variance between subject and pedagogy scores was also reflected in the manner in which teachers managed the scenarios, with teachers gravitating towards and utilizing their respective strengths. This is an important finding, in that it indicates how teachers will use their knowledge in formulating and implementing their teaching practices. If one aspect is greater than the other, then the teacher is likely to utilize their relative strengths. The MKiT assessment has the benefit of producing a standardized score that can be compared across a large number of teachers. However, certain aspects of knowledge were demonstrated in the scenarios that could not have been detected through the assessment. For example, two teachers with low pedagogy scores came up with creative pedagogical techniques for how they would approach the particularly pedagogical content knowledge. On the other hand, one teacher with a high assessment score came up with very basic methods for dealing with student misconceptions, possibly suggesting that knowledge that can be demonstrated through the medium of a written test is not necessarily easily elicited and drawn upon in classroom situations. The three micro case studies provided insights into how the MKiT (as captured through the teacher assessment) and MKiT (as captured through the scenario interviews) appear to be correlated, but also how they diverge in certain aspects. CS3 provided the best example of how the assessment and scenario results are correlated. CS3 had the highest score of all teachers on the subject matter portion of the assessment and had a slightly higher than average score in the pedagogy portion relative to the entire sample, but only average compared to the other case study teachers. In the scenarios, CS3 demonstrated a strong knowledge of the mathematical concepts. This was particularly true in the case of Scenario 5, which requires a relatively high level of advanced mathematical thinking and which created problems for many other teachers, but which CS3 managed to address with ease. However, in terms of the pedagogical aspects of the scenarios, CS3 tended to struggle more than other teachers. This was particularly true in regards to “knowledge of students”. In Scenario 2, CS3 was prompted multiple times to conjecture as to why the student had made an incorrect statement. He persisted on saying that he did not know because he was not the student. This came through in other scenarios as well. Most of the other teachers would often discuss the issue from the perspective of the student, but CS3 tended not address the scenarios in this manner. 4.4 RELATIONSHIP OF TEACHER KNOWLEDGE WITH STUDENT LEARNING OUTCOMES Of all school system factors explored, teacher knowledge had the strongest relationship to student learning outcomes. The results indicate that both subject matter knowledge and pedagogical knowledge, in almost all cases, were well above the generally accepted 5 percent statistical significance threshold and typically above the 1 percent significance threshold (see Figure 17). The significance levels tended to drop when including teacher background variables such as educational level and experience, likely indicating that these other teacher characteristics also play an important role in student learning. 48 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Mathematical Knowledge in Teaching Teacher subject knowledge tended to have a stronger relationship with student learning than did pedagogical knowledge. This may in part be due to the measurements themselves, with subject matter knowledge lending itself well to a test whereas pedagogical knowledge is much more difficult to measure through a paper test. Still, the fact that pedagogical knowledge also tended to have strong results indicates its likely important role in learning outcomes. Figure 17: Relationship between Teacher Knowledge and Student Learning17 Overall Subject Pedagogical assessment score assessment score assessment score Direction and level of statistical signi cance 4 Thresholds of statistical signi cance 4 3.74 3.75 3 3.22 3.05 3 1% 2 2.19 5% 2.02 2 10% 1 1 0 Regression without teacher variables Regression wth teacher variables Coe cients 0.20 0.17 0.16 0.14 0.13 0.13 Std. Coe cients 0.24 0.23 0.24 0.23 0.15 0.13 Note: Full regression results can be found in the technical report. In interpreting the regression coefficient for the degree and its relation to student posttest scores, a percentage point increase on a teacher’s assessment score is related to a 0.17 percentage point increase in a student’s posttest score after controlling for multiple background factors and the student’s pretest score. Teacher assessment scores ranged from 7 percent to 75 percent, with an average score of 48 percent. For example, a 10-percentage-point higher teacher assessment score resulted in a student’s posttest score being 1.7 percentage points higher. Again, because the average score on the posttest was only 34 percent, the 1.7 percentage point difference represents a difference of 5 percent relative to the average score. (Similar but lower coefficients were found when looking at the subject and pedagogy score in relation to students’ posttest score, with a 0.14 and 0.10 percentage point increase, respectively.) The standardized coefficients indicate that a 1.0 standard deviation increase in teacher assessment scores is related to a 0.23 standard deviation increase in student test scores. 17 A great deal of caution should be used in assessing a teacher through a pedagogy examination because a teacher’s pedagogical effectiveness is not about knowing facts about teaching methods that can be reduced to a question in a multiple-choice examination. The video study is specifically intended to explore how teachers use their pedagogical knowledge “in the moment.” Still, the fact that such a strong relationship exists between the pedagogy score and student learning indicates that the test is, in fact, capturing some aspect of the teachers’ knowledge and appears to provide a proxy for what they do in the classroom. Main Report 49 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms 4.5 KEY POINTS ON TEACHER KNOWLEDGE There are clear distinctions in how teachers approached the mathematical scenarios, with these distinctions based on their level of subject and pedagogical knowledge. Furthermore, it appeared that teachers would shape the scenarios according to their relative strengths, seeing the issues in either more mathematical or more pedagogical terms according to their strengths and developing strategies accordingly. This point is particularly relevant when exploring relationships and is carried over into the analysis in Section 6.3. While generally there was a clear relationship between the MKiT Assessment scores and the knowledge demonstrated in the mathematical scenarios, there were cases where teachers with high scores appeared to struggle to put that knowledge into play when discussing the scenarios. On the other hand, there were teachers who demonstrated a high level of knowledge, particularly pedagogical knowledge, when discussing the scenarios, despite the fact that they obtained relatively lower pedagogy scores. In relating teacher knowledge to student learning outcomes, there is a clear positive relationship, but this is only an initial step in understanding why knowledge might be an important factor in student learning. Knowledge is explored in much greater detail through relations with teaching practices, both with quantitative analysis and more in-depth qualitative analysis. These results are presented in section 6.3 and later presented as a thematic exploration in section 8.1. 50 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning CHAPTER 5 TEACHER BELIEFS ON TEACHING AND LEARNING A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms The systematic study of teachers’ conceptions of mathematics has been of continuing interest to researchers, with these conceptions being considered by many researchers to play a significant role in shaping teachers’ characteristic patterns of instructional behavior. Wilson and Cooney (2002, page 128) stated what appears to be a basic premise of mainstream research into teacher beliefs: “Over the last 15 years, there has been a considerable amount of research on teachers’ beliefs based on the assumption that what teachers believe is a significant determiner of what gets taught, how it gets taught, and what gets learned in the classroom.” Teachers do not just rely on their pedagogical knowledge or on curriculum guidelines. Rather, their beliefs act as a filter through which they make decisions related to teaching practices (Clark and Peterson, 1986). While teaching practices shape what happens in the classroom, it is teacher beliefs that can explain why these practices are utilized. From the point of view of teachers’ professional development, it is important to understand the nature of teacher beliefs. Cooney et al. (1998, page 331) state that, “An analysis of belief structures, attention to the intensity with which beliefs are held, and the nature of the evidence that supports beliefs can provide a forum by which our teacher education programs will be better able to address issues of reform.” Thompson (1984, page 106) focused on the negative consequence of not taking teacher mathematical beliefs into account when he stated that: “A failure to recognize the role that the teachers’ conceptions might play in shaping their behavior is likely to result in misguided efforts to improve the quality of instruction in the schools.” Kagan (1992) argues that teacher beliefs may be the most significant measure determining teachers’ professional growth. Table 5: Beliefs Lenses and Description of Categories Used in the Analysis Lens Categories Main Stance/Emphasis* Transmissionist Typically more teacher-centered and traditional instructional approach with verbal explanations Orientation Connectionist More student-centered, involving students in conceptually-focused discussion; (Askew et al., 1997) learning about and making connections amongst concepts Discovery Teacher acts as a guide in student-led practical activities where students discover methods themselves. Instructor Skill mastery Role of Teacher Explainer Conceptual understanding (Ernest, 1989) Facilitator Confident problem solving Learner Focus Stresses the learner’s construction of mathematical knowledge through social interaction Teacher’s Focus in Approach Content - Conceptual Focus Driven by the content itself, but with an emphasis on understanding concepts (Kuhs and Ball, Content - Performance Focus Values performance as the key goal whose attainment depends on the 1986) mastery of rules and procedures Classroom Focus Mathematics teaching based on research knowledge of effective classrooms School - knowledge Believing that teaching is an act of passing information on to others while Purpose of learning involves the process of reproducing that information Schooling (Renne, 1992) Child-Development More likely to consider children’s needs and characteristics as the primary factors in instructional decision making Teaching Approach Progressive Gives recognition and value to new instructional strategies in which students (Handal, 2003; (socio-constructive) are able to learn mathematics by personally and socially constructing Dewey, 1938) mathematical knowledge Traditional Emphasize transmission of knowledge and stress the pedagogical value of (behaviouralist) formulas, procedures and drill, and products rather than processes 52 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Teacher Beliefs on Teaching and Learning What are the mathematical beliefs and orientations of Indonesia’s eighth grade mathematics teachers? The analysis of these beliefs focuses first on an analysis of the mathematical belief interviews and then of the mathematical beliefs survey. The two analyses are then compared and contrasted with the aim of determining the nature of the mathematical beliefs held by the case study teachers and how these beliefs relate to those of the larger set of 104 teachers surveyed. 5.1 TEACHER MATHEMATICAL BELIEFS SURVEY The mathematical beliefs survey consisted of items that the participants ranked from 1 to 5 on the Likert scale. The survey also includes sections in which teachers ranked statements in relation to their beliefs, allocating proportions to three separate statements based on the extent to which they agreed with each statement relative to the other two. Thus, a teacher might give 70% to the statement with which they most agreed, followed by 20% for a statement with which they agreed to a lesser extent, and 10% for the statement with which they agree the least. The distribution of responses for each question is summarized in graphs that can be found in Appendix 12.2. The full Technical Report provides a step-by-step analysis of the survey results. The results presented in Figure 18 below include items to reach a large majority of teachers provided either positive or negative responses. Figure 18: Likert Scale Statements with Large Majority of Responses being Either Positive or Negative Strongly Disagree Disagree No opinion Agree Strongly Agree Study of mathematics is suited mostly to males 56% 37% 3%2%2 Telling the children the answer is an e cient way of facilitating their mathematics learning 18% 56% 0% 23% 2% For many citizens it is not necessary to know mathematics beyond simple calculations (primary school level) 15% 60% 3% 15% 8% Students learn mathematics best in classes with students of similar abilities 6% 61% 2% 27% 4% Mathematics is learnt well by working individually on mathematical problems 5% 48% 4% 37% 7% Students can solve most mathematical problems on the basis of memorized formula 5% 44% 2% 47% 3% Listening carefully to the teacher explain a mathematics lesson is the most e ective way to learn mathematics 4% 37% 5% 43% 12% The applications of mathematical result are more important than its proof 3% 27% 6% 50% 14% The most e ective way to teach mathematics is to rst explain the topic, give example(s) and then ask students to solve mathematical tasks 18% 1% 52% 28% It is the teacher’s responsibility to provide children with clear and concise solution methods for mathematical problems 1 11% 0% 65% 23% Persistent questioning has signi cant e ect on children’s mathematical learning 9% 6% 62% 23% Mathematical material is best presented in an expository style: demonstrating, explaining and describing concepts and skills 8% 2% 55% 35% I function in my classroom as a learner and partner in learning with my students 6% 1% 72% 21% Students learn most e ectively when learning tasks are broken down into small sequential steps 1 4% 4% 76% 15% It is important that mathematics content be presented to children in the correct sequence 4% 2% 51% 43% Teachers can create, for all children, a non-threatening environment for learning mathematics 3%2% 50% 45% Children always bene t by discussing their solutions to mathematical problems with each other 3%1% 57% 39% Learning mathematics in school is essentially an active process 2%2% 59% 37% Main Report 53 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms Teachers tended to express strong agreement with statements of a social nature that might indicate disadvantage or distinction for some group. The highest level of disagreement was expressed for the statement: “The study of mathematics is suited mostly to males.” Teachers also tended to express disagreement with statements that students learned best in classes with students of similar abilities and that mathematics at the primary level was sufficient for some citizens. For questions related to teaching approaches and how student learn, teachers tended not to express strong disagreement. However, teachers did tend to express moderate disagreement with some statements regarding teaching practices, with most teachers expressing at least moderate disagreement to the proposition that telling students the answer was efficient for student learning. In fact, many teachers expressed strong disagreement with this statement. Teachers also tended to express agreement with statements related to certain social aspects of teaching, with nearly half of all teachers expressing strong agreement with the statement that teachers can create a non-threatening learning environment for students. Ninety-six percent of teachers expressed at least moderate agreement with the statement that learning mathematics in school is essentially an active process. This is probably due to an Indonesian policy espousing “active and joyful learning,” making the term ‘active’ resonate with teachers. In the area of statements related to teaching technique, 90% of teachers expressed at least moderate agreement with the statement that mathematics material is best presented in an expository manner, involving demonstrating, explaining and describing concepts and skills. A significant proportion of teachers also expressed at least moderate agreement with statements related to the correct sequencing of teaching and breaking down learning tasks into small sequential steps. In the case of some statements, the proportion of teachers who expressed either agreement or disagreement was fairly evenly split. For example, the proportion of teachers who expressed agreement or disagreement with the statement that listening carefully to the teacher explain a mathematics lesson is the most effective method for learning mathematics was almost equal. Such statements were most useful in analyzing the relationship between beliefs and practices, since there was a wider range of responses. Teaching Practice Frequency Items In comparison to the teachers’ responses through the Likert scale, the teachers’ responses to questions regarding the frequency with which they utilized certain teaching practices in their own lessons were more evenly spread. The least frequent response was almost never and the most frequent response was occasionally, followed by almost always. 54 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Teacher Beliefs on Teaching and Learning Figure 19: Distribution of Frequency Scale Responses in the Mathematical Beliefs Survey Most of the time 19% Almost always 25% Half the time 14% Almost never 10% Occasionally 32% Of course, the responses must be viewed in light of the fact that teachers may have been responding in terms of what they felt they were supposed to do, rather than what they actually did. Thus, they may have considered that a response of almost never should be avoided. Still, the responses can be taken at face value or “stated frequency” and later compared to the videos to check the actual frequency with which a practice was utilized by a teacher and thereby to determine the extent to which the two coincide.18 Figure 20: Frequency Scale Results on a Sample of Teaching Practices Almost never Occasionally Half the time Most of the time Almost always I only go through one method for doing each question 24% 51% 11% 8% 7% I jump between topics as the need arises 24% 48% 9% 12% 8% I teach each student di erently according to individual needs 23% 37% 8% 13% 20% I encourage students to make & discuss mistakes 16% 30% 19% 13% 23% I nd out which parts students already understand and don’t teach those parts 17% 25% 11% 25% 23% I tend to teach each topic separately 13% 30% 16% 19% 23% I draw links between topics and move back and forth between topics 8% 42% 13% 19% 19% Students learn through discussing their ideas 7% 52% 18% 16% 8% Students compare di erent methods for doing questions 5% 38% 14% 16% 28% I tend to follow the textbook or worksheets closely 3% 46% 18% 18% 16% Students work on their own, consulting a neighbour from time to time 1% 36% 17% 31% 16% I tell students which questions to tackle 4% 26% 19% 24% 27% I avoid students making mistakes by explaining things carefully rst 6% 18% 11% 32% 33% Students use only the methods I teach them 8% 23% 14% 17% 38% I teach the whole class at once 9% 11% 5% 23% 53% I know exactly what maths the lesson will contain 5% 14% 5% 24% 53% 18 An absence of practices in the 3-5 videotaped lessons would, of course, not mean the teacher never uses the practices, but at least general triangulation can be done. Main Report 55 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms A surprisingly high 75% of respondents stated that they almost never or only occasionally utilize only one method for addressing a particular question, implying that they often used multiple methods. Similarly, 72% of respondents stated that they almost never or only occasionally jumped between topics as the need arises. On the other hand, a large proportion of teachers stated that they almost always or most of the time used practices that involved: (i) always knowing what mathematics the lesson will contain; (ii) teaching the whole class at once; and (iii) the students only using methods given by the teacher. Certain practices received an even spread of responses, from almost never to almost always. Statements in this category included those relating to encouraging students to make and discuss mistakes; tending to teach each topic separately; and finding out what students already know and not teaching those parts. Potentially conflicting results also emerged. For example, only 20% of respondents stated that they almost always or most of the time “jumped between topics as the need arises.” By contrast, 38% of respondents stated that they almost always or most of the time “drew links between topics and moved back and forth between them,” a statement somewhat similar to the first statement. Possibly the teachers perceived a difference between the two statements, but the variance in the responses may also be a warning regarding their accuracy. As with the Likert scale statement, only a portion of the frequency statements (16 of the 27) are shown above. However, the full results can be found in Appendix 12.2. 5.2 CASE STUDY INTERVIEWS ON TEACHER MATHEMATICAL BELIEFS The full analysis of the case study interviews on beliefs can be found in the Technical Report, including individual responses to the Beliefs Orientation game, as well as the results of general interviews into how the teachers beliefs were formed; how the perceived students learn best; and what teaching practices they believed are most effective in the teaching-learning process. The following subsection summarizes the key findings from the interviews: No Teacher Fell Completely into an Orientation Category From the outset of the exploration into teachers’ mathematical beliefs, it was not expected that all the data for a particular teacher would point to a single categorization. The results confirmed this. Teachers may express beliefs that would tend to align with certain categorizations. However, even the teachers who could most easily be placed in a particular category would also tend to express beliefs that would not fully align with, or that would even contradict, the categorization. Building on these results, the next stages of analysis must keep this in mind and avoid oversimplifying or typecasting the case study teachers. Instead, their categorizations should be viewed as tendencies or as classifications of their dominant expressions. Put simply, teachers, like any other people, are capable of holding different beliefs that may appear to contradict each other. Beliefs are Contextually Fluid It became very clear through the interviews that there is a distinction between beliefs in an abstract, general sense and beliefs in a practical, situationally specific sense; there is a dance that takes place between the two. At times the abstract and specific embrace as though they are a single entity, at other times they are apart and only lightly touching one another. This dance took place as teachers spoke of their beliefs, with some teachers appearing to be overtly aware of the distinction, while others blended belief and action as if there were no contradiction. 56 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Teacher Beliefs on Teaching and Learning For example, it appears that teachers may believe a certain teaching technique is ideal in a general sense, but due to policy, student achievement level, community and family environment or a host of other possible influencing factors, the teacher’s belief in how to actually teach in his or her current circumstances may not fully align with those general beliefs. One teacher was highly aware of this, frequently making statements such as: “I would prefer to teach using a more advanced method, but due to the level of my students, I must use a different technique… I must adapt my teaching to the level of the students.” This teacher even stated that within a single lesson, his techniques would vary based on the circumstances, with these variations affected by the purpose of the lesson segment and the nature and quality of the students with which he was working. How, then, should this be taken into account when examining the teacher’s videotaped lessons or analyzing comments made in interviews relating to teaching practices? As mentioned earlier, a distinction can be drawn between espoused and enacted beliefs (e.g. Ernest, 1989; Thomson, 1992; Butt and Raymond, 1989; Anderson, 1997), but this can be misleading. The distinction between espoused and enacted beliefs seems to imply that if there are differences between the two, then the teacher is either not aware of the contradiction or is prevented from following through on his or her espoused beliefs due to existing constraints. How should it be interpreted when a teacher is aware of the contradiction and states that in different circumstances he or she would teach differently? Possible reasons for this occurring include: 1. The teacher holds a belief or orientation (as espoused in the interview), but consciously decides it is not applicable to the specific situation and uses a different approach (enacted in the videotaped lessons). This could be interpreted to mean that in fact, the teacher holds multiple beliefs. While the teacher tends to favor a certain approach, he or she is also willing to use alternative approaches based on the context. 2. The teacher holds a belief or orientation (espoused), but is unaware that his/her actions (enacted) are do not align with that belief.19 3. Possibly due to pressures (policy, peers, what was learned in training, etc.), the teacher expresses what he or she thinks is the “right” answer or response, but he or she does not truly believe it. 4. The teacher is not actually sure or aware of what belief he or she holds, but provides a response anyway. 5. The teacher does not hold a strong belief or tendency towards a certain orientation, so when it comes to actual classroom situations, the espoused beliefs are easily discarded. There Appeared to be Limited Direct Reflection on Beliefs The beliefs interview contained some sections that addressed beliefs in an abstract, general sense, while other sections addressed beliefs in a more direct, concrete and practical sense. While all teachers were able to discuss the more concrete aspects of teaching and learning, the sections that tended to address beliefs in a more abstract sense presented challenges. It appeared that some teachers did not tend to think about the nature of mathematics, how students learn, and how to teach in an abstract sense. The case study teachers were asked what they thought about the beliefs survey in which they had participated and were also asked to what extent they had thought about and reflected upon their beliefs. All but two teachers indicated that they had never directly thought about their beliefs in a direct or philosophical sense and found the process a bit strange. On the other hand, four teachers indicated that while they may or may not have directly considered these issues previously, doing so was an enjoyable learning experience. 19 Alternatively, it is possible that it is only a perceived contradiction by me, but that in the teacher’s mind there is no contradiction. It is therefore important that I not unreasonably infer differences and contradictions. Main Report 57 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms Adherence of Beliefs versus Pragmatists Some teachers spoke passionately and eloquently about their views, while others struggled to form and express their ideas or did not feel certain about taking a stance. Some teachers had clearly thought about issues and consciously chosen not to take a stance or expressed their response in a nuanced manner, often prefacing their statement with reference to contextual factors. The degree to which teachers may hold their beliefs is extremely important, as the degree to which they hold their beliefs is likely to play a role in how they incorporate these beliefs into their practices. This relates back to the literature that discusses how some teachers doggedly cling to their beliefs (e.g. Kagan, 1992; Nespor, 1987; Swan, 2006; Calderhead, 1996). However, some research shows that teachers can and do change their beliefs, with teachers holding more relativistic orientations to teaching mathematics being more likely to consider and adopt new ideas (Arvold and Albright, 1995). This concept of adherents versus pragmatists is an important concept for analysis. In later sections of this report, the concept of adaptability and flexibility are explored in greater detail. 5.3 LINKAGES BETWEEN BELIEFS SURVEY AND BELIEFS INTERVIEW Through the survey, teachers provided responses to specific items, with it possible to group the items in relation to specific categories. The interview provided insights into teachers’ thoughts regarding mathematical beliefs, affording a much more elaborate and detailed view of the teachers’ mathematical beliefs, uncovering important aspects of these beliefs that could not be captured through a survey. The survey responses were compared to the interview responses to determine whether the responses appeared to be consistent and whether the responses from one could enrich the understanding of the responses to the other. The data from the surveys and interviews were linked in two ways. First, individual questions within the survey were related to what teachers say in the interviews. For example, the survey statement: “Allowing a child to struggle with a mathematical problem, even a little tension, can be necessary for learning to occur” was specifically discussed as one of the topics in the mathematical beliefs interview. For this item in the survey, there was significant variation in the case study teachers’ responses, with four disagreeing, three agreeing and one strongly agreeing. Relative to the responses in the interview, six of the teachers’ responses were consistent, one was mixed and one was clearly inconsistent. In the case of the inconsistent response (CS2), it is likely that the teacher either did not understand the survey statement or interpreted it differently when participating in the survey. It is also possible that when discussing the topic in the form of an interview, she had more time to consider the issue and so gave a more nuanced response that contradicted her response through the survey. The comparison between the multiple survey items and interview results showed that there was a high level of consistency between the two sources. This was particularly true when grouping items. For example, survey items related to how the teacher pictured his or her role were generally very consistent with the interview results. However, at the individual item level, for the statement: “It is more effective to provide students with the information they need to know, rather than encouraging them to experiment,” the teachers tended to make statements in the interview that contradicted their responses through the survey as they attempted to verbalize the degree to which they would let students work on their own and how much guidance they would need to provide. 58 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Teacher Beliefs on Teaching and Learning In general, the interview results highlighted the complexity of beliefs. In discussing a topic, a respondent might make multiple statements that seem to be contradictory if they are viewed in isolation. However, when viewed in the context of an overall narrative, it was possible for the apparent contradictions to be resolved. Seen in this light, the topic of beliefs is perhaps much better investigated through an interview, in which teachers are able to elaborate on their ideas, than through a survey, which tends to lead to oversimplification. With survey statements, teachers sometimes clearly misunderstood their meaning or interpreted them differently when the same subject was discussed in the interview. Due to the complex nature of teachers’ beliefs, the profile given created through the interview could be considered considerably more trustworthy than the profile created through the survey, although, as stated earlier, there was a considerable degree of alignment between the two. 5.4 BELIEFS RECONCEPTUALISED Various studies have conceptualized teacher beliefs by applying the use of categorizations. Many of these categorizations have been used as lenses for analysis. A key finding of this beliefs analysis was that while some teachers exhibited tendencies that align them with particular categorizations, others defied categorization. Sometimes, the defiance was implicit, with the survey data and beliefs interview not enabling a clear categorization. In the beliefs interview in particular, some teachers often used the expression “it depends” when discussing the application of teaching approaches to a particular situation, indicating that for them teaching practices are dynamic and context- specific. For case study teachers CS3 and CS9, this defiance was explicit, with both specifically stating that they did not regard their teaching methods as being susceptible to categorization. To further emphasize the point, CS9 gave examples of how she would switch from one type of approach to another within a single lesson. This point features prominently in subsequent steps of the analysis that relate beliefs to knowledge and practices. It also has a significant impact on the development of the new conceptual model on the interaction between beliefs-knowledge-practice presented in Chapter 7. As will be discussed later, a key finding is that teachers with a higher level of knowledge also tended to have a greater degree of flexibility in their beliefs. It appears that this flexibility allows for more adaptation in practices and, in turn, more effective teaching. Main Report 59 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms 60 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning CHAPTER 6 EXPLORING THE BELIEFS- KNOWLEDGE-PRACTICES RELATIONSHIP A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms Previous chapters have defined and analyzed the components of beliefs, knowledge and practices in isolation. Building on these previous chapters, this chapter explores the relationships between these components. With reference to the Analytical Steps Framework (see section 2.10.2), this chapter explores dual relationships between Knowledge-Practices (Point D); Beliefs-Practices (Point E); and Beliefs-Knowledge (Point F). The intersections between these components have been marked with a circle in the illustration at the beginning of each subsection. This section addresses a fundamental research question on the beliefs-knowledge-practices relationship: What is the relationship between teaching practices, teacher mathematical beliefs and mathematical knowledge in teaching? Through the analysis and results that are outlined in this section, it becomes clear that a model is necessary to explain conceptually what has been learned through the findings regarding the relationship between the components. This model will be described in greater detail in Section 7. 6.1 THE BELIEFS-KNOWLEDGE LINK Most of the existing research into Beliefs and MKiT tends to PRACTICES examine either the beliefs-practices link or knowledge-practices Video Teaching practices used in the lessons link. While some recent studies (such as COACTIV, MT21, A Askew et al, 1997) have examined beliefs, knowledge and Stimulated recall D How teacher describes events within the lessons and explains decision/actions E practices together, the focus has tended to be primarily on the relationship between beliefs and practices on the one hand and G MKiT Scenarios MATHEMATICAL BELIEFS Beliefs Interview then on the relationship between knowledge and practices on Demonstrated knowledge of students, knowledge of practices and subject knowledge Espoused beliefs through general discussion and beliefs situations the other. Their studies tend not to touch upon the relationship B F C between beliefs-knowledge or, if this relationship is mentioned Teacher Assessment Subject and pedagogical knowledge test scores Beliefs Survey Questions covering nature of mathematics, how students learn, how to teach and at all, it is not analyzed in a rigorous manner. In this study, the beliefs-knowledge relationship is viewed as an important link teacher’s own practices in considering why teachers teach the way they do. 6.1.1 MACRO VIEW: LINKING THE BELIEFS SURVEY AND TEACHER ASSESSMENT RESULTS Analytical Approach Regression Analysis The regressions were conducted following the same general multi-level model regression methodology specified in Section 2.12. In the case of comparing beliefs and knowledge, each of the responses to the statements in the Beliefs Survey was individually regressed against the teacher’s (i) total assessment score; (ii) subject matter score; and (iii) pedagogy score. In the case of this analysis, the goal was only to identify relationship patterns, so the variable of interest was the t-statistic. Through this variable, it could be determined which beliefs statements were positively or negatively correlated to a statistically significant level with the above three knowledge measures. As with the model regression methodology utilized throughout this study, matrices are used to describe statistically significant positive relationships (which are highlighted in green) and negative relationships (which are highlighted in red). 62 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Exploring the Beliefs-Knowledge-Practices Relationship Figure 21: Proportions of the Relationship between Beliefs Questions and Knowledge Breakdown of Response Relationships Number Percent Percent within Relationship No Relationship 37 42 Both Subject and Pedagogy 24 27 46 Subject only 19 21 37 Pedagogy only 8 9 15 Opposite directions for subject and pedagogy 1 1 2 TOTAL 89 HOW TO INTERPRET THE BELIEFS KNOWLEDGE FIGURES IN THIS SECTION In order to present the relationships found between beliefs and knowledge in a clear and simple manner, matrices were created that show the beliefs statement along with columns representing MKiT dimensions as well as dimensions on student achievement level. Below is an example matrix, followed by explanations of the columns: Entry Level ME Degree Pedagogy Learning Beliefs Statements Distribution Subject MKiT For many citizens it is not necessary to know mathematics beyond simple calculations (primary school level) Students can solve most mathematical problems on the basis of memorized formula It is important that mathematics content be presented to children in the correct sequence • The “Distribution” column shows the Likert scale distribution of responses from all 104 teachers to the beliefs statement. This is included as a reference for interpretation of the MKiT scores. Dark orange=strongly disagree; light orange=agree; brown=neutral; light blue=agree; and dark blue=strongly agree The remaining columns are relationship indicators that are color coded based on the t-statistic result in the regression analysis. Dark red indicates a negative relationship and statistical significance at the 1% level; medium red at the 5% level; and light red at the 10% level. Dark green indicates a positive relationship and statistical significance at the 1% level; medium green at the 5%; and light green at the 10%. The threshold of 5% is used as the threshold for defining statistical significance, but the 10% is still shown to indicate that in the case of that item, this threshold was approached. • ”Entry Level” column: This column indicates the relation between the beliefs statement with the entry achievement level of students in the teachers’ classes. A positive relationship would indicate that teachers having higher entry achievement level students tended to agree more with the statement. • ”Learning” column: This column indicates the relation between the beliefs statement with the learning outcome of students in the teachers’ classes. A positive relationship would indicate that teachers whose students learning outcomes increased to a greater extent tended to agree to a greater degree with the statement. • “MKiT” column: This column indicates the relation between the teachers’ overall teacher assessment scores and the beliefs statements. A positive relationship would indicate that as teachers’ MKiT scores increase, there is a greater tendency to agree with the statement. The “Subject” column indicates the relation with the mathematics (subject) section of the teacher assessment, while the “Pedagogy” column indicates the relation to the pedagogy section of the teacher assessment • ”ME Degree” column: This column indicates the relationship between the teachers’ relative level of agreement with the statement with whether they hold a mathematics education degree. A positive relationship would indicate that the teachers holding a mathematics education degree tend to agree with the statement to a relatively greater degree than those with a pure mathematics degree or a non-mathematics degree. Main Report 63 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms Item Results The results indicate that in the case of 42% of the Likert scale beliefs statements, no statistically significant relationship emerge between the statements and MKiT scores. When a relationship could be determined, nearly half the time (46%), it emerged for both subject and pedagogy knowledge. In 37% of cases, it only emerged for the subject portion, while in 16% of cases, it only emerged for pedagogy. One curious case involved an item had in which both subject and pedagogy relationships emerged, but in a contradictory direction. A breakdown of these relationships will be presented and discussed in this chapter. Comparison of Subject and Pedagogy Results In the case of some statements, the results for both subject and pedagogy were related to the beliefs statement to a statistically significant degree. In the case of other statements, a relationship only emerged for either subject or pedagogy. There was also one statement with which both subject and pedagogy relationships emerged, but in a contradictory direction. The following analysis examines each of these cases to see if patterns might explain why certain differences emerged. Beliefs statements that had a statistically significant relationship with both subject and pedagogy scores are shown in Figure 22 below. One interesting finding that emerged from an examination of these statements is that teachers with relatively higher MKiT scores tended to express a belief in the importance of being thorough; of covering everything in a topic; and of presenting in the correct sequence (positive relationship), while disagreeing to a relatively greater extent with statements suggesting that it was only important to cover important ideas or to only to use one method for each question. They also tended to disagree to a greater extent with statements suggesting that students were able to solve most mathematical problems on the basis of memorized formula or that learning with understanding can be achieved through practicing the mathematical exercises provided in the textbooks. From the example presented in the graphical analysis section, teachers with higher levels of knowledge tended to disagree with the statement that it was not necessary for many citizens to have a knowledge of mathematics beyond that required to do simple calculations. One surprising result is that teachers with higher levels of knowledge tended to disagree to a relatively greater extent with the statement that drawing links between topics and moving back and forth between topics was an effective approach. If “making connections” is taken to indicate a connectionist approach, then the higher knowledge teachers tended to disagree with this connectionist-oriented statement. 64 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Exploring the Beliefs-Knowledge-Practices Relationship Figure 22: Beliefs Statements Having a Relationship with Both Subject and Pedagogical Knowledge Entry Level ME Degree Pedagogy Learning Beliefs Statements Distribution Subject MKiT For many citizens it is not necessary to know mathematics beyond simple calculations (primary school level) The more students practice mathematical exercises given in the textbooks, they learn mathematics with understanding Students can solve most mathematical problem on the basis of memorized formula It is important that mathematics content be presented to children in the correct sequence I try to cover everything in a topic I draw links between topics and move back and forth between topics I only cover important ideas in a topic I only go though one method for doing each question As stated previously, there were cases in which there was a statistically significant relationship between certain beliefs statements with either subject MKiT or pedagogy MKiT, but not both. These statements provide interesting insights into possible distinctions between types of MKiT. The statements shown in Figure 23 are those in which there was a statistically significant relationship with subject knowledge but not with pedagogical knowledge. The statements tended to involve aspects related to working on mathematical problems. On their statements related to what takes place in their own classroom, teachers with relatively higher levels of subject MKiT stated that they used the approach of having students invent their own methods; choosing which questions they tackle themselves; and learning through discussing their ideas, to a relatively lesser extent than other teachers. This result was somewhat surprising, in that the researchers had hypothesized that teachers with progressively higher levels of subject MKiT would tend to be more comfortable letting their students have more freedom to experiment. A possible explanation is that these approaches would imply higher levels of knowledge in terms of pedagogical theory, but that the teachers with higher levels of knowledge of subject MKiT tend to be more concerned with providing structure and therefore remained focused on the problems they had selected. For example, CS3 had the highest score in terms of subject MKiT, but only an average score in the area of pedagogy. He wanted students to follow his methods and therefore he tended to present problems in small, sequential steps, with clear instructions on how to solve them. However, this would contradict the result for another statement that tended to attract a higher level of disagreement from teachers with higher subject MKiT levels: “It is the teacher’s responsibility to provide children with clear and concise solution methods for mathematical problems.” Main Report 65 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms Figure 23: Beliefs Statements Having a Relationship with Subject But Not Pedagogical Knowledge Entry Level ME Degree Pedagogy Learning Beliefs Statements Distribution Subject MKiT Students learn most e ectively when learning tasks are broken down into small sequential step It is the teacher’s responsibility to provide children with clear and concise solution methods for mathematical problems Paper pencil test is the best way to examine students’ mathematical learning Mathematics is learnt well by working individually on mathematical problems Students choose which questions they tackle Students learn through discussing their ideas Students invent their own methods Students learn through doing exercises In the case of other beliefs statements, there was a statistically significant relationship with the level of pedagogical knowledge, but not with the level of subject knowledge, as shown in Figure 24. Teachers with higher pedagogy scores tended to agree with statements that mathematics is a creative field of endeavor. Thus, a greater proportion of these teachers agreed with statements such as: “Mathematics is a creative subject like arts and music”, while disagreeing with statements such as: “Expecting students to be creative is unreasonable and does not warrant the use of precious lesson time.” They also disagreed to a relatively greater extent with a statement that telling students the answer is an effective way of facilitating the learning process. Generally, statements in this category tend to imply a more student-centered approach to the learning process, with these teachers more prepared to take the student’s perspective into account and to develop and adjust the lesson to meet the students’ needs. Figure 24: Examples of Beliefs Statements with a Relationship with Pedagogical But Not Subject Knowledge Entry Level ME Degree Pedagogy Learning Beliefs Statements Distribution Subject MKiT Mathematics is a creative subject like arts/music Expecting pupils to be creative in mathematics is unreasonable and doesn’t warrant the use of precious lesson time I adjust my lesson plan based on results of homework assignments Telling the children the answer is an e ecient way of facilitating their mathematics learning I adjust my lesson plan based on result of homework assignments Students work on their own, consulting a neighbour from time to time 66 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Exploring the Beliefs-Knowledge-Practices Relationship There was only one statement, the statement with which there was a statistically significant and opposite result between the teachers with relatively higher subject knowledge scores and those with relatively higher pedagogy scores. This statement was: “I avoid students making mistakes by explaining things carefully first” shown in Figure 25, The result was somewhat surprising, in that it was conjectured that teachers with relatively high pedagogy MKiT scores might see value in using mistakes as opportunities for learning, while teachers with relatively high subject MKiT scores might be more inclined to attempt to explain things carefully in order to avoid mistakes. One possible is that those with relatively higher subject MKiT scores tend to be more comfortable with mathematical mistakes and misconceptions that would require their knowledge to explain. Teachers with relatively low subject MKiT scores, on the other hand, might have wanted to avoid these situations. The term “mistake” is also one that is open for interpretation and possibly the teachers with higher pedagogical MKiT scores tended to consider that the term “mistakes” in this statement referred to errors arising from poor initial instruction on the part of the teacher. Figure 24: Example of Beliefs Statement with Subject and Pedagogy MKiT Scores in Opposite Directions Entry Level ME Degree Pedagogy Learning Beliefs Statements Distribution Subject MKiT I avoid students making mistakes by explaining things carefully rst Responses with Knowledge and Student Learning in Opposite Directions The results from the knowledge analysis described in Section 4 highlighted the fact that there was a strong positive relationship between levels of both teacher subject knowledge and teacher pedagogical knowledge with student learning outcomes. When comparing the responses to the beliefs statements with both levels of teacher knowledge and student learning outcomes, it is generally the case that in the cases of responses to beliefs statements in which there was a statistically significant relationships with both student learning outcomes and levels of teacher knowledge, the relationships would both tend to go in the same direction. However, there were some cases in which the levels of teacher knowledge and student learning outcomes would go in opposite directions. These statements and their associated relationships are presented in below. Main Report 67 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms Figure 26: Beliefs Statements where MKiT and Student Learning have Opposite Relationships Entry Level ME Degree Pedagogy Learning Beliefs Statements Distribution Subject MKiT For many citizens it is not necessary to know mathematics beyond simple calculations (primary school level) The more students practice mathematical exercises given in the textbooks, they learn mathematics with understanding Students can solve mathematical problems on the basis of memorized formula It is important that mathematics content be presented to children in the correct sequence I try to cover everything in a topic I draw links between topics and move back and forth between topics I only cover important ideas in a topic I only go through one method for doing each question The second statement indicates that teachers with higher levels of MKiT, both in subject knowledge and pedagogical knowledge, tended to agree to a lesser extent with the statement that textbooks and other published materials are the best source for curriculum development, with the negative relationship indicated in red for Overall, Subject and Pedagogical components of MKiT. However, students of teachers who tended to agree with the statement tendered to have relatively better learning outcomes, with the positive relationship indicated in green under (L)earning. How might this be interpreted?20 The case studies provide a likely explanation. CS3 and CS9 both scored very high on the MKiT assessment and both spoke of their enjoyment in creating their own examples for classroom work. It is likely that a teacher with higher levels of MKiT is much more likely to be able to define and tailor examples than a teacher with lower levels, who is likely to feel much more comfortable relying on textbooks and other pre-defined examples. CS2, on the other hand, had a very low MKiT score. In the case of two of her observed lessons, she spent the instruction time reading step-by-step instructions for calculating an inner and outer tangent to her class and having her students write down the steps. Further, when asked in the stimulated recall interviews to explain why the steps were important she would repeatedly say: “They are listed in the steps,” rather than demonstrating a mathematical understanding of why the steps were important. But why might there be a positive relationship with learning outcomes? There tends to be a wide variation amongst Indonesian teachers’ levels of MKiT. For those teachers with lower levels, the textbook and other published materials may provide useful and possibly necessary support for those teachers. It is possible that teachers with higher levels of MKiT can be effective in creating their own materials, without having to rely on a textbook. However, by contrast, teachers with lower levels of MKiT may benefit from the use of such tools, but their students benefit from this. 20 It is important to note that the regressions take into account (control for) multiple variables, including the entry achievement level of students. Second, it is important to note that higher MKiT teachers could conduct their class using certain approaches that would lead to higher learning outcomes for them, but would lead to lower learning outcomes for teachers in general. 68 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Exploring the Beliefs-Knowledge-Practices Relationship For statements such as: “Students only use methods I teach them” and “I only go through one method for doing each question”, there is also a negative relationship with levels of MKiT, but a positive relationship with student learning outcomes. Similar to the example regarding the use of textbooks, a possible explanation is that teachers with higher levels of MKiT may be comfortable with allowing their students to explore methods and engaging in discussions with students on alternative methods. However, teachers with lower levels of MKiT teachers may prefer to have students use a single method, but this could in fact reduce confusion for the students. The pattern of comfort and ability of the teacher could also apply to the statement: “I adjust my lesson plan based on results of homework assignments.” The statement: “A highly effective way to teach mathematics is to first explain the topic, give example(s) and then ask students to solve mathematical tasks” is one with which teachers with higher levels of MKiT tended to agree to a lesser extent, but with which there was a positive relationship with student learning outcomes. In this case, it may be that higher MKiT teachers tend to utilize alternative methods. In terms of actual practices, the results in Section 6.3.1 indicate that teachers with higher levels of MKiT tended to use a wider variety of practices. The more expository approach implied in this statement was used less often by teachers with higher levels of MKiT. The general theme in these “opposite direction” statement relationships is that teachers with higher levels of MKiT tended to agree to a relatively greater extent with statements that could be seen to imply more advanced techniques and approaches, but that there was not always a positive correlation with higher student learning outcomes. My interpretation of this is that while certain approaches might be seen to be more advanced or more effective, given that there is a very wide variation in the levels of MKiT amongst Indonesian teachers, teachers tending to agree with beliefs statements related to more traditional or basic techniques may be more effective in using the associated approaches or techniques. This has important implications. It may be desirable to have teachers believe in and use practices that encourage students to come up with their own solution methods; to have teachers who adjust their lesson on the basis of homework assignment results; and not to have teachers who think that telling the students the answer is an efficient way of teaching. It may also be the case that teachers with higher levels of MKiT are able to use the associated practices effectively. However, it may be that teachers with relatively lower levels of MKiT could in fact be more effective using what would be considered less advanced approaches. This implication is one of the key drivers for the model that is described in Chapter 7. Flexibility of Beliefs Explored with Standard Deviation on Responses One result derived from the analysis of the case studies earlier in this section is that teachers with lower levels of knowledge, as measured through the assessment, tended to hold a more focused or narrower set of beliefs. On the other hand, teachers with higher levels of knowledge were much more nuanced and “uncategorizable” in their responses to questions related to beliefs. The teachers with higher levels of knowledge often expressed a high degree of flexibility regarding which practices to use, with their choice of practices depending on the classroom and subject context. Could this ‘flexibility’ be detected through the survey responses? One way to determine this was to separate the teachers into higher and lower assessment score groupings and to compare the average standard deviation of responses to each Likert scale item. A higher standard deviation would indicate a greater variation in responses, while a lower standard deviation would indicate a tendency for teachers within the grouping to respond in a similar manner. Main Report 69 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms On average, the standard deviations of the teachers in the high-scoring group (1.22) were higher than was the case for teachers in the low scoring group (1.01). By using the independent group’s t-test, it can be determined that this difference is statistically significant. This result can be seen as providing at least some evidence to support the hypothesis that the beliefs of the teachers with relatively high levels of knowledge do tend to vary more, while those of the teachers with lower levels of knowledge tend to more easily fall into given categories. The regression results presented in the next section take this idea further. 6.1.2 CASE STUDY INSIGHTS ON THE BELIEFS-KNOWLEDGE RELATIONSHIP A key finding of the beliefs analysis was that while three teachers expressed belief statements that enabled the teachers to be easily categorized, all other case study teachers expressed a range of beliefs. This gave rise to the hypothesis that, rather than attempt to categorize these teachers or declare their beliefs contradictory, it was important to recognize that this expressed range is significant. It suggests that some teachers are more flexible in their views, adapting these views according to context. Of particular interest in examining the relationship between beliefs with levels of knowledge is determining whether teachers who tended to express a more focused or oriented set of beliefs differed from those tending to express a range of beliefs and who specifically discussed flexibility and adaptability in terms of their level of knowledge. Case studies CS2 and CS10 expressed relatively focused viewpoints that, when seen through the analytical beliefs lenses, were transmissionist (orientation lens), traditional (approach lens) and instructor-centric (teacher role lens) in nature. This Transmissionist-Traditional-Instructor grouping will be specified as TTI for brevity throughout the remainder of the study. The comments of CS5 also tended to be clearly orientated towards the other end of the spectrum. She tended to make statements and show preferences that were clearly Connectionist, Progressive and Facilitator-oriented (which for brevity’s sake will be referred to as CPF). However, the remaining case study teachers expressed a range of beliefs, with three teachers in particular (CS4, CS4 and CS9) specifically emphasizing flexibility. While appearing to lean towards what could be a CPF orientation, at the same time, they saw value in practices could be described as more transmissionist, traditional and instructor in nature. The contextual aspect was critical. These teachers would often emphasize the importance of context in determining how to teach and what methods to use. CS3, CS6 and CS9 openly defied the notion of a beliefs orientation, arguing that different situations call for different approaches. These teachers specifically discussed the need for flexibility in teaching. However, an important distinction in this concept of concept is that CS3 tended to consider the use of a particular approach as being more appropriate for particular sets of students, with more emphasis on TTI for students of lower achievement levels, but a greater emphasis on CPF for more advanced students. On the other hand, CS6 and CS9 advocated the use of multiple approaches for the same set of students, even jumping from approach to approach within a single lesson, depending on the topic and context. How, then, might a teacher’s orientation relate to their levels of knowledge and how does this notion of flexibility play out when comparing beliefs and knowledge? 70 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Exploring the Beliefs-Knowledge-Practices Relationship All Case Studies In Figure 27, the categorized beliefs expressed by the teachers are overlaid with the teacher subject and pedagogy assessment scores. One thing that immediately stands out is that those teachers whose responses tended to be TTI- oriented also tended to record scores at the lower end of the teacher assessment, particularly in terms of subject matter knowledge. On the other hand, teachers who tended to be more CPF-oriented also tended to have relatively high subject and pedagogy scores, but the results were mixed. The teachers who directly discussed the importance of flexibility tended to have scores at the upper end of the range. Figure 27: Scatter Plot of Teacher Subject and Pedagogy Scores on the Teacher Assessment 100% 90% Flexibility emphasis 80% 70% CS9 TTI Orientation Pedagogy Score 60% CS5 CS7 CS6 50% CS4 CS3 CS2 CS10 40% 30% CPF Orientation 20% 10% 0% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Mathematics Score Note: TTI=Transmissionist-Traditional-Instructor lens grouping and CPF=Connectionist-Progressive-Facilitator lens grouping. Case Study Conclusions However, it must be stated that the very limited number of case studies involved does not provide sufficient proof to conclude that it is teachers with lower levels of knowledge that tend to hold this beliefs perspective. Indeed, the fact that the flexible teachers appreciated and used transmissionist/traditional/instructor approaches and that some of these teachers scored among the highest levels in the assessment suggests that it may be inappropriate to draw such a clear-cut conclusion. Still, it does raise an interesting area for further exploration to determine whether there is a reason that teachers with lower levels of knowledge tend to hold a TTI viewpoint. Main Report 71 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms 6.2 THE BELIEFS-PRACTICES LINK The research community is particularly interested in PRACTICES teachers’ mathematical beliefs because of the likely influence of these beliefs on teaching practices. In Video Teaching practices used in the lessons particular, the research has tended to focus on the extent to A D which espoused beliefs are consistent with enacted beliefs. Stimulated recall How teacher describes events within the E lessons and explains decision/actions This branch of research has its roots in the work of Ernest MKiT MATHEMATICAL BELIEFS (1989), who proposed a model of espoused and enacted G Scenarios Demonstrated knowledge of students, knowledge of practices and subject knowledge beliefs, in which the two types of belief are separated by a layer Beliefs Interview Espoused beliefs through general discussion and beliefs situations B of constraints and opportunities defined by the social context F C Teacher Assessment Subject and pedagogical knowledge test scores of teaching. Results of some studies indicate consistency (such Beliefs Survey Questions covering nature of mathematics, as Thompson, 1984; Philipp, 1995) while others suggest how students learn, how to teach and teacher’s own practices a higher level of inconsistency, with some of these studies positing conjectures as to why these inconsistencies may exist (such as Butt and Raymond, 1989; Anderson, 1997; Anderson and Piazza, 1996). Lerman (2001, 2002) and Skott (2001, 2010) and a number of others have challenged the highly individual perspective underlying beliefs research, suggesting instead that researchers should take a more social viewpoint. Lerman (2001, page 36) argues that there may be a ‘family resemblance’ between beliefs across contexts, but that beliefs are qualitatively different because of these contexts. According to this approach, it makes little sense to talk about beliefs in themselves, irrespective of context, and to expect a one-to-one correspondence between espoused and enacted beliefs. Skott (2010) argues that the individualistic approach leaves little room for a more social understanding of teaching practices. He states that a more participatory view of the role of teachers in classroom practice should be adopted, recommending a shift from the emphasis on ‘beliefs’ to one on ‘patterns of participation’. This study was designed to allow for a comparison of beliefs across the full set of 104 teachers, with espoused beliefs being captured the Beliefs Survey and enacted beliefs being captured through coded video data. At this level of analysis, the many small cases in which a teacher’s espoused beliefs are not consistent with his or her actual practices would not be captured to the extent that they could be through the case study analysis. However, this overall view does provide an indicator as to whether the espoused beliefs of the teachers, as represented by their responses to belief statements, are related to the practices they used, as captured using low inference coding of the video footage. This method has a number of recognized deficiencies that will be discussed together with the analysis of the case study component. However, it still provides advantages in terms of the identification of general patterns in the relationship between beliefs-practices. As such, this analysis was approached with the intention of identifying patterns. The results presented in this subsection indicate that when examining beliefs and practices to identify patterns across the whole sample of teachers, there is a clear relationship between teacher beliefs and practices. This lends validity to the idea that understanding a teacher’s belief system can provide important insights into how the teacher may structure the lesson; what practices the teacher may use; and how the teacher may approach certain situations. However, it is certainly not the case that there is a clear, one-to-one relationship between teacher’s belief and the practices that he or she always uses. 72 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Exploring the Beliefs-Knowledge-Practices Relationship The case studies provide many insights into the dynamics and complexities involved in the selection of practices by teachers on an event-by-event basis within their classroom. As mentioned previously, some teachers demonstrated a high level of flexibility in their beliefs regarding approaches and practices, which indicates that context is an important factor in determining the selection of teaching practices. Teachers do not operate in isolation from their environment when they choose which practice to utilize. Rather, they take into account a wide range of influencing factors. 6.2.1 EXPLORING PATTERNS IN BELIEFS-PRACTICES Analytical Approach With the full sample of quantitative data, a pattern analysis approach was adopted to identify the relationships that emerged. Prior to running these regressions, the beliefs items were categorized according to the beliefs lenses (see Section 5.2). In addition, the practices were conceptualized on the basis of what would be expected to occur in relation to those beliefs. For example, using the lens of traditional-progressive beliefs, the practice category of public (whole class) interaction would be hypothesized to involve teachers making statements more towards traditional beliefs to have a larger proportion of teacher-only interaction than teachers making more progressive statement who would be hypothesized to have more teacher-student or student-only interaction. It is important to understand how these results should be interpreted. The identification of relationships involves comparing teachers’ rating of beliefs statement relative to other teachers. Similarly, it involves comparing the proportion of time teachers use various practices relative to other teachers. For example, a high proportion of teachers may strongly disagree with a particular beliefs statement. A teacher who expresses only moderate disagreement with the statement would still have a higher relative level of agreement with the statement compared to other teachers. Thus, a positive relationship in the analysis below between a beliefs statement and a specific practice does not necessarily mean that the teachers in question strongly agreed with the statement. Rather, it means that they agree with it more strongly than the average respondent. The same is true with practices. A positive relationship for a given practice simply means that teachers tend to use that practice for a greater proportion of time than do teachers. It does not necessarily mean that they use it for a greater proportion of time in an absolute sense. Emergent Links between Beliefs and Practices Two different approaches were adopted in the analysis of patterns of beliefs-practices. First, certain themes were examined to determine which teaching practices could be matched with beliefs-practices in terms of statistically significant positive or negative correlations. Second, the axis was flipped and certain practice themes were examined to see which beliefs statements emerged as having a positive or negative relationship. The regressions were run taking into account and controlling for multiple student, classroom, teacher, school, and community variables. The t-statistics were then collected to determine whether a statistically significant positive or negative relationship existed between the beliefs statement and the associated teaching practice. A green box indicates a statistically significant positive relationship, while a red box indicates a statistically significant negative relationship. Main Report 73 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms The following results are presented by themes. The first theme is discussed in detail, with each belief statement being analyzed in order to demonstrate the analytical thought process that took place for all statements. For the sake of brevity, subsequent sections do not provide the same level of detail, but rather summarize key observations. HOW TO INTERPRET THE BELIEFS-PRACTICES FIGURES IN THIS SECTION In order to present the relationships found between beliefs and practices in a summarized form, matrices were created with rows to show the beliefs statements, along with columns representing various practices. Below is shrunken view of a matrix. Interaction Types Approaches Problem and Question Types Private (group and individual) Open problem and question Public: teacher and student Private: teacher monitoring Mathematical language Qeustion and answers Non-routine problem Public: student only Public: teacher only Public (whole class) Private: group work Real world context Private: no teacher Private: seat work Routine problem Problem solving Practical work Non-problem Investigation Discussion Exposition Problem ROLE OF TEACHER The teacher is the main source of mathematical knowledge I guide students in finding their own answers to academic problems It is more effective to provide students with the information the need to know, rather than encouraging them to experiment It is the teacher’s responsibility to provide children with clear an concise solution methods for mathematical problems I function in my classroom as a learner and partner in learning with my students Teachers can create, for all children, a non-threathening environment for learning mathematics I wait for students to approach me before offering extra help The cells for each belief-practice combination are color-coded on the basis of the result of the relationship revealed through the regression analysis. Dark red indicates a negative correlation with a statistical significance at the 1% level; medium red indicates statistical significance at the 5% level; while light red indicates statistical significance at the 10% level. Similarly, dark green indicates a positive correlation with a statistical significance at the 1% level; medium green indicates statistical significance at the 5% level; and light green indicates statistical significance at the 10% level. The level of 5% is used as a threshold for meeting the criteria of statistical significance, but the level of 10% is still shown to indicate that an item was approaching statistical significance. The columns of practices are the same as were used for the analysis of the relationship between knowledge-practices in Section 6.3. 6.2.2 BELIEFS THEME: THE ROLE OF THE TEACHER Figure 28 shows the relationship between statements under the theme “role of the teacher” with the associated use of practices. Of the different themes explored, the results for the “role of the teacher” statements conformed most closely with predicted responses, possibly indicating that the association between beliefs statements regarding the role of the teacher and practices is clearer than in the case of belief statements related to how students learn or how to teach. From even a brief glance at the figure, it appears that the area with the most positive and negative relationships is in the interaction type found in the classroom, which include the first nine columns on the left. 74 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Exploring the Beliefs-Knowledge-Practices Relationship Figure 28: Matrix with Beliefs Role of the Teacher related to Actual Practices Interaction Types Approaches Problem and Question Types Exposition Investigation Practical work Problem solving Problem Non-problem Mathematical language Real world context Routine problem Non-routine problem Qeustion and answers Open problem and question Private (group and individual) Public: teacher and student Private: teacher monitoring Legend of Statistical Significance and direction Positive 1% 5% 10% Public: student only Public: teacher only Public (whole class) Private: group work Private: no teacher Negative 1% 5% 10% Private: seat work Discussion ROLE OF TEACHER The teacher is the main source of mathematical knowledge I guide students in finding their own answers to academic problems It is more effective to provide students with the information the need to know, rather than encouraging them to experiment It is the teacher’s responsibility to provide children with clear an concise solution methods for mathematical problems I function in my classroom as a learner and partner in learning with my students Teachers can create, for all children, a non-threathening environment for learning mathematics I wait for students to approach me before offering extra help For this theme, the relationships seem to generally conform to the predicted results: • “The teacher is the main source of mathematical knowledge”: Teachers who agreed to a relatively greater extent21 with this statement also tended to utilize practices involving a higher proportion of whole class time. Furthermore, their whole class time included a relatively higher proportion of teacher-only public interaction. The teachers also tended to use exposition to a relatively greater extent. The use of these practices is in line with the predicted/hypothesized direction of the relationship, as they all are associated with more traditional teaching methods, with the statement being categorized as orientated towards traditional, transmissionist teaching. Under the teacher role lens, the statement itself would fall into the instructor or explainer category, rather than the guide category. This matches the use of exposition (lecturing). The negative relationship with practical work also conforms to the predicted results, because the practice involves students learning by exploring, rather than learning derived from the knowledge of the teacher. • “I guide students in finding their own answers to academic problems”: This statement contrasts with the “main source” statement above, with the relationships tending to go in the opposite direction. There is a positive relationship between this statement and teacher-student public interaction, indicating that student interaction in whole-class situations is guided by the teacher. For individual interaction, it is also the case that the teacher tends to monitor or be involved in these situations, again possibly indicating the teacher’s guiding role. Finally, there is a positive relationship with the use of questioning, again possibly indicating a guide role. 21 In this case a teacher’s level of agreement is based on his or her Likert scale ranging from 1 to 5 for the statement and compared to other teachers. If a teacher marks 4 (agree) and the majority of teachers tend to disagree with the statement (marking 1, 2) then this teacher would have a relatively higher agreement of the statement. If, on the other hand, the majority of teachers strongly agree (marking 5) then that same response would be relatively low relative to other teachers. Main Report 75 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms • “It is more effective to provide students with the information they need to know, rather than encouraging them to experiment”: There is a positive relationship between the level of agreement with this statement and the extent to which teachers relied upon teacher-only public interaction, with a negative relationship with student-only interaction. In private interaction, the teachers also tended to monitor the group and seatwork to a relatively greater extent. Again, this is possibly related to the perception that a higher level of teacher involvement is a more effective approach. • “It is the teacher’s responsibility to provide children with clear and concise solution methods for mathematical problems”: In the case of this statement, a somewhat surprising result was recorded, in that teachers who expressed agreement to a relatively higher extent also tended to use a relatively lower level of public interaction and a higher level of private (group and seatwork) interaction. There was also a negative relationship with the level of teacher involvement in private interaction. However, there is a positive relationship with the extent to which teachers utilized the Discussion and Exposition approaches, and a negative relationship with the use of Practical Work is negative. These results conform to the initial hypothesis. The teachers also tended to use Mathematical Language to a relatively greater extent. • “I function in my classroom as a learner and partner in learning with my students”: In the case of this statement, there was a positive relationship between the level of belief with the extent to which teachers utilized student-only public interaction, and a negative relationship with the extent to which they utilized teacher-only interaction, as was predicted. The approach of Discussion may also be considered to fit the learner and partner in learning approach. In this case, one unexpected result was that such teachers were expected to use questioning to a relatively greater degree, but this result did not occur to a statistically significant level. 6.2.3 BELIEFS THEME: TOOLS AND RESOURCES, MANAGEMENT AND EVALUATION The analysis revealed multiple relationships in terms of Tools and Resources, particularly in terms of interaction practices. Teachers who tended to agree to a relatively greater extent with the statement that textbooks are the main resource for the teaching of mathematics also utilized whole class teaching practices to a relatively greater extent. Teachers who agreed that it is important to cover all the topics in the mathematics curriculum and in the sequence prescribed by the textbook also used whole class teaching practices to a relatively greater extent, and the nature of the public interaction tending to be more teacher-oriented. There was also a tendency towards a greater use of mathematical language. In the area of management and evaluation, teachers who tended to agree to a relatively greater extent with the statement that: “Paper-pencil testing is the best way to examine students’ mathematical learning” also tended to dedicate more time to problem-solving and to private time involving seatwork rather than group work. Together, these statements appear to fit the belief system of teachers who would tend to place an emphasis on testing. An analysis of the responses to the statement: “It is important that I establish classroom control before I become too friendly with students” also reveals a number of significant relationships. Focusing on the teacher-student interaction portion, in the case of public interaction, teachers tended to utilize a higher proportion of student-only time and for private interaction, the teacher tended to be less involved in monitoring activities. The relationship between teacher-student interaction practices with the level of both public and private interaction is negative, which could relate to these teachers tending to not want the teacher-student relationship to remain distant. 76 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Exploring the Beliefs-Knowledge-Practices Relationship The grouping of Curriculum Assessment and Policy statements stands out for revealing relatively few statistically significant relationships. Figure 29: Matrix with Beliefs on Teaching Approach, Assessment, and Resources Related to Actual Practices Interaction Types Approaches Problem and Question Types Exposition Investigation Practical work Problem solving Problem Non-problem Mathematical language Real world context Routine problem Non-routine problem Qeustion and answers Open problem and question Private (group and individual) Public: teacher and student Private: teacher monitoring Legend of Statistical Significance and direction Positive 1% 5% 10% Public: student only Public: teacher only Public (whole class) Private: group work Private: no teacher Private: seat work Negative 1% 5% 10% Discussion MANAGEMENT AND EVALUATION I base student grades primaliy on homework, quizes, and tests I adjust my lesson plan based on results of homework assignments I prefer to assess students informally through observation and conferences It is important that I establish classroom control before I become too friendly with students Paper pencil test is the best way to examine students’ mathematical learning I prefer to cluster students’ desks or use tables so they can work together TOOLS AND RESOURCES Textbooks are the main resource to teach mathematics It is important to cover all the topics in the mathematics curriculum in the textbook sequence To be sure that I teach students all necessary content and skills, I follow a textbook or workbook I generally use the teacher’s guide to lead class dicussions of a story or text I find that textbooks and other published materials are the best sources for creating my curriculum I encourage parents to follow up on classroom activities with students at home CURRICULUM, ASSESSMENT AND POLICY The examination syllabus should not be the main factor in determining the school syllabus Teachers should be able to influence public examination boards in the style of examination, syllabus, content, etc Public examination syllabus, style, etc. should be decided by experts Commerce and industry’s needs should determine the mathematics curriculum The testing program in my state’district dictates what mathematics content I teach I believe that expanding on students’s ideas is an effective way to build my curriculum Main Report 77 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms 6.2.4 BELIEFS THEME: PRACTICES FOR AND PROCESSES OF LEARNING In general, in the case of themes of Practices and Processes of Learning, the analysis revealed fewer statistically significant relationships than was the case in the previous sections. In a sense, this is surprising, as this area would seem to be more directly related to practices than the previous themes. Figure 30: Matrix with Beliefs on Learning Related to Actual Practices Interaction Types Approaches Problem and Question Types Private (group and individual) Open problem and question Public: teacher and student Private: teacher monitoring Legend of Statistical Significance and direction Mathematical language Qeustion and answers Positive 1% 5% 10% Non-routine problem Public: student only Public: teacher only Public (whole class) Private: group work Real world context Private: no teacher Negative 1% 5% 10% Private: seat work Routine problem Problem solving Practical work Non-problem Investigation Discussion Exposition Problem PRACTICES FOR LEARNING Students learn mathematics best in classes with students of similar abilities Listening carefully to the teacher explain a mathematics lesson is the most effective way to learn mathematics Expecting pupils to be creative in mathematics is unreasonable and doesn’t warrant the use of precious lesson time Mathematics is learnt well by working individually on mathematical problems The more students practice mathematical exercises given in the textbooks, they learn mathematics with understanding PROCESS OF LEARNING Allowing a child to struggle with a mathematical problem, even a little tensions, can be necessary for learning to occur It is important that mathematics content be presented to children in the correct sequence Mathematical material is best presented in an expository style: demonstrating, explaining and describing concepts and skills Learning mathematics in school is essentially an active process Students learn mathematics by finding the correct answer to mathematical problems Student can solve most mathematical problems on the basis of memorized formula Use of calculator hinders students’ understanding of mathematics Children always benefit by discussing their solutions to mathematical problems with each other Persistent questioning has a significant effect on children’s mathematical learning The statement: “Learning mathematics in school is essentially an active process” fits well in terms of interaction results, with the proportion of public time tending to have more teacher and student interaction and private interaction tending to involve more group work than seat work. Questioning, though, is less and this seems to go against what would be predicted for a teacher with an active process perspective. 78 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Exploring the Beliefs-Knowledge-Practices Relationship The results of the analysis for the statement: “Listening carefully to the teacher explain a mathematics lesson is the most effective way to learn mathematics” is also in line with the expected interaction, with teachers expressing a relatively high level of agreement with the statement tending to engage in public interaction to a relatively greater extent, with the interaction tending to have a higher proportion of teacher-only time and less student-only time. However, while it was hypothesized that there would be a positive relationship between this statement and an expository approach, while the t-statistic is positive, the relationship did not reach the level of statistical significance. In the case of the statement: “Expecting pupils to be creative in mathematics is unreasonable and does not warrant the use of precious lesson time”, the relationship went in the opposite direction to that expected, with teachers expressing agreement with the statement tending to dedicate a relatively higher proportion of time to student- teacher pubic interaction, with less teacher-only interaction. It also is unexpected that these teachers’ practice involved a higher proportion of practical work, which could possibly include exploratory or creative activities. General Comments on the Pattern Analysis The pattern analysis, which revealed both expected and unexpected relationships, can be somewhat overwhelming when examined item by item. However, when considering the picture from a general perspective, the results mostly conformed to the initial hypotheses, showing that many aspects of teachers’ beliefs play a role in defining their choice of teaching practices. As was concluded after the initial beliefs analysis, it can be seen that beliefs are complex and a Likert scale survey is unable to capture the complex manner in which teachers’ beliefs interact with their practices. Nonetheless, certain aspects appeared to provide insights into the interaction. In particular, responses to the items under the theme “Role of the Teacher” tended to be particularly closely linked with interaction types. It may be that this aspect of teachers’ beliefs clearly translates into how the teacher chooses to interact with students. Looking at the Approaches grouping of practices, it was frequently found that an expositional approach was associated with belief statements that indicated a belief in teacher-centered, rather than student-centered, approaches to learning. The responses to belief statements related to Problem and Question Types tended to have a lower level of association with practices, although the teachers’ use of questioning tended to be associated with beliefs involving a more student- centered approach. 6.2.5 CASE STUDY TEACHER INSIGHTS ON BELIEFS-PRACTICES Case Study Teachers With the case study teachers, the relationship between beliefs and practices was first analyzed in broad, general terms across all of teachers’ videotaped lessons. This was undertaken to determine whether any teacher’s actual overall approach to teaching might be considered to be noticeably different from the results of the beliefs analysis presented in Section 5. This was followed by an examination of specific events within the lessons. The inputs for this included not only observation of the video but also, critically, stimulated recall interviews in which the teachers explained their choice of actions and methods. Main Report 79 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms The Broad Sweep A broad sweep was conducted across all lessons of each case study teacher, with consideration given to the practices used in relation to the statements they made during the beliefs interview and their responses to the beliefs survey. There were no cases when a teacher’s overall approach was significantly different from his or her stated beliefs. Thus, there were no cases in which a teacher’s statements indicated a certain orientation, but they generally22 taught in a completely different way. The practices are utilized and the environment created in the classroom typically conformed to the results and conclusions of the beliefs analysis. Teachers CS2 and CS10, who tended to make statements indicating an orientation towards transmissionist or traditional teaching methods, generally used a relatively more teacher-centered approach than those whose statements indicated an orientation towards a more connectionist or progressive approach. These teachers tended to use an expositional approach, with the proportion of ‘student and teacher’ and ‘student only’ interaction for public (whole class) time being relatively lower than with the other case study teachers. These teachers took on the role of instructor relatively more frequently than they took on the role of guide or explainer. When these teachers utilized questioning in the classroom, the questions tended to be control and testing type questions rather than genuine or provoking type questions. Teachers CS4, CS5, CS6 and CS9, who tended to make statements indicating that they saw their role as being a guide to the students, also tended to utilize genuine and provoking questions to a relatively greater extent than the more instructor-oriented teachers, who tended to use control and testing-oriented questions of a greater extent. The resulting classroom environment was more student-led, with less teacher-led exposition and with a greater proportion of student-led investigative and problem solving activities. This involved a higher proportion of ‘student and teacher’ and ‘student only’ public interaction time. One general difference between beliefs and practices was that there was a tendency for teachers to utilize traditional teaching approaches to a greater extent than might be implied by their beliefs statements. Some teachers tended to stress beliefs regarding levels of student involvement and the use of non-routine or open approaches, but in practice these aspects were less evident. This point is explored in greater detail after discussing individual instances. Beliefs-Practices “Differences” in Individual Teaching Instances and Teacher Explanations When examining specific instances within the lessons, many interesting interactions and incidents occurred that helped generate insights into the relationship between beliefs-practices and the complexity of this relationship. The stimulated recall interviews played a particularly important role in the comparison of beliefs- practices, because they provided insights into teachers’ reasons for the adoption of the approach taken in specific classroom events. Often in research literature, differences between espoused and enacted beliefs are seen as a conflict. In a sense, this word captures the fact that teachers do not choose their practices in isolation. Rather, at times, the environment and context influences their choice of practices and these practices may not match the teacher’s idealized general conception of the teaching-learning process. However, in discussing the findings of this research, the term ‘conflict’ has a misleading connotation. 22 This stage is taking the teaching practices as a whole rather than on an instance-by-instance basis. 80 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Exploring the Beliefs-Knowledge-Practices Relationship The differences that emerged can be grouped according to five themes, as specified below: (a) Alternating amongst Approaches In examining approaches on an instance-by-instance basis, the validity of the initial hypothesis when defining the theoretical framework became clear: All teachers adopted various approaches and no teacher’s actions necessarily conformed to a single category of beliefs 100% of the time. In a sense, this is somewhat obvious. For example, it would have been unrealistic to expect teachers who made statements indicating that they saw their role as being a ‘guide for their students’ to never undertake activities that were instructional or explanatory in nature (the other two categories under the teacher role lens). Many of the case study teachers stated that in planning the learning process for a topic, which could extend over a number of lessons, the steps within the learning process required different levels of student involvement and different teaching approaches. CS9 made an interesting statement, saying that she felt it was important to alternate amongst approaches and that this was more beneficial than always following a single method or pattern of teaching. By varying practices by, for example, sometimes group work, sometimes individual seatwork, students could be exposed to different ways of thinking and approaching mathematics and mathematical problems. (b) Matching Approaches with Topic or Situation Some teachers spoke of tailoring their approaches to suit the topic. For example, CS1 explained that her game of creating various forms of nets of a cube was not necessarily something that matched her usual style of teaching, but that she had got the idea from another teacher and felt that the topic required the use of a visual stimulus, which lent itself well to a game using group work. However, she regarded this as an exception to a general rule. On the other hand, some other case study teachers were very conscious of adjusting their practices according to the situation. The manner in which CS9 adopted alternating approaches has already been discussed. CS4 also spoke of how he would reflect upon the topic to be taught and consider various ways to approach the topic. While the majority of cases when teachers attempted to match a topic with situation emerged through the stimulated recall sessions of their own lessons, many instances also emerged through the Mathematical Scenarios interview. Some teachers would speak specifically about how the given topic would be best approached through a certain type of practice. In other cases, they wanted to know the context to the situation before selecting an approach. (c) Choice Due to Perceived Environmental Factors All case study teachers pointed to various factors that played a role in their determination of how and what to teach. The curriculum dictated or influenced not only which topics to cover, but also the amount of time teachers could spend on the topic, which in turn influenced the approach they adopted to cover that topic. When directly asked about the national assessment and whether it played a role in the way they teach, all teachers agreed that it was a factor, but to varying extents. CS5 noted that her class was advanced and that they were covering topics beyond the level prescribed by the curriculum and the national assessment process. CS3 was at the opposite end of the scale, stating that his class had a low entry achievement level and that he needed to focus on fundamental issues that were not theoretically part of the curriculum and national assessment process. Some teachers noted that Main Report 81 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms the national assessment has become more influential in recent years, following the practice of publishing schools’ results. Ragatz (2013) noted an emerging trend between two points in time, 2007 and 2011, with practices possibly changing as a result of the influence of the national assessment. Over this period, the proportion of teachers who said they were influenced by the national assessment increased from 70% to 75%. There was also a trend towards changes in the degree to which various practices were utilized. There was a decrease in the degree to which teachers used non-routine approaches to problems and to which they used open problems and questioning. In terms of teaching approaches, there was a decrease in investigative and practical work and an increase in expositional and problem solving. There are many factors that go towards explaining teachers’ choice of practices, but the changes could possibly be interpreted as methods oriented toward “teaching to the test” and that the trends may in part be due to teachers adjusting their practices based on the national assessment. In the stimulated recall interviews, teachers frequently explained their choices of practices in the context of their students’ achievement levels and needs. CS3 was particularly intriguing in terms of how he expressed his beliefs in relation to teaching practices and justifying the practices used in the observed classroom. He specifically pointed out that he would tailor his teaching to his students’ ability levels. When teaching more advanced students, as he did in previous assignments at other schools, and when tutoring, he would use approaches considered more progressive, including student-led activities, the use of open problems and non-routine approaches. He stated that in these contexts, he would play the role more of a guide than of an instructor. For the class observed in this research, the teacher repeatedly mentioned his students’ low levels of achievement. In this situation, he felt it was more appropriate to use more traditional approaches. He felt he needed to direct and control the lesson to a greater extent and could not conduct activities in which the students took on creative tasks or were asked to complete unstructured tasks. He spoke of his preference for individual seat work rather than group work with this class, because he often felt that the group work would be unproductive. The comments of CS3 bring up multiple interesting points regarding the relationship between beliefs-practices. It is certainly debatable whether this ‘tailoring’ is beneficial or whether the fact that the teacher chose not to use what he generally believed to be better practices might limit the students’ opportunities to learn (Clarke, 2011). However, for the purpose of comparing beliefs-practices, what is of interest is the teacher’s perception that his generally preferred method of teaching was not applicable for his students. Does this mean he did not “believe” in the methods that he actually used? However, while it might seem that he did not believe that these were ideal methods, he did appear to believe that these alternative methods were appropriate when applied in the context of his classroom situation. One way to interpret this is that CS3 has a range of beliefs that defied categorization within specific orientation. Rather, his practices have an accompanying range. Rather than thinking of beliefs as a singular point within a beliefs map, his beliefs imply a range across the map, with the context determining which practices he chose to apply. (d) Differing conceptions and interpretations of beliefs It became clear through the interviews that the abstract and complex nature of beliefs makes them particularly difficult to pin down to a specific approach. The interpretation of certain beliefs can differ from teacher to teacher. Indeed, while the definition and interpretation used in this study may be founded through a review of the literature, it is simply another interpretation. 82 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Exploring the Beliefs-Knowledge-Practices Relationship Atweh and Abadi (2012) noted that multiple curriculum reforms in Indonesia have played an important role in influencing teachers’ beliefs, often in ways that were uneven and unintended. They attributed differing conceptions of pedagogical practices and teaching approaches to the inconsistent dissemination of the curricula. In the case of the Student Active Learning Curriculum of the mid-eighties, which placed a heavy focus on student-centered learning, there was only a limited provision of professional development and teaching guides. They noted that: “While teaching materials in terms of activities and posters crowded the classrooms, teachers’ lack of clarity about their expected roles lead to the overuse of homework and activities leading to concerns by parents and educators about the decline of actual teaching and learning in schools” (page 2). Teachers such as CS2 and CS10 indicated that they believed the teachers should act as a guide or facilitator, yet they tended to use more traditional, teacher-centered practices that fit with a more transmissionist approach. However, in discussions with CS2, it was clear that she interpreted the terms “guide” and “facilitator” in line with her transmissionist teaching approach. There were also instances within her lesson where students conducted group work and she did show aspects more related to a facilitating role, although it still had elements of a transmissionist approach. The concepts of student-centered learning, active learning and student participation are also good examples of how definitions can vary. For example, some teachers pointed to choral responses of students as examples of student-centered learning, active learning and student participation. (e) Belief in Practices, but Struggling to Implement The approach of CS7 in the beliefs-practices context contrasts somewhat with the example of CS3. It was her first year teaching at her school, and she noted that she was still in the process of determining the best way to teach her students. She strongly believed that her students should be active participants in the learning process. However, the school was in a poor, rural area and the teacher described the community culture for education as one in which the students were not accustomed to speaking in the classroom. The video lessons provided multiple examples in which the teacher attempted to prompt the students to answer questions, but struggled to get students to respond. She would change the question wording, ask the class as a whole, the thing specific students and using other tactics. However, she often failed to obtain satisfactory responses and at times became visibly frustrated.23 In the stimulated recall interviews, she explained that she got frustrated because she felt she had not figured out “how to inspire the students”. This example brings up many interesting points. First, it emphasizes the fact that beliefs can be held, but, when evoked through practices, those practices operate within the classroom environment. Beliefs and practices are not static, but rather dynamically evolve over time and are likely to play off one another, each shaping the other to some extent. The teacher was new at the school and it may be that over time, the students could adjust to the practice and become more participatory, as the teacher was hoping. Alternatively, it could be that over time, the teacher adjusts her practice, by refraining from attempting to get students to participate to the extent she initially hoped. This is conjecture only, but it does raise an interesting area for exploration into how the beliefs-practices interaction may evolve over time, based on the classroom environment. 23 The teacher was asked in the Stimulated Recall interviews if she thought the cameras had an effect on the students. She said no, this was the way they typically responded to her attempts to get them to participate. Main Report 83 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms 6.2.6 KEY CONCLUDING POINTS ON BELIEFS-PRACTICES In this analysis, there has been a simultaneous tearing down and building up of the concept of the relationship between beliefs-practices. The general patterns from both the case studies and the full-sample analysis indicate that there is a relationship between beliefs and practices, with a general consistency in terms of the link between certain practices with mathematical beliefs conforming to expectations. At the same time, certain hypothesized relationships in the full-sample analysis did not always emerge as being statistically significant. Although rare, in some instances, the relationship flowed in a direction opposite from that which had been expected. In part, this reflects the complexity of beliefs, indicating that no simple relationship exists: a teacher’s level of agreement with a statement does not automatically indicate what practices they would use in a particular situation. The case studies provided a deeper understanding by enabling teachers to explain how they selected their practices and how their beliefs may have directly or indirectly played a role in the process. As has been discussed in other research on beliefs, teachers must take into account a wide range of environmental and situation-specific contextual factors when selecting their teaching practices, both in preparation of the lesson and in on-the-fly decision making. Most teachers indicated that they operated in an environment that generally meshed well with their beliefs (CS2, CS4, CS5, CS6, CS9), but all teachers pointed to factors that influenced their teaching in ways that they might not otherwise have chosen. Other teachers (CS3, CS7) operated in environments that generated elements of greater tension (real or simply perceived) with their beliefs. These teachers noted that these factors created particular challenges for them in the use their preferred practices and approaches. This concept of operating outside of their preferred set of beliefs or even having at time to use practices that fall outside of their beliefs range is an important finding that will play a key role in the model proposed in Section 7. Another important concept that emerged was that some teachers had “flexible” beliefs, in the sense that they openly recognized the value of practices that implied a range of sometimes contradictory beliefs. In this study, the higher performing teachers were generally those without a strong orientation, in the sense that they saw value in various techniques and would switch as they deemed appropriate. So, what are beliefs when viewed in relation to practices? The definition may be different from simply viewing beliefs at a theoretical level or in isolation from the classroom. The results in this section indicate that teachers might have an orientation or preference that can be captured to a certain extent through their relative level of agreement with statements and through an analysis of their explanations regarding their philosophy of teaching, learning and the nature of mathematics and of their justifications and explanations for the choices they made when teaching. Their expressions of beliefs did generally reflect their teaching practices, but their beliefs may shift or adapt depending on the environmental context. In fact, the results indicate that, far from it being a negative attribute, this flexibility and adaptability may prove to be a strength for many of the teachers, playing a role in their teaching effectiveness. 84 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Exploring the Beliefs-Knowledge-Practices Relationship 6.3 THE KNOWLEDGE-PRACTICES LINK Ideally, the link between knowledge-practices would PRACTICES involve identifying instances within lessons where the Video Teaching practices used in the lessons teachers invoke different aspects of their knowledge, A D because regardless of the extent of a teacher’s knowledge in Stimulated recall How teacher describes events within the E lessons and explains decision/actions subject and theory, it is the actual invocation of knowledge MKiT MATHEMATICAL BELIEFS in the classroom that actually makes a difference in the G Scenarios Demonstrated knowledge of students, knowledge of practices and subject knowledge teaching-learning process. A number of ambitious research Beliefs Interview Espoused beliefs through general discussion and beliefs situations B studies have attempted to identify this link, with varying levels F C Teacher Assessment Subject and pedagogical knowledge test scores of success. Some of the most prominent initiatives have come Beliefs Survey Questions covering nature of mathematics, from the work of Hill, Rowan and Ball (2005), who examined how students learn, how to teach and teacher’s own practices the predictive validity of their level of knowledge competence by using the item response theory score to predict elementary students’ learning outcomes. They drew on a sample of schools participating in three comprehensive school reform programs and a matched group of control schools. Multilevel analyses showed that elementary teachers’ level of mathematical knowledge for teaching indeed correlated with students’ learning outcomes in two different grades. In fact, the effect was practically linear. This study produced the first conclusive evidence for the practical importance of teachers’ mathematical knowledge in terms of both the mathematical knowledge that adults use in everyday life and the specialized knowledge that teachers use in classrooms. The work of Hill, Rowan and Ball (2005) was considered a ground-breaking quantitative study in its unique approach to finding that teachers’ mathematical knowledge was significantly related to levels of student achievement. This result, while consistent with findings from the educational production function literature, was obtained through a measure focusing on the specialized mathematical knowledge and skills used in teaching mathematics. However, it had many limitations and could not distinguish between their theorized domains of knowledge. A grounded approach to data analysis by Rowland, Thwaites and Huckstep (2005) involving the use teacher videos led to the identification of a ‘knowledge quartet’, with four broad dimensions, or ‘units’, through which mathematics-related knowledge of the teachers could be observed in practice. These four dimensions were defined as: foundation, transformation, connection and contingency. While a methodical coding of the videos in the context of MKiT was not undertaken, the stimulated recall interviews brought out many insights in the teachers’ levels of MKiT when discussing specific instances in the lesson. The scenarios interview, while not involving actual interviews, did provide simulated situations that could enable a comparison of teachers. The detailed aspects of the scenarios analysis can be found in the Technical Report, but specific aspects of the knowledge-practices relationship are summarized here. Main Report 85 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms 6.3.1 LINKING PRACTICES WITH MKIT ASSESSMENT SCORES The results of this study and of those of a large body of similar studies indicate that there is a strong positive relationship between levels of teacher knowledge as measured through the teacher MKiT assessment and the learning outcomes of students.24 Teachers do not directly pass along their knowledge to students: rather, it is passed along through actions, through their teaching practices in the classroom.25 Therefore, it is important to understand how teacher knowledge manifests itself in the classroom and how teaching practices differ according to teachers’ various levels of MKiT. The influence of knowledge on practices can manifest itself in three possible ways. First, it is possible that teachers with higher levels of MKiT could tend to use different practices than those with lower levels. Under this scenario, the teachers with higher levels of MKiT might possibly have a larger set of teaching practice “tools” and that these additional tools are beneficial for student learning. Second, teachers with higher levels of MKiT might tend to use certain teaching practices more or less frequently, based on their knowledge. With this scenario, it may indicate that the exposure of students to certain practices could result in better or worse learning outcomes. Third, it is possible that teachers use the same approach, but that their execution of that approach varies according to their level of knowledge. Under this scenario, students may achieve better outcomes because a teacher with a higher level of knowledge may tend to use a particular practice or practices more effectively. The analysis explores these different possibilities as a means to explain how learning outcomes tend to be better for students who have teachers with higher levels of knowledge. As this section discusses, the results at both the case study level and from the full study level indicate that to some extent, differences exist in terms of the types and frequency of practices used, with teachers with higher levels of knowledge tending to use a wider range of practices and tending to use certain practices more frequently. Still, these differences are by no means dramatic. In fact, differences in the use of teaching practices are much more distinct when comparing teachers with a mathematics education degree to teachers with a pure mathematics degrees, or when comparing teachers with varying levels of experience. This seems to indicate that other factors are at play in the teaching practices of teachers with higher levels of knowledge. When examining specific practices in detail, including teachers’ use of questioning, the results indicate that knowledge also manifests itself in the approach to practices, with higher student learning outcomes likely to result in part from teachers with higher levels of knowledge tending to use certain practices more effectively. 6.3.2 USE OF PRACTICES IN RELATION TO KNOWLEDGE INDICATORS Do teachers with higher levels of knowledge, as measured through the teacher MKiT assessment, teach any differently? The strong positive relationship that was found between levels of both subject matter knowledge and pedagogical teacher knowledge, as measured through the MKiT assessment, and student learning outcomes, as measured through tests at the beginning and end of the eighth grade year, indicate that teachers with higher levels 24 Multiple factors play an important role in student learning beyond the teacher. This result is after controlling for various student, home, classroom, school, teacher background and community characteristics in order to isolate and capture the relationship of teacher knowledge and student learning as accurately as possible with the given data. The teacher may work with students outside of the classroom, but teacher tutoring is a variable that is controlled for and it is assumed that 25 most of the teacher influence in the learning process takes place in the classroom. 86 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Exploring the Beliefs-Knowledge-Practices Relationship of MKiT tended to teach more effectively.26 This would imply that somehow, teachers with higher levels of MKiT are doing something different in the classroom. In terms of teaching practices, the difference could come about through: (i) using different practices; or (ii) using the same practices, but using them more effectively. The use of certain practices and the frequency of these practices coded through the Video Study enable an exploration as to whether teachers with different levels of MKiT use different practices than do teachers with lower levels. As Figure 31 shows, a few differences do emerge when comparing teacher knowledge scores to the measured teaching practices. Teachers with higher MKiT scores tended to use a higher proportion of Mathematical language and symbols relative to of Real world contexts. They also tended to use questioning techniques more often. In terms of teaching approaches, they tended to use Investigation to a greater extent. In the case of all other teaching practices, there were no statistically significant correspondences. While the teaching practices identified as having a statistically significant correlation are of interest, it may be even more interesting that relatively few such practices were identified as being statistically significant when compared to other teacher characteristics, such as teacher degree type. Since teacher knowledge has a strong positive relationship with a large effect size, this lack of differences would imply the impact on student learning might in large part be due to teachers with higher levels of knowledge being able to using the same practices in a more effective manner. This proposition will be explored later in the chapter. Figure 31: Teaching Practices of Teachers in Relation to their Overall MKiT Score Interaction Types Approaches Problem and Question Types Exposition Investigation Practical work Problem solving Problem Non-problem Mathematical language Real world context Routine problem Non-routine problem Qeustion and answers Open problem and question Private (group and individual) Public: teacher and student Private: teacher monitoring Legend of Statistical Significance and direction Positive 1% 5% 10% Public: student only Public: teacher only Public (whole class) Private: group work Private: no teacher Negative 1% 5% 10% Private: seat work Discussion MKiT Assessment Score Subject Matter Section Pedagogy Section Mathematics Education Degree Subject Score Do teachers with higher subject knowledge use the measured practices more or less frequently? While a comparison of teachers’ overall MKiT scores with the teaching practices they adopted did not reveal many correlations, it is possible that examining the subject and pedagogy scores separately could reveal additional insights. Subject matter knowledge is also shown in Figure 31. While little changed, it does appear that teachers with higher subject scores tended to use more Non-routine and Open approaches to problem solving. The fact that these relationships emerge specifically in relation to the teachers’ level of subject knowledge may indicate that as teachers have more knowledge, which may include aspects such as ability, tools and confidence, they may know how to use more complex or less standard approaches, such as Non-routine and Open problems. 26 This result cannot be conclusively said to be causal, but there are strong theoretical reasons to believe that the relationship is a result of teachers with higher MKiT teaching in a way that led to better learning outcomes. Main Report 87 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms Pedagogy Score Do teachers with higher levels of pedagogical knowledge use the measured practices more or less frequently? When looking at the relationship between practices with teachers’ scores on the pedagogy portion of the assessment, two new findings emerge. First, teachers with higher pedagogical scores tended to use Exposition to a relatively lesser extent than did teachers with lower scores. Indonesia’s Ministry of Education and its teacher training colleges generally tend to discourage the use of teacher-oriented approaches in favor of more student- oriented, at least in recent years. Teachers with higher pedagogy scores may have been exposed to a greater level of training, which may explain the result. Second, the teachers with higher pedagogy scores also tended to be involved to a greater extent when private interaction (group or seatwork) took place. This activity would typically involve the monitoring of students during an activity and could involve asking questions and guiding students. It seems reasonable that teachers with higher pedagogy scores would see value in these practices. Degrees in Mathematics Education versus degrees in Pure Mathematics Do teachers with a mathematics education degree use the measured practices more or less frequently? When comparing teachers with mathematics education degrees to those with degrees in pure mathematics, the differences in the practices adopted by the two groups are more striking than when comparing teachers with varying levels of MKiT. While the results for some aspects are similar (Questioning, Open and Investigation), there are differences in approach, with a relatively much lower use of Exposition, and a relatively higher use of Discussion and Problem Solving by teachers with mathematics education degrees. Interestingly, even more differences were noted in terms of interaction. Mathematics education degree teachers tended to use Private interaction more frequently and, as with the teachers with higher pedagogy scores, these teachers tended to monitor private interaction more often. Surprisingly, for public interaction there was a tendency for teachers with mathematics education degrees to apply proportionally more Teacher-only time and less Teacher and student time. This is somewhat surprising given Indonesia’s pedagogical push for teachers to conduct more student-centered activities. Summary of Practices Use In examining the relationship between practices and MKiT assessment scores, a few relationships did emerge, but it is the lack of a large number of relationships that is of most interest. This finding suggests that the positive relationship between levels of teacher knowledge and student learning outcomes may not be due so much to teachers using different practices in terms of type or proportion, but rather to the fact that when they use similar practices, they tend to use them more effectively. 88 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Exploring the Beliefs-Knowledge-Practices Relationship 6.3.3 CASE STUDY RELATIONSHIPS Teachers with higher levels of knowledge tended to use questioning more effectively, both in terms of the types of questions they asked and in terms of integrating questioning into their lesson approach. They used genuine and provoking questions to a greater extent, while teachers with lower levels of knowledge tended to use questions to a lesser extent, with these questions tending to be testing and closed type questions. This suggests that varying levels of knowledge are a factor in teachers using similar practices more or less effectively. Knowledge was also generally demonstrated through discussions regarding teaching practices during the stimulated recall interviews. Teachers with higher levels of knowledge tended to be more comfortable discussing practices, with their explanations demonstrating a more focused and intentioned uses of the practices. 6.4 KEY CONCLUDING POINTS ON RELATIONSHIPS An important contextual basis for the beliefs-practices analysis was that the students of teachers who scored higher on the subject matter and pedagogical assessment tended to have higher learning outcomes. Although the relationship cannot be declared causal, there are certainly theories as to how a teacher’s level of knowledge would likely play a role in their effectiveness. In turn, this would have a causal relationship with student learning outcomes. This could in part be due to: (i) teachers utilizing certain practices with greater frequency; (ii) teachers using different practices; or (iii) teachers using the same practices, but utilizing them more effectively. Investigating these three possibilities, it became clear that while there were signs of (i) and (ii), these factors did not tell full story. Rather, it became clear that (iii) was also an important contributing factor. The practices used by teachers with higher levels of subject knowledge did differ from those with lower knowledge, but the differences tended to be less significant than other aspects, such as teacher degree type, which is taken to be another indicator of contrasting pedagogical vs. subject knowledge. In addition, other contextual factors, such as the teacher’s years of experience, also played a role. This relative lack of difference in the teaching practices adopted by teachers with varying levels of knowledge despite the relatively strong relationship with student learning outcomes suggests that teacher knowledge manifests itself not only through differences in the practices used, but in terms of how effectively these practices are used. Two teachers with varying levels of knowledge might use a similar set of practices, in terms both of the selection of practices and the proportional time devoted to these practices, but the teacher with a higher level of knowledge is able to use those practices more effectively. In the case of the use of questioning, teachers with higher levels of knowledge used many more genuine, prompting and directing questions than did the teachers with lower levels of knowledge. They appeared to use questioning as an effective tool for stimulating student engagement and thought. Lower knowledge teachers tended to use questions more for the purposes of controlling and testing only. Main Report 89 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms The distinction between teachers with varying levels of knowledge also arose in relation to practices in the mathematical scenarios interview. Teachers with higher levels of knowledge tended to more quickly and accurately assess the situation, also developing a coherent plan regarding what practices they would utilize to address the situation. Finally, as became clear through the stimulated recall and mathematical scenarios interviews, the teachers with higher levels of knowledge also tended to have a larger toolbox of practices upon which they could draw to address different situations. Thus, there were able to select from a wide range of tools to choose what they thought was the most appropriate tool for the given situation. When examining beliefs in relation to levels of knowledge, it became clear that teachers with lower assessment scores tended to exhibit an orientation towards a set of beliefs more readily than teachers with higher assessment scores. Based on interviews with the teachers, it appeared that this expression of a range of beliefs indicates a more flexible view of teaching, with a teacher with higher levels of subject and pedagogical knowledge being more likely to not see teaching approaches as black and white, but rather in shades of grey. If knowledge is seen as a set of tools that the teacher can draw upon, then a higher level of knowledge may allow the teacher to draw upon a larger set of tools and therefore to determine the appropriate approach for any given situation. This finding regarding the flexibility of beliefs does not require a rejection of the use of categories. Categories can be helpful in conceptualizing the various ways in which teachers might perceive important aspects of the teaching-learning process. Further, the fact that in this research, many statistically significant relationships were identified between teaching practices and mathematical beliefs (Section 6.2) indicates that there is a relationship between teachers’ perceptions regarding teaching and learning and their practices. According to the hypothesis proposed in the analytical framework, these perceptions play a key role in determining the practices used.27 However, these findings do call into question the degree to which a teacher’s mathematical beliefs should be seen in “either-or” terms. The flexible view often expressed by the higher knowledge teachers indicates that teachers can hold a range of viewpoints that cut across categories and that, rather than being detrimental, this flexibility may in fact assist the teacher in being more effective by: (i) enabling the teacher to adapt to a particular situation depending on what the teacher sees as the educational needs of the students; and/or (b) enabling the teacher to mix practices within and across lessons, which could provide the students a more diverse set of experiences and perspectives in thinking about mathematics. 27 Correlations between beliefs and practices do not prove causation, but the theoretical model posits that a teacher’s beliefs play a role in determining the practices the teacher chooses and it is not logical for the relationship to go in the opposite direction where a teacher’s practices determine her beliefs. 90 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning CHAPTER 7 A NEW CONCEPTUAL MODEL OF BELIEFS- KNOWLEDGE-PRACTICES FOR INDONESIA’S TEACHERS A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms The MOEC has attempted to mandate the adoption of new teaching practices at various points with the goal of generate improvements to student learning outcomes. However, there is often little support provided to teachers to help them to understand the practices and to know how to use them or to understand why they are important. The result is typically that teachers simply do not adopt the mandated practices or that they do so only to a very partial extent, based on their own interpretation or in the context of the interpretation of their direct managers. This study has explored the why of teaching practices to establish clear links between teaching practices and a teacher’s knowledge and beliefs. In order to convey the complex findings in a simple and clear manner, a model was developed for use as a tool for to facilitate discussions on policies on teaching practices and on determining the measures required to ensure that they are actually implemented in the classroom. The need for this discussion is particularly apparent in the context of the implementation of Indonesia’s 2013 curriculum. This curriculum was hurriedly implemented on a nationwide scale, before implementation was halted to conduct an evaluation in December 2014. This model can be used as a tool to consider the measures necessary to ensure that the desired practices are actually implemented in the classroom. 7.1 REPRESENTING BELIEFS, KNOWLEDGE AND PRACTICES AS DISCS The main purpose of the model28 is to depict the interaction between beliefs, knowledge and practices. These components must therefore be represented in a way that allows for a representation of their interaction. All three components are highly complex and have a slightly different nature and composition, with practices being specific, concrete actions; while knowledge is measurable but difficult to fully capture in action; and beliefs are even more abstract. Despite the disparity in their make-up, these three components be represented as discs and ascribed with the following meanings: • Practices (actions): This disc represents the set of a teacher’s actions in the classroom in their entirety; • Knowledge (tools): This disc represents all MKiT held by the teacher, including subject knowledge, knowledge of students, knowledge of teaching practices and reflective competencies. As a whole, this knowledge could be considered as the teacher’s set of tools, from which he or she draws when conducting the practices. Knowledge could conceivably be separated into separate discs of subject and pedagogical knowledge (this option is discussed later in this chapter). However, in order to maintain the clarity of the model, they are kept together. • Beliefs (perceptions): This disc represents the full array of mathematical beliefs that a teacher holds, including his or her full set of perceptions regarding the nature of mathematics, how students learn mathematics, and effective methods of teaching mathematics. 7.2 DISC SIZE SIGNIFICANCE The size of the each disc holds important meaning: • Practices: As stated above, this disc represents the set of a teacher’s actions in the classroom in their entirety.29 A teacher who uses a narrow range of practices in the classroom would be represented by a small practices disc, while a teacher who utilizes a wide range of practices would be represented by a large disc. The timeframe and context play an important role in defining this disc, with the model having dynamic aspects. 28 The conceptual model of Beliefs-Knowledge-Practices, are also part of a doctoral thesis by Ragatz (2013a). 29 For the purposes of this study the range of practices would be only those used in the teacher’s 8th mathematics classroom. 92 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning A New Conceptual Model of Beliefs-knowledge-Practices for Indonesia’s Teachers • Knowledge: The size of the disc represents the extent of the teacher’s mathematics subject knowledge, knowledge of teaching approaches, knowledge of how students learn mathematics, and reflective competencies. A teacher with a lower level of knowledge would be represented by a small disc, while teachers with a higher level of knowledge would be represented by larger discs. The disc size can be thought of as representing both the depth and the range of the teacher’s knowledge. • Beliefs: As stated above, this disc represents the full array of mathematical beliefs that a teacher holds. Thus, the size represents the extent of that array. A teacher with a very focused set of beliefs (for example, a teacher who strongly believes that students learn mathematics best through the discovery approach and that discovery methods should be used for all mathematical topics and in as many learning situations as possible) would have a small disc. On the other hand, a teacher with a wide array of beliefs (for example, a teacher that sees value in a wide range of approaches and believes that different contexts call for different approaches) would have a large disc. This is primarily intended to indicate the extent of the array of beliefs and, by extension, the degree of flexibility that might exist in the teacher’s belief system. The concept of orientation is in line with Schoenfeld’s (2010) conception, which is also brought into the model through the dynamics to be discussed later. It is important to note that the size of the discs is only conceptual in nature. In representing cases of teachers from this research, the relative sizes of the discs were determined on the basis of evidence of quantitative and qualitative data gathered, but there are no clearly specified measures. For example, the knowledge disc represents an array of evidence gathered, including the teacher’s score on the teacher assessment as well as aspects of knowledge captured in the Mathematical Scenarios interview and in knowledge-related comments from the stimulated recall interviews. Although the teacher assessment involves an actual score and is quantifiable, there was no attempt to quantify the other aspects. 7.3 DISC INTERACTION As explained in the initial conceptual framework (Section 2.6), a teacher’s knowledge and beliefs are hypothesized to be critical inputs in the determination of what practices the teacher decides to use in the classroom. In the conceptual framework, the flow of interactions was represented through arrows. This model attempts to reframe the interaction in subtle but important ways to allow for the representation of more dynamic relationships amongst the three components. The overlapping of the discs in a Venn diagram represents the interaction of the components. The extent of the overlap is critical. Following the typical interpretation of a Venn diagram, the interior, overlapping portion symbolically represents the elements of a given set, while the exterior represents elements that are not members of the set. Each zone plays an important role in explaining the interaction amongst the components. Of particular importance is one of the areas where there is no interaction, where teachers use practices that are not in line with either their knowledge or beliefs. The model is illustrated in Figure 32 below and will be referred to as the BKP (Beliefs-Knowledge-Practices) Congruency Model. This is followed by descriptions of the different zones of the model, followed by their definitions. Main Report 93 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms Figure 32: Conceptual Model of Beliefs-Knowledge-Practices Interaction PRACTICES (actions) Dissonant Zone Kn ence Ab ne ow Zo s sen lief Zo led ne Ab Be ce ge Congruent Zone (practices-knowledge-beliefs all aligned in congruence) Kn s rac o ow d no tice in p ed t an led t u sed t ti ge sed r u ut no sep in dg ld, b ara pra te wle he eo fro tices kno eliefs m c be Practice B KNOWLEDGE BELIEFS lie fs (tools) Absence Zone (perceptions) The practices disc is central to the model because it ultimately contains all of the actions that actually take place. Therefore, this aspect is most directly related to student learning. Teachers can operate within four possible zones when using teaching practices. These zones are defined by whether there is overlap between the utilized practice with the teacher’s knowledge and beliefs. • Congruent Zone: The practices are congruent with the teacher’s beliefs and the teacher has the associated knowledge tools to effectively implement the practices. In this case there is a harmony between all components, with actions being backed by both conviction and ability. • Dissonant Zone: The practices are not in harmony with the teacher’s beliefs and the teacher also does not have the necessary tools to effectively implement the practices. • Knowledge Absence Zone: The practices are in harmony with the teacher’s beliefs, but the teacher has a deficit in the area of the associated knowledge tools. This could be refined further with the statement that this represents a situation in which a practice is within the teacher’s content knowledge, but that the teacher does not yet have the pedagogical tools necessary to effectively implement the practice. Alternatively, the teacher may possess the pedagogical skills necessary to implement a practice, but a lack of content knowledge impedes the effective implementation of the teaching practices to help student learning. 94 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning A New Conceptual Model of Beliefs-knowledge-Practices for Indonesia’s Teachers • Belief Absence Zone: The teacher possesses the knowledge (tools) necessary to implement the practices, but does not perceive the practices to be the most correct or effective ones to use. This could be either because of a lack of belief in the practice generally or possibly a belief that it is not the appropriate practice in a given situation. A fifth zone that falls outside of the practices disc is also of interest: • Practice Absence Zone: The teacher possesses both the knowledge (tools) and beliefs (perceptions) that would potentially lend themselves to application of related practices in the classroom, but the teacher does not implement these practices. This could be due to a variety of factors, as discussed later in this chapter. There are also two zones that have little bearing on how teachers actually teach. One zone represents a situation in which a teacher has accumulated knowledge, but this knowledge is not connected to any of his or her beliefs and is not incorporated into his or her practices. The other zone is made up of beliefs that are not tied to the teacher’s knowledge or used in his or her practices.30 The term “absence” was chosen to label areas in which one of the three components is missing. In this case, the term is meant to simply convey “the non-existence or lack of ” (Dictionary, O. E., 1989). Various other terms were considered (e.g. “gap” or “aperture” to represent an opening or hole), but the term “absence” provides a relatively straight-forward and accurate meaning. One concern is that the term absence may be interpreted as having a negative connotation. However, it should not automatically be interpreted as negative, as while absence of knowledge would in almost all cases be considered undesirable, the absence of certain practices or beliefs may in fact be a positive attribute. Each of these zones was derived from evidence gathered through the various steps of analysis and will be discussed as the dynamics of the model are described below, first in describing the discs and then relating these dynamics to actual situations as found in the analysis. 7.4 GRAVITATIONAL AND REPELLING FORCES WITHIN THE MODEL The model is not static and there are potentially strong internal and external forces at work. The clearest gravitational pull would be toward the center of the Congruent Zone, which would involve the teacher choosing practices that are most in line with his or her beliefs and where he or she possesses the tools that allow him or her to effectively implement the practice. If sub-layers were to be imagined within the Congruent Zone, the practices at the heart would be those that the teacher believes in and is most capable of utilizing. This heart of the Congruent Zone would represent the teacher’s orientation. Toward the edges of the Congruent Zone would be practices that the teacher may believe in, but with less conviction (toward the left-hand side of the zone) or where the teacher has a strong conviction, but where his or her knowledge may not be particularly strong (the right-hand side of the zone). Amongst the case studies, the teachers who best demonstrated the gravitational and repelling forces were CS3 and CS7, who identified policy and environmental forces that pushed them to use practices that they would not choose if those forces were not exerted. These gravitational forces also fit Skott’s (2009, page 31) conception that 30 It could be argued that a teacher cannot hold a belief on something on which he or she does not have knowledge. A counter to this would be that the actual knowledge of the concept overlaps with the knowledge disc, but that the beliefs-only dimension refers to the lack of knowledge of how to put it into practice. Alternatively, some beliefs can be held at a gut-instinct level without a deep knowledge of it. Main Report 95 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms beliefs should not be seen in isolation but rather as: “... shifting, intersubjectively established, and continually (re-) generated settings. This means that contexts are construed by individuals as they participate in practices that evolve in interaction.” He argues that it makes little sense to talk about beliefs in themselves, irrespective of context, nor to expect a one-to-one correspondence between beliefs as espoused in interviews and practice. This view conforms to the model used in this analysis. Although many teachers would prefer to operate deep within the Congruent Zone, the gravitational pull would be less strong for those teachers who are eager to develop their skills and expand their array of practices. These teachers might utilize some practices that fall within the Belief Absence Zone or the Knowledge Absence Zone. An example of a teacher operating in the Knowledge Absence Zone would be that of a teacher who believes that a new practice would be applicable in his or her classroom and is willing to try it, but that she still lacks the tools necessary to effectively implement the practice. This may be most applicable to newer teachers, who are still open to exploration, but who have yet to develop the necessary tools. Specific examples will be discussed later. The Belief Absence Zone may be entered by choice if the teacher has the appropriate tools to implement a practice, but the teacher is still somewhat skeptical regarding its effectiveness. Most likely a teacher would not enter this zone completely voluntarily if he or she teacher truly does not believe in the practice. Figure 33: Comparison of Teachers Reacting to Influences External in uences External in uences on which practices to use on which practices to use PRACTICES PRACTICES Dissonant Zone Dissonant Zone Internal in uences Internal in uences pushing outward pulling inward Congruent Zone Congruent Zone KN KN OW OW S S EF EF LE LE LI LI BE BE DG DG E E Teacher open to learning and experimenting Teacher wanting to stay in congruent zone with new practices and use same practices 96 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning A New Conceptual Model of Beliefs-knowledge-Practices for Indonesia’s Teachers For teachers operating in the Knowledge Absence Zone and Belief Absence Zone, there is a possibility that the practices in this area will eventually become part of the Congruent Zone. Working in the absence zones may be less effective than the Congruent Zone initially, but it may be beneficial in the longer-term. A teacher might, through use and repetition of a practice, develop the knowledge necessary to implement it effectively or become convinced that the practice is in fact effective and thus incorporate it into his or her belief system. In contrast to the Congruent Zone, the teacher would not choose the Dissonant Zone of his or her own volition, because it does not fit either the teacher’s personal knowledge set or beliefs set. This state would only arise if the cultural, system or environmental forces are so strong that the teacher feels obligated to use practices other than those they would choose without such influences. Potentially large outside influences include the requirements of a curriculum, evaluation requirements established by the principal or supervisor, or a teacher working in a school located in a cultural environment that differs significantly to the one with which the teacher is accustomed. Operating in this zone is likely to be detrimental because the teacher would lack the tools to properly implement the practices and would also lack conviction or enthusiasm. Most teachers would have a particularly strong incentive to resist this situation. Even if they attempt to use the prescribed practices initially, they would tend to slip back into the Congruent Zone after their probable failure to use the given practices effectively. A teacher may be fortunate enough to be working in a system or school environment that is in alignment with her/his beliefs and utilizes her/his set of tools. In this case, the Congruent Zone would be large and the Dissonant Zone would be almost non-existent. The teacher may also operate in an environment that would provide support for ventures into the Knowledge Absence Zone and Belief Absence Zone through training, supportive peers, or other elements that would help the teacher to expand his or her Congruent Zone. On the other hand, if there are cultural, system or environmental forces that push the teacher outside of this zone, the teacher may resist and attempt to stay within this zone. If a large proportion of a teacher’s practices fall within the Dissonant Zone, such as in a case where an educational reform process requires a teacher to use a new and completely foreign teaching approach, then it would almost certainly have a severe negative effect on the teacher, both through deterioration in his or her level of confidence (due to lack of tools) and to a deterioration in his or her morale (due to teaching in ways that are not in line with the teacher’s beliefs). 7.5 THE CULTURAL DIMENSION AS A KEY DRIVER One of the key underlying factors that may determine whether teachers are able to adopt specific teaching practices is the extent to which the practices fit within teachers’ cultural sphere or context. For example, a student-centered approach may go against certain norms that are deeply culturally ingrained. If teachers are traditionally viewed as the source knowledge and sayings like, “Quiet students, happy teacher”31 shape what is considered acceptable student-teacher interaction, then it would be very difficult to change their practices through training course that simply provides knowledge of the approach. Even teachers who are open to the idea of a student-centered learning approach would likely internalize the training within their own (at times deeply ingrained) conceptions and possibly interpret and reshape the approach to fit within their own pre-existing beliefs. Indeed, Indonesian teachers do tend to state beliefs that are in line with active learning and student-centered approaches, but what is seen in the classroom that may tends to be involvement of students, but through a teacher-led approach. Practices are more likely to be adopted if they align with cultural norms or if the training takes into account the cultural aspect and is designed to specifically reshape conceptions and beliefs. 31 This saying was found carved into one of the ceiling beams of a rural school in Indonesia. Main Report 97 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms 7.6 APPLICATION OF THE MODEL TO SPECIFIC INSTANCES The model can be transformed in a variety of ways, particularly in terms of the degree of overlap and size of the discs. The following are important examples of how the model can change, but is by no means an exhaustive list. 7.6.1 LARGE DISSONANT VERSUS LARGE CONGRUENT ZONE The congruent and dissonant zones represent the biggest potential competing tensions faced by a teacher in terms of teaching practices. As mentioned above, all teachers would attempt to minimize operating in the Dissonant Zone. A teacher operating in an environment that is either in line with his or her knowledge and beliefs or in an environment where the teacher has control over the practices he or she chooses will have a minimal Dissonant Zone. On the other hand, a teacher facing pressure to use practices outside his or her Congruent Zone might feel obligated or forced to use practices that place him or her in the Dissonant Zone. The extreme example below could only come about through a major shift, such as in the case of a reform through which teachers are required to completely change their practices. Even if such a state could exist in the short term, a teacher is very unlikely to remain in this unbalanced state in the long term, as he or she would likely gravitate back to the Congruent Zone and stop using the dissonant practices. It may also be possible that in the long term, the teacher would come to accept and incorporate the practices into his or her Congruent Zone, although it would be unlikely to occur unless the teacher were to receive intensive support that would build both a belief in the practices and the associated knowledge tools required to effectively utilize the practice. Figure 34: Large Dissonant vs. Large Congruent Zone PRACTICES Dissonant Zone PRACTICES Congruent Congruent Zone Zone KN KN FS IE OW OW L BE LE LE S DG DG EF E LI E BE 98 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning A New Conceptual Model of Beliefs-knowledge-Practices for Indonesia’s Teachers 7.6.2 COMPARISON OF HIGHER AND LOWER KNOWLEDGE TEACHERS As has been mentioned, three of the key findings from the analysis are that: 1. Teachers with a lower level of lower knowledge appear to have a clearer or narrower set of beliefs than teachers with higher levels of knowledge, who express a range of beliefs, recognize context and note that practices vary depending on the context; 2. Teachers with a higher level of knowledge tend to utilize a larger variety of teaching practice tools, such as a larger range of questioning techniques; and 3. Teachers with a higher level of knowledge and a larger array of beliefs vary their practices to a greater extent. These findings were drivers for the design of this model in terms of meeting the need to establish a means to show overlap amongst beliefs-knowledge-practices and also to demonstrate the extent and size of these three components. The example below is based on the analysis of the data on CS9 and CS232 and illustrates the above points. When the teacher with a relatively high level of knowledge (CS9) also has a larger array of beliefs (consistent with Point 1 above), this is related to greater flexibility in his or her teaching practices (Point 2) and ability to vary the practices (Point 3). On the other hand, the teacher with a relatively low level of knowledge (CS2) has a lower knowledge disc, which limits the extent to which practices can fall within the Congruent Zone. The teacher’s beliefs disc is also shown to be smaller than the higher knowledge teacher’s, but it is sufficiently large that a large portion of the beliefs disc falls outside the knowledge disc, creating a large Knowledge Absence Zone.33 Figure 35: Illustrative Contrasting Example between Higher and Lower Knowledge Teachers PRACTICES PRACTICES Large Knowledge Absence Zone Congruent Zone Congruent Zone KN S OW EF KN LI LE BE DG OW S EF E LE LI DG BE E 32 The sizes and overlap of the discs are based on interpretation of the teacher assessment and mathematical scenarios interview, beliefs survey and beliefs interviews, and lesson observation of practices along with the associated stimulated recall interviews. The disc sizes and overlaps are not exact measurements and therefore should be understood to be interpreted only. 33 The knowledge aperture zone is reflective of the teacher’s attempt to teach inner and outer tangents, but had to rely completely on the textbook and only read the steps to the students without being able to explain the purpose of the steps. Main Report 99 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms A key observation from the figure is the highly contrasting sizes of the Congruent Zone as a result of the large differences in the size and degree of overlap of the knowledge and beliefs discs. The higher knowledge teacher has a much larger set of tools, a larger array of accompanying beliefs and can therefore utilize a larger set of teaching practices. The size of the Dissonant Zone is also minimal in the case of the higher knowledge teacher, while the lower knowledge teacher has a larger Dissonant Zone due to the need to teach all aspects of the curriculum without necessarily having all the tools required to properly teach all the topics required by that curriculum. The observed case of CS2 provides a good example of the Knowledge Absence Zone in practice. She received a lesson plan from a different teacher on how to draw inner and outer tangents. It was a numbered list of steps involved. She did not fully understand the steps involved, so she spent the lesson requiring the students to copy down the steps. She struggled to draw the tangent on the board herself and made some minor errors. In the stimulated recall interview, she was asked to explain the purpose behind the specific steps, but only responded that they were the instructions, so the students should do them. These factors indicated that she was lacking certain aspects of knowledge (both subject and pedagogical) to effectively teach that particular topic. This can be contrasted with the case of CS3, who taught the same topic. This teacher obtained the highest subject score on the MKiT Assessment and was able to conduct the lesson without notes, being easily able to describe the steps involved and their purpose. In his case, the practices he used were within his Congruent Zone. Another example of the Knowledge Absence Zone from the case studies is that of CS7. She was a relatively young teacher and although she had three years of teaching experience, it was her first year in her current school. She strongly believed that students should be actively engaged, but mainly due to the context in which she operated (a poor rural environment in which the cultural norm appeared to be for students to refrain from proactively speaking out in the classroom), she was struggling to get the students to participate. In the stimulated recall interviews, she expressed her frustration, claiming that she could not figure out how to get them to become more engaged with the process. She clearly had a belief in a practice and was attempting to implement it, but she did not possess the knowledge required to engage them. She may persist with the use of this practice and, over time, develop the knowledge required to motivate students to participate more actively. In this case, the knowledge disc would expand and the Knowledge Absence Zone would close. Another possibility is that, over time, her beliefs will change and she will no longer see active engagement as an effective way to teach. In this case, she would reduce her Practices and Beliefs discs, and the Knowledge Absence Zone would also be reduced. The Belief Absence Zone is an area in which a teacher would rarely operate. It would require that a teacher use practices that are within the knowledge disc and that the teacher has in fact chosen to use the practice, but that the teacher does not have an underlying belief in the effectiveness or usefulness of the practice. One case study example that at least approaches that zone is CS3, who indicated that he was not using his ideal set of practices (which would involve a student-centered approach in which he would act as a guide and facilitator) because the achievement level of his students was low and the practices would therefore not be effective. Instead, he chose to take on a more instructor-oriented role. It appeared that he had both the subject and pedagogical knowledge to utilize this instructor-oriented approach, so its use would not place him in the Dissonant Zone. While this case might be considered one where the teacher does not believe in the practice, it could perhaps be better described as a situation in which the teacher believes in the practice within the context of the given situation, in which case it would not fall in the Belief Absence Zone. 100 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning A New Conceptual Model of Beliefs-knowledge-Practices for Indonesia’s Teachers A more likely scenario is one in which teachers feel forced to use practices due to curriculum requirements, school requirements or other authoritative directives that require a specific practice to be used. The case study teachers did discuss how the curriculum influences their practices and CS3 expressed a level of frustration that his system influences limited his freedom to teach in the manner that he felt appropriate. However, the case studies did not produce a clear example of a teacher operating in the Belief Absence Zone. In a sense, it is a zone that remains theoretical. 7.6.3 KEY CONJECTURES FROM THE MODEL The model reflects many important points derived from the analysis, enabling a range of conjectures related to the interaction of teaching practices, teacher knowledge and teacher beliefs. A few major implications are briefly discussed below. There has been an immense amount of research in relation to effective teaching, both in terms of what teacher background factors characterize effective teachers and of what practices are most effective for generating higher student learning outcomes. This study has also attempted to explore the relationships between teaching practices and student learning outcomes and to gauge how knowledge and beliefs relate to student learning outcomes, although it is important to reiterate that it is the practices that are the conduit to student learning. By stepping back and viewing effective teaching within this model, what conclusions can be drawn regarding effective teaching? One key implication of the model is that effective teaching practices are relative rather than absolute, emerging through their relationship with knowledge and beliefs. What might be described as an effective practice for a teacher with a high level of knowledge might be ineffective for a teacher with a lower level of knowledge, if using that practice places the teacher outside his or her Congruent Zone. For example, many would argue that the practice of provoking and directing questions is more effective than simply using testing questions. However, would that practice necessarily be effective for all teachers in all situations? Most likely not, if utilizing the practice places the teacher outside his or her Congruent Zone. If a teacher does not have an understanding of why provoking/directing questions is an effective method, then the practice may not fall within the teacher’s belief disc as an effective method. If the teacher lacks the pedagogical knowledge of how and when to use provoking/directing questions and also lacks the subject matter knowledge to effectively direct student thinking and then to respond appropriately to students questions, then it falls outside the teacher’s knowledge disc. Pushing the teacher to use a practice for which he or she is not equipped with adequate knowledge or in which he or she does not believe in its effectiveness could in fact result in a deterioration of the learning environment. The teacher might guide the students in the wrong direction; it could create confusion for both the students and teacher; the teacher might lack enthusiasm, and; when students provide responses, the teacher may not have the tools necessary to provide effective responses. The model would imply, therefore, that effective teaching practices are relative rather than absolute and are dependent on individual teacher characteristics, including knowledge and beliefs. Teachers who are forced to use practices or approaches that place them outside their Congruent Zone are less likely to use them effectively. This is particularly true if the practice falls outside both their knowledge and beliefs. Main Report 101 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms A related conclusion is that in order for teachers to adopt a new teaching method it is important to address not only the “how to” aspect of the practice (knowledge), but also the “why” aspect (beliefs). Often training gives a basic level of “how to” and provides some theory on “why”, but in order for it to be truly understood and adopted, the practice must be readily applicable and practical. It also would tend to require a process over a period of time rather than a one-shot intervention. Often it is said that teachers who attend a course in a training center will tend to not apply it once they return to the classroom. Even if they apply it then they often quickly fall back into their previous pattern. An extended process which involves teachers learning, practicing in their own classroom, reflecting and re-applying is more likely to build deeper understanding and a belief in its applicability and effectiveness. The process must be extensive enough that over time the new practice is enveloped within the teachers’ knowledge and beliefs. This is explored in section 10, which describes a pilot professional development module designed to change teaching practices through building of knowledge and reshaping of beliefs. 102 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning CHAPTER 8 THEMES ON TEACHING PRACTICES AND POLICIES A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms The Technical Report provides a step-by-step analysis for each individual teaching practice. While this is useful on its own, it becomes more meaningful when considering the practices in a larger context. The following section presents the most relevant findings in a thematic format. This is followed by a thematic discussion from a policy perspective. As will be seen, there are overlaps and connections amongst the themes, highlighting the nature of the interconnections between the different perspectives on what takes place in the classroom; why it takes place; and how to provide support to teachers to enable them to improve their practices. 8.1 HOW TEACHER KNOWLEDGE IS ENACTED IN THE CLASSROOM Indonesia’s teachers have low levels of both subject and pedagogical knowledge, as shown by the results of the Ministry of Education and Culture (MOEC) teacher competency exam, which includes both subject and pedagogical items. This study and a recent study on certification in Indonesia (de Ree, 2014) make it clear that the level of teacher subject and pedagogical knowledge is the strongest educational factor in student learning outcomes. Therefore, only by addressing the deficits in teacher knowledge will student learning outcomes start to improve. Figure 36: Distribution of Scores of Mathematics Teachers in UKG All Secondary 57% 23% 20% Junior Secondary 43% 28% 29% Senior Secondary 77% 17% 6% Vocational Secondary 87% 11% 2% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 1 to 50 51-60 Over 60 Indonesia teachers have very low levels of subject and pedagogical knowledge The MOEC recently developed a teacher competency assessment (Ujian Kompetensi Guru, or UKG) that includes items related to both subject matter and pedagogy. To this point, more than one million certified teachers have taken the assessment. The results for 50,000 mathematics teachers indicate that there is a wide range of competency levels, but that a very large proportion of teachers have low levels of subject and pedagogical knowledge, with more than half of all teachers scoring below 50%. The scores at the senior secondary level were particularly low, with 77% of general school teachers (SMA) and 87% of vocational school teachers (SMK) falling below the 50% mark. If 60% is considered as a threshold for a base level of competency, very few teachers reached this level. Only 20% obtained a score over 60%, with just 6% of SMA teachers and 2% of SMK reaching this mark. 104 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Themes on Teaching Practices and Policies Teacher knowledge is crucial in improving student learning outcomes While there is broad agreement that it is good to have teachers with strong competencies, some believe most teachers already have sufficient levels. However, two recent studies34 by the MOEC and the World Bank indicate that this is far from the case. In both studies, teacher subject and pedagogical knowledge emerged as the education system factor with strongest relationship with student learning outcomes. While the positive relationship is not surprising, the fact that the relationship was so strong highlights just how essential teacher subject matter and pedagogical knowledge are in the Indonesian context, particularly since a large proportion of teachers have very low competency scores. The results indicate that teachers with particularly low levels of subject and pedagogical knowledge produce only small benefits in terms of student learning outcomes. Improvements in the knowledge of teachers with the lowest levels could produce the greatest impact in student learning outcomes. However, even for the teachers with relatively high scores, increases in competency still produce increased learning gains. This result indicates that there is no point at which teachers reach a sufficient level. Therefore, continuously improving teacher knowledge for teachers of all levels is beneficial. Why teacher knowledge plays such an important role in student learning International research into teacher knowledge has explored its complex nature and multiple dimensions. Mathematical Knowledge in Teaching (MKiT) has emerged as a commonly used term in research to describe the specialized nature of the knowledge that mathematics teachers should possess. A typical distinction is to separate subject matter from pedagogical content knowledge, but further distinctions can also be made. Askew et al. (2003) break the knowledge down into three specific categories: • Subject matter knowledge: What teachers know about mathematical concepts and their ability to solve mathematical problems. • Knowledge of students: What teachers know about students’ knowledge of mathematics, including typical misconceptions students have, how to assess student knowledge and an understanding of their own students’ ability levels and the varied ways they best learn mathematics. • Knowledge of teaching: What teachers know about how to teach the subject of mathematics, including the knowledge of planning and enacting mathematics effectively in the classroom. How higher teacher knowledge plays a role in Indonesia’s classrooms The Indonesia TIMSS35 Video Study explored teaching practices in Indonesia’s eighth grade mathematics classrooms through extensive quantitative and qualitative analysis. The study goes beyond just identifying the relationship between teacher competency and student learning outcomes. It also explores how different teachers use their knowledge in the classroom and how this makes a difference to student learning outcomes. The study found that there are clear differences in the practices of teachers with higher levels of knowledge. These differences include not only the range of practices used, but, more importantly, the level of effectiveness with which the same practices were executed. 34 DeRee (2014) Teacher certification and beyond: An empirical evaluation of the teacher certification program and education quality improvements in Indonesia, World Bank 35 Trends in Mathematics and Science Study. Main Report 105 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms A larger toolbox… Teachers with higher levels of knowledge tended to utilize a greater range of practices. Techniques such as investigation, open-ended questioning, the use of mathematical language and symbols, and the use of non-routine problems and applications were found more often in classrooms in which teachers had higher levels of knowledge. In part, this appears to be because the techniques themselves require a greater amount of subject mastery to be conducted effectively. For example, open-ended questioning requires the teacher to develop questions that can be answered in a variety of ways, with these questions tending to be relatively complex. When in the classroom, teachers must be prepared for the different possible responses of students and they must have contingencies for the different responses, which possibly involve higher-order thinking. …containing better tools While higher-knowledge teachers tended to have a larger toolbox, they also tended to use the same tools more effectively. While higher- and lower-knowledge teachers may utilize the same methods and practices, such as group work, higher-knowledge teachers were more effective. They were able to interact with students more, spot misconceptions that the group might have, and also come up with better ways to help correct the misconceptions. Further in-depth qualitative analysis uncovered important differences in the ways higher-knowledge teachers taught more effectively while using the same practices. Teachers with higher levels of knowledge tended to: 11. Make fewer mathematical errors; 12. Use more accurate language and be more concise in their explanations; 13. Come up with new problems that were not originally in the lesson plan; 14. Make greater attempts to probe student thinking; 15. Be more able to spot student misconceptions and effectively correct them; 16. Involve a greater range of students, including those who might not provide correct answers; 17. Demonstrate more confidence and command in the classroom; 18. Stimulate more teacher-student interaction, with the interaction involving a greater degree of higher order thinking; 19. Use questioning to a greater extent and encourage students to think and share ideas rather than just to follow instructions; and 20. Make connections between procedures and concepts. The above points could play a role in student learning from a variety of angles. The use of more concise language and the making of fewer errors would likely lead to less confusion and a greater level of understanding among students. Misconceptions that students hold may be: (i) identified; (ii) probed to determine the cause; and (iii) addressed using appropriate techniques so that they do not become embedded in the student’s mind and cause future problems and misunderstanding at later stages of learning. A mastery of both the subject and pedagogical techniques can allow teachers to run their lessons with greater confidence, possibly leading to a higher level of involvement by students. Higher levels of pedagogical knowledge can enable a higher level of involvement of students because of the teachers’ better understanding of the thought processes of students. A teacher with higher levels of pedagogical knowledge will also be better able to address situations that do arise in the classroom. Teachers with higher levels of knowledge are also able to better utilize problems; set up activities; and provoke higher-order thinking. 106 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Themes on Teaching Practices and Policies A contrasting approach was found in many Indonesian classrooms While certainly no two classrooms are the same and no two teachers teach in exactly the same way, Indonesia’s eighth grade mathematics classrooms exhibit many common features in their teaching practices and follow similar patterns. In the majority of classrooms, whole-class interaction (as opposed to group work or seat work) predominantly involves one-way teaching practices, with little meaningful student involvement. Often, the cues provided by teachers for student interaction are simple testing questions, with almost 90% of student responses consisting of single word answers. Indeed, often these questions are pseudo-questions that involve filling in the blank (teacher: “centi…” students: “…meter”) or questions where students repeat an answer to a question in chorus as the teacher points to on the board. Such interaction may serve the purpose of keeping the attention of students, but it does not provoke deep thinking. By contrast, teachers with higher levels of knowledge tend to play the role of guide; to use more effective questioning techniques; and to conduct activities conducive to higher-order thinking. Contrasting approaches in the handling of a mathematical scenario How might teachers with varying levels of knowledge, given the same situation, respond differently? To determine an answer to this question, ten teachers were asked to take part in in-depth case studies. They participated in interviews to probe their level of mathematical knowledge in teaching by discussing a number of hypothetical mathematical situations that might be encountered in the classroom. Below is an example scenario where a student had a misconception related to fractions: Arif was asked by his teacher to make drawing to compare 3/4 and 5/6. Arif made the following drawings: and claimed that 3/4 and 5/6 are the same amount. Main Report 107 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms The following chart summarizes the general tendencies of teachers with higher levels of knowledge compared to those with lower levels, although it should not be taken to represent the cases of all teachers. LOWER-KNOWLEDGE TEACHERS HIGHER-KNOWLEDGE TEACHERS Identification of the mathematical error All were able to identify the error, but some had difficulty Tended to not only identify, but also concisely explain the error. verbalizing it. Understanding the student’s reasons for the misconception Some struggled to identify likely reasons. Some connected it to similar difficulties faced by their own One teacher felt it was not the teacher’s role to “guess or assume” students. what the student was thinking. All came up with the most likely reason, but some also attempted to come up with other possible explanations. If they were in the teacher’s position, how would they approach the situation? None mentioned attempting to work with Arif to identify why he Some teachers wanted to first ask Arif questions in order thinks they are the same. to understand why he made the error and the cause of the misconception. Tended to jumped into definitions of fractions and mathematical Many used creative tasks involving cognitive conflict to help Arif explanations. see the error and then construct a new, correct understanding. This and other scenarios used in the interview revealed important insights into how the multiple dimensions of teachers’ knowledge of subject, of students, and of how to teach, could be employed in classroom situations. For the scenarios that were more challenging mathematically, the teachers with lower levels of knowledge tended to struggle to identify and explain the issues. The teachers’ knowledge of students came out mostly in terms of not only whether they could attempt to “get into the mind of the student” and thereby to determine possible reasons for the error, but whether they took steps to come up with questions to probe students’ reasoning and to understand why they came to the wrong conclusion. Finally, teachers with higher levels of knowledge tended to utilize creative tasks to help correct errors and misconceptions. They also tended to include the use of cognitive conflict or new examples, rather than just relying on the existing example. Policy Implications Related to Teacher Knowledge Teacher quality is an extremely complex, multi-dimensional concept. Teacher knowledge is certainly only part of the equation. Still, the strong correlation between teacher competency test scores and student learning outcomes indicates that it is a relevant indication of teacher quality. Further qualitative analysis indicates that this relationship is causal, in that the knowledge could be seen to be employed in the classroom practices in ways that would be likely to produce better learning outcomes. It also indicates that policies supporting and encouraging teachers to strengthen their subject and pedagogical knowledge are likely to lead to improved student learning outcomes. Indonesia’s Teacher Professional Management System (TPMS) The MOEC has undertaken the massive task of developing a Teacher Professional Management System (TPMS) in which the key elements form an “integrated framework” of requirements. The system makes the teacher directly accountable for both work performance, through an annual performance-based teacher appraisal (Pemantapan Kerja Guru, or PKG), and professional development, through continuous professional development (CPD). The 108 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Themes on Teaching Practices and Policies appraisal procedure requires open discussion and agreement between the teacher and an assessor, and approval by the principal, if the principal is not acting as the assessor. Both the appraisal and CPD processes are monitored and supported by supervisors at the school level. Teachers must also pass a minimum standards test in professional and pedagogic competency (Uji Kompetensi Guru, or UKG) in order to progress up through the professional levels, with the rate of progression depending on the teacher’s total number of credit points, which is established on the basis of the results in the PKG and CPD. The UKG is used as a contextual input for the PKG, and the two of them together should then be used to define the CPD plan of each individual teacher. Figure 37: Integrated Framework for the Teacher Professional Management System What a teacher does Feedback based on observation Performance Appraisal Continuous Competency Professional Test Development What a teacher knows Tools for improvement Identi cation of knowledge areas for improvement Tailored plan based on identi ed needs The possible uses of the teacher competency assessment (UKG) The UKG can serve important purposes in teacher management, including the hiring of teachers, identifying areas of improvement for incumbent teachers, providing the appropriate support to improve competencies, and rewarding teachers who demonstrate high levels of competency and a strong rate of improvement. • Hiring of teachers: The UKG can be used in the hiring process. Only teachers who can demonstrate a threshold mastery or strong subject and pedagogical knowledge should be hired; • Professional development support of teachers: The UKG is already incorporated into MOEC’s TPMS. The results can be used to identify teacher’s strengths and areas where improvements are required. The areas for improvement can then be addressed through the teacher’s annual professional development and results plan; Main Report 109 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms • Progression and promotion: The TPMS also uses the UKG results to determine decisions related to a teacher’s progression and promotion. The thresholds have not yet been established but, if the results are to encourage teachers to improve their knowledge, the thresholds must be meaningful and teachers must meet the competency requirements in order to be promoted. But the effective use of competency testing requires strong political will… If utilized properly, the UKG can serve as a tool to help teachers identify areas for improvement; to set goals and targets; and to seek out ways to improve their level of competency. However, to encourage improvements in levels of competency, the UKG must also serve the purpose of setting thresholds and ensuring that teachers reach those thresholds. This requires strong political will. However, if implemented, it could play an important role in improving levels of teacher competencies and generating better student learning outcomes over the long term. …as well as a reliable and effective instrument If the UKG is to fulfill these roles, the instrument and the way it is administered must be generally regarded as a valid, reliable and credible measure of teacher competency. An invalid tool could lead to poor decisions, poor outcomes and a frustrated teaching workforce. The assessment items must be validated thoroughly and must be based on subject and pedagogical knowledge that is truly relevant for teachers for each subject and at each school level. Investing in a valid, reliable and credible instrument will ensure its acceptance by all stakeholders. By contrast, an instrument that lacks these attributes will create serious credibility issues that could have the opposite of the intended effect. The importance of continuous professional development (CPD) One of the key roles of the UKG and the performance appraisal is to serve as a feedback mechanism. By identifying areas in which teachers need to improve, they can engage in professional development activities that will address their deficiencies. This requires teachers to be aware of and to have access to a wide range of materials that are necessary to address these deficiencies. This would involve the cataloging and mapping of materials available from teacher training centers, district/city governments, third-party vendors and online resources. Teachers should be provided with clear support to engage in CPD. They should also receive credit for their efforts, perhaps by being accredited on the basis of their passing tests or producing specific outputs through the CPD process. Support should be provided for avenues to enable teachers to participate in CPD, with these avenues including teacher working groups (KKG/MGMP36); online courses, such as the MOEC’s mathematics education teacher training center (P4TK Matematika); and courses offered by the Open University (Universitas Terbuka, or UT) and by teacher training colleges (LPTK). A clear strategy involving the provision of a number of options will be critical for the TPMS to function optimally and to achieve real improvements in levels of teacher competency and performance. Teacher working groups at the primary school level are Kelompok Kerja Guru (KKG) and at the secondary school level are Musyawarah 36 Guru Mata Pelajaran (MGMP). 110 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Themes on Teaching Practices and Policies 8.2 TEACHER-STUDENT INTERACTION One of the key focal areas of the Video Study was on teacher-student interaction, with the findings providing a new perspective on teaching-learning culture and norms in Indonesia, as well as showing how student-teacher interaction relates to student learning outcomes. The study shows that the approaches encouraged through the 2013 curriculum tended to have positive relationships with student learning outcomes. However, it also shows that the type of teacher-student interaction envisioned by the new curriculum was rare. The new curriculum encourages the increased use of a student-centered learning approach in which the teacher’s role is to serve more as a guide and facilitator in the process of student learning. Therefore, the challenge will be to enable teachers to facilitate changes in classroom culture and to assist students to better understand their role within this new student-centered learning culture. The transition towards higher levels of student interaction and the adoption of measures to encourage higher-order thinking could shape the next generation, building their confidence to express ideas and ensuring that they are more able to undertake tasks requiring higher-order thinking. However, in order to achieve this, teachers will require a great deal of support if they are to be able to effectively utilize the new classroom practices. Somewhat disturbingly, the trend in the period from 2007 to 2011 indicates that the extent to which teacher- student interactions take place in whole-class situations is not only low, but actually decreasing. By contrast, the extent to which lecturing and teacher-centered learning takes place is increasing. In other words, the quality of teacher-student interaction is actually getting worse. 8.2.1 WHAT IS A STUDENT-CENTERED APPROACH? When student-centered instruction predominates in the classroom context, students and teachers share the focus, with much higher levels of student participation. Instead of just listening to the teacher, students and teachers interact to a much higher degree. Group work is encouraged, discussion and questioning is more prevalent, and students learn to collaborate and communicate with one another. The perceived benefits of student-centered learning include the following: • Students learn important communication skills through more intense interactions with the teacher and other students; • Through deeper discussions and the sharing of ideas, students may be able to develop a deeper understanding of concepts; • Students learn to direct their own learning process by asking questions and completing tasks independently; and • The construction of knowledge for oneself can lead to greater genuine learning and mastery of subject. Different teaching practices promote the development of different cognitive skills. More traditional, teacher- centered approaches tend to promote factual knowledge and routine problem solving skills, while student-centered practices promote reasoning skills. There are benefits and tradeoffs to each of these approaches. Main Report 111 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms 8.2.2 INTERACTION AND RECENT POLICIES ON TEACHING PRACTICES In 2013, Indonesia implemented a substantial reform to its national curriculum, with this reform involving not only a change in content, but also a change in emphasis on the specific approaches adopted in the teaching-learning process, with a far greater emphasis on student-based learning.37 As will be discussed in Section 8.3, the models of learning emphasized in policies over the past decade are encouraging students to proactively seek out knowledge and to make independent observations. Students are encouraged to formulate problems (ask), rather than just solving problems (answer). Students are also taught to work together and collaborate to solve problems to a far greater degree. In turn, teachers are encouraged to use classroom practices that foster independent learning; observation; questioning; processing; making judgments; communicating and creating a more open and student-centered teaching/learning paradigm (MOEC, 2013). The teacher’s role is to serve more as a guide and facilitator, rather than being the source of all information. According to this model, interaction and engagement with students becomes central to the teaching-learning process. Various studies have indicated that, even with the major reforms to the Indonesian educational system that took place more than two decades ago, the student-centered approach is relatively uncommon. Somerset (1997) described the teaching-learning process as almost always involving the use of traditional, teacher-centered approaches. Hendayana et al. (2003) agree, stating that teachers tend to dominate mathematics lessons, preventing students from participating to a meaningful degree and in meaningful manner. Through their observation of indignation classrooms, they found that activities tended to predominantly involve students copying notes and listening to teachers, with very little interaction between students and with a large number of students appearing to be disengaged. Fauzan et al. state that the classroom climate generally has the following characteristics: pupils are passive throughout the lesson; ‘chalk and talk’ is the preferred teaching style; there is an emphasis on factual knowledge; questions tend to require only single word responses, often provided in chorus; there is a lack of learning questioning; only correct answers are accepted and acted upon; and whole-class activities often involve writing, with no hands-on work carried out (2002). This study provides a more recent picture of what takes place in the classroom, with its two phases in 2007 and 2011 providing the opportunity to spot emerging trends and changes. Following the same analytical and coding approach to that adopted in the ground-breaking 1999 TIMSS Video Study (Hiebert et al, 2003) also enabled a cross-country comparison of results. 8.2.3 INTERACTION OF TEACHERS AND STUDENTS When examining the proportion of time allocated to whole-class activities compared to that allocated to group and seatwork, Indonesia’s classrooms tend to have a slightly higher than average time allocated to whole-class interaction, at 66%, with this proportion being in the middle compared to other countries that participated in the 1999 TIMSS Video Study. 37 The 2013 curriculum was initially rolled out to 6,221 schools in 2013, then rolled out nationally for the 2014/2015 school year, but the rollout was put on hold in order for a more formal evaluation to be conducted. It is expected that refinements will be made to the curriculum and that it will be rolled out again at a slower pace between 2015 and 2019. 112 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Themes on Teaching Practices and Policies Figure 38: Proportions of Whole-class vs. Group and Seatwork Hong Kong SAR 80 20 Czech Republic 79 22 United States 68 32 Japan 66 34 Indonesia 64 36 Switzerland 55 45 Australia 52 48 Netherlands 44 55 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Public (whole-class) interaction Private (group work and seatwork) interaction While Indonesia tends not vary too greatly from other countries when it comes the time allocated to whole-class and private interaction, a deeper examination reveals that not only is the student-centered learning approach relatively uncommon, but the use of this approach is actually declining. The average proportion of time allocated to whole- class activities has increased from 59% in 2007 to 64% in 2011. When private interaction is broken down into group work and seatwork, it is clear that the increased whole-class time has come at the expense of group work, with the proportion of time dedicated to this work declining from 21% to 15%, despite the fact that the group work is considered a cornerstone of student-centered learning. Figure 39: Whole-class Work is on the Increase, while Group Work is Declining 2007 59 21 20 2011 64 15 21 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Whole class Group work Individual seatwork Whole Class Group Work Seatwork Whole-class interaction by type Whole-class interaction does not necessarily exclude a student-centered approach. Interaction between teachers and students was broken down into teacher-only, teacher-student, and student-only interactions. Teacher-only interaction is considered to imply a traditional approach, while teacher-student and student-only interactions are often associated with progressive and student-centered learning approaches. These divisions also relate to the teacher’s conception of his or her role as instructor (for skill mastery); explainer (for conceptual understanding); or facilitator (for confident problem solving) (Kuhs and Ball, 1986). Main Report 113 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms Whole-class interaction time by teachers and students in 2007 and 2011 2007 61 21 18 2011 74 11 15 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Teacher only Teacher and student Student only Whole Class Note: The changes between 2007 and 2011 are statistically significant for “teacher only” and “teacher and student.” Whole-class time refers to time when the interaction, regardless of the participants, is intended for the whole class. Again, the trend indicates that student-centered approaches are increasingly less commonly utilized. The analysis shows that a much greater proportion of classroom time was teacher-only time in 2011 than in 2007, with this proportion increasing from 61% to 74%. This time typically involved expositional or lecture activities. On the other hand, student involvement in the form of teacher-student interaction, declined from 21% to 11%. A troubling aspect of this decrease is that the students of teachers who used more teacher-only time tended to have lower learning outcomes, while those with more teacher and student interaction had higher learning outcomes.38 The relationship with student learning indicates that the 2013 curriculum’s student-centered learning emphasis may lead to better learning outcomes, but the proportion and trends indicate that implementation of the new curriculum will face challenges. Teacher role in group work and seatwork In looking at time used for group work and seatwork, another important finding emerged. While there were no statistically significant differences in student learning outcomes relative to the proportion of time allocated for group work relative to seatwork, there was a positive relationship between classrooms in which the teacher was actively monitoring and providing feedback with student learning for both group and seatwork activities. Even when a student centered approach is adopted, the teacher still has a significant role to play as a guide and facilitator. 38 Student-only time did not have a statistically significant relationship with student learning. 114 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Themes on Teaching Practices and Policies Private (group and seatwork) interaction time in 2011 Group Work Individual Seat Work Teacher Students Teacher involvement only involvement Students 37% 48% 52% only 63% Group Work Seatwork Student-teacher interaction is important The fact that a positive relationship was identified between levels of teacher-student interaction and student learning outcomes in whole-class, group work and seatwork indicates that active engagement with students, regardless of the type of activity in which they are involved, is likely to be an effective means of improving these outcomes. It also indicates that student-centered learning should not be treated as student-only learning. In the Indonesian context, the involvement of the teachers as guides and facilitators appears to play a crucial role in this regard. Interaction in terms of words spoken Average teacher and student words per 50 minutes of public interaction 7,000 6,000 5,798 5,902 5,360 5,452 5,536 5,148 5,000 4,000 3,243 3,000 2,000 1,016 1,018 1,000 766 824 810 509 640 0 Indonesia Netherlands Switzerland Czech Australia Hong Kong United Republic States Average Teacher Words Average Student Words Main Report 115 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms Another way to gain an understanding of teacher-student interaction is through the amount of actual speaking that occurs. Counts of words spoken focused only on full-class interaction, rather than group or individual work, and were standardized to indicate the number of words spoken over 50 minutes. Indonesian teachers generally spoke far fewer words than teachers in other countries, with only 3,243 words in an average lesson, compared to a range from between 5,198 to 5,902 in other countries. The number of words spoken by students was also lower than in other countries, but the difference was less dramatic, at an average of 509 in Indonesia compared to a range from between 640 to 1,108 in other countries. Figure 40: Ratios of Words from the Teacher for Every Word from a Student Indonesia 11 Hong Kong 16 Netherlands 13 Switzerland 10 Czech Republic 9 Australia 9 United States 8 0 2 4 6 8 10 12 14 16 18 The lower rate of verbal communication by both teachers and students may signal a slower pace, or possibly less active and engaged participation. Although the number of words spoken by both teachers and students is lower than in other countries, the ratio of teacher talking time to student talking time shows that in Indonesia, teachers uttered 11 words for every word spoken by students, compared to a range from between 8 to 16 words in other countries. 8.2.4 WHAT TYPE OF INTERACTION REALLY MATTERS? While the proportion of time spent on different types of interaction can provide a starting point for understanding what sort of practices may lead to higher learning outcomes, it is what the teacher does in the available time that really matters. A particularly important practice is the use of questioning in the classroom. Students whose teachers utilized questioning to a greater extent tended to have better learning outcomes. However, the reality was that in many classrooms, questioning tended to be superficial. The majority of questions that teachers asked were testing questions that would typically require only one-word responses and little or no higher order thinking. The teachers who were most effective tended to use more advanced forms of questioning that would encourage higher order thinking. The study shows that students of teachers who employed this technique tended to achieve better learning outcomes. Unfortunately, however, the use of open-ended questions by teachers was quite rare. 116 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Themes on Teaching Practices and Policies Box: The Use of Questioning from the Perspective of the Teacher’s Goals Questioning can be a powerful instrument in a mathematics teacher’s repertoire of teaching practices, serving various purposes in defining the dynamics of the classroom and developing student learning. Through interviews and the observation of the case- study teachers in the Video Study, an interesting picture emerged regarding how teachers conceive of and use questioning. Teachers said that they used questioning to check understanding (the majority); keep the students’ attention; direct the students’ attention; and understand what students are thinking. They also explained that they called on individual students as a way of building their self-confidence, although some teachers also sometimes used it as a form of punishment involving the ritual humiliation of a student for not paying attention. Ainley (1988) developed a framework of questioning based on the teacher’s goals by which questions were categorized in the following way: • Pseudo questions: Questions intended to establish an acceptable behavior or a social contract with students • Testing questions: Questions intended to find out whether students respond correctly • Genuine questions: Questions intended to seek information (thus, teachers do not know the answers to these types of questions) • Direct or provoking questions: Questions intended to provoke students’ thinking by making new connections or clarifying existing ones and by exploring new areas of mathematical knowledge MAJORITY VERY RARE Pseudo Testing Genuine Directing Questions Questions Questions Questions Establish an The teacher knows The teacher really does The teacher wants to acceptable behavior. the answer and not know what the make the student Questions only wants to check to see student is thinking and think more about requiring or allowing if the student can wants to know or something or to think agreement provide it better understand of it in a di erent way That’s a really interesting Good answer. Now, can Arewe Are ready weready answer, but i don’t know you explain the 2 + 3 = ... ? tobegin? to begin? how you got it. Could you implications for solving explain your logic? with a di erent triangle? In Indonesia’s classrooms, the use of pseudo and testing questions predominated. However, relatively effective teachers would use genuine questions and direct or provoking questions to a far greater extent. The latter are considered to be particularly effective at encouraging higher-order thinking. That’s a really interesting answer, but i don’t know Main Report 117 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms Teachers’ explanations of their use of questioning Teachers were interviewed regarding their practices as they watched their own videos. One teacher had a particularly profound response to how she saw the role of questioning. She explained that: “Questions provide a window that let me see into the minds of my students. Their responses give me clues about how to adjust my teaching.” Teachers also saw questioning as a tool to support specific students. One teacher explained, “I called on Rivan to answer. He is behind the other students and needs to build his self-confidence. The question was easy and I was pretty sure he could answer it.” The study also highlighted the difficulty of engaging students in interaction, particularly in whole- class situations. Although there are exceptions, the Indonesian cultural norm in a whole-class setting tends to be one in which students are not accustomed to verbalizing responses that require more than one word. Cases where students spoke up without any prompting from the teacher were extremely rare. Despite the challenges, there are ultimately benefits of actively engaging students In order for teachers to transform their teaching practices from ones that involve a teacher-centered approach to ones that involve student-centered learning, a great deal of continuous training, support and materials will need to be provided. Teachers will need to change the culture of the classroom, particularly with regards to students’ understanding of their new role within a student-centered learning culture. The Video Study results indicate that the focus on greater student interaction and the encouragement of higher-order thinking are a move in the right direction. Such a move could shape the next generation, enabling its members to have greater confidence to express their ideas; a stronger ability to undertake tasks requiring higher-order thinking; and, in turn, to more effectively shape Indonesia’s future. 8.3 TEACHING PRACTICES ENCOURAGED OVER THE PAST DECADE The policies on teaching practices over the past decade have tended to encourage progressive practices that focus on active learning (as explored in detail in Section 8.2) with higher levels of student interaction and collaboration and a greater emphasis on reasoning, innovation, creativity and higher order thinking. This approach was emphasized to an even greater extent in the 2013 curriculum.39 The findings of the Video Study show that many of these practices have a positive relationship with student learning outcomes. However, the problem is that the Video Study also finds that there is a very wide gap between current practices and those expected of both students and teachers, with many of the desired practices rarely found in mathematics classrooms. A holistic approach to supporting teachers to improve their knowledge and skills will be essential in addressing this gap and could be driven by the MOEC’s new Teacher Professional Management System (TPMS), which involves a component of continuous professional development (CPD). 39 The 2013 curriculum was first rolled out to 6,221 schools in August of 2013 and then rolled out nationwide in August 2014 to all schools across Indonesia. In December 2014 the newly elected administration made the decision to halt the rollout for schools that had implemented the curriculum less than one year, citing an insufficient evaluation and rushed rollout. The 2013 curriculum continues to be used in the schools for the initial rollout while the evaluation and plans for re-rolling out to all remaining schools is in process. 118 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Themes on Teaching Practices and Policies The active learning approach The active learning approach encourages students be more interactive and collaborative as well as to seek out and make observations. Students are directed to formulate the problem (ask), not just to solve the problem (answer). They are expected to think analytically (make a decision), rather than just in a mechanistic manner (routine). Students are also expected to work together and collaborate in solving problems. The role of the teacher This more open and student-centered teaching/learning paradigm requires that the teacher become more of a guide and facilitator than simply the provider of information through one-way teaching. Classroom practice must foster independent learning, observation, questioning, processing, making judgments, communicating and creating. Practices in the context of mathematics For mathematics education, there has been an emphasis on the use of investigative and practical (hands-on) teaching approaches with more of a connection to real-world contexts. The lesson activities should encourage higher order thinking through techniques such as the use of open-ended questioning, non-routine problems, and the application of various enquiry methods. A study of practices used by mathematics teachers The Indonesia TIMSS40 Video Study examined in extensive detail the practices of 200 eighth grade mathematics teachers. The findings of the study provide useful insights into what currently takes place in Indonesia’s classrooms, with particular relevance to the 2013 curriculum. The findings indicate that many of the practices encouraged in the 2013 curriculum have a positive relationship with student learning outcomes, providing evidence that the new approaches could lead to better learning outcomes in the Indonesian context. However, the study also found that there is an extensive gap between current practice and what is expected in the 2013 curriculum, with many of the desired practices rarely found in mathematics classrooms.41 The study also goes beyond what practices teachers use and explores in detail why teachers use the teaching practices. This gives critical insights into what would be required to support teachers in actually changing what happens in the classroom. What happens in Indonesia’s mathematics classrooms? Classrooms are complex environments in which teachers must make many decisions and deal with students who learn at different paces and in different ways. Every classroom is different and each teacher faces a unique set of circumstances. While a summarized snapshot of Indonesia’s classrooms tends to hide this complexity, it is still extremely useful in understanding what teaching practices are normally used and their relative frequency. 40 Trends in International Mathematics and Science Study (TIMSS). 41 The filming of classrooms took place two years before the 2013 curriculum was introduced, so the differences are unconnected with whether teachers are now adopting the new curriculum. Rather, it provides a comparison of what practices were being used before the curriculum was introduced and can also serve as a benchmark for, at some point in the future, determining the extent of changes in practices resulting from the new curriculum. Main Report 119 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms Interaction: As was highlighted in section 8.2, in Indonesia’s mathematics classrooms, a teacher-centered approach tends to be the norm. A large proportion of the lessons are dedicated to whole-class time (where the whole class participates together), while only 15% is made up of group work, which is considered a cornerstone of a student- centered approach. The 2013 curriculum’s emphasis on collaborative learning would tend to require a greater amount of group activities. In whole-class situations (where all the students and the teacher are working together), 74% of lesson time was teacher- only, generally indicating a strong teacher-centered approach. Only 11% involved teacher-student interaction and the remaining 15% was student-only. The small amount of teacher-student interaction is an indicator that teachers tend to avoid taking on the role of guide and facilitator, as required by a more student-centered approach. Methods: In terms of teaching methods, 61% of mathematics time was devoted to exposition and another 22% to problem-solving, with discussion (9%) practical work (7%) and investigation (1%) being much less common. The 2013 curriculum for mathematics emphasizes activities such as investigation and practical (real-life) methods, which are currently very rare. Figure 41: Proportion of Lesson Time by Methods Used 9 61 1 7 22 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Discussion Exposition Investigation Practical work Problem solving Mathematical problems and questions: The approach to questions and problems provides an extremely stark contrast, with the dominant approaches being closed questioning (97% compared with 3% open-ended); routine problems (96% compared with 4% non-routine); and mathematical language and symbols (93% compared with 7% real-world contexts). Figure 42: Problem and Question Contexts Real-world contexts Open Non-routine 7% 3% 4% Math Closed Routine language 97% 96% 93% 120 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Themes on Teaching Practices and Policies A gap to fill These findings from the Video Study demonstrate that teachers will need to teach in a very different style than they do currently and that there is a large gap to fill between the ideal approach and that to be adopted. In nearly every case, the practices encouraged through policies promulgated over the past decade are rarely used. The encouraged student-centered learning approach clearly contrasts to existing practices. While problem-solving was used 22% of the time, the use of practical work and investigation, which align with the espoused practices, were rare. The use of open-ended questioning, non-routine problems and real-world contexts were particularly rare. As a whole, the results indicate that there is a significant gap between the practices actually used and those that are expected to be used. Since these policies have been in place over the past decade, the fact that they are so rarely found highlights the extent of the challenge in translating policy into actual classroom practice. Relationship of practices to student learning A critical question is whether the approaches encouraged by policy are applicable and effective in the Indonesian context. The Video Study utilized student pre- and post-testing in order to measure student learning outcomes over the course of the eighth grade year. The changes in learning outcomes were then correlated with the practices of the teachers (while controlling for a variety of student, classroom, school and community characteristics) in order to identify relationships.42 While caution should be taken in interpreting the relationships as being causal, they do provide a useful entry point for considering how practices relate to learning in the Indonesian context. The results were then further probed in a qualitative manner to better understand how certain practices are being used by different teachers and how these practices may benefit student learning outcomes. Interaction: Practices that involved increased student-teacher interaction and student-centered learning tended to have a positive relationship with student learning outcomes. Conversely, there was a negative relationship between teacher-centered activities, such as teacher-only lecturing, with student learning outcomes. There was also a positive relationship between a higher level of involvement of the teacher in monitoring group work and seatwork activities, indicating that even in activities in which students are engaging in activities on their own, they benefit from teacher support. In the Indonesian case, it appears that the teacher still has a critical role to play in monitoring and support, even in student-centered activities. • Task time: As would be expected, teachers who devoted a higher proportion of lesson time to the study of mathematics (rather than on organizational or non-mathematical activities) tended to achieve better student learning outcomes. Students with teachers dedicating a higher proportion of time to solving problems also tended to perform better. • Method: Classrooms in which the investigation method was used were rare, but when these practices were implemented, there was a positive relationship with student learning outcomes. By contrast, there was a negative relationship with the extent to which Teacher lecturing and exposition predominated. In the case of methods, such as problem-solving, practical work and discussion, no clear relationship with student learning outcomes could be determined. 42 In the analysis multiple student, classroom, teacher, school and community variables were controlled for in order to understand how practices and learning outcomes are related across contexts. Main Report 121 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms Problem and question context: Students of teachers who utilized open-ended questioning and non-routine problems to a relatively greater extent tended to achieve better learning outcomes. Interestingly, however, there was a negative relationship between the extent to which teachers used of real-world contexts (rather than the use of mathematical language and symbols) with these outcomes. Teacher knowledge and practices There was also a positive correlation between the results of the teacher competency tests in mathematics and pedagogy with student learning outcomes, as well as with the quality of teaching practices. The role of teacher knowledge is explored as a theme in Section 8.1. Of all the education system factors measured, teacher competency scores had the strongest positive correlation with student learning outcomes. This indicates the critical importance of having teachers who have a mastery both of their subject and the associated pedagogical methods. Importantly, the teachers with higher levels of knowledge also tended to use the practices encouraged over the past decade to a relatively greater extent. On examination of the videos, it is clear that the practices of these teachers are more advanced, requiring a higher level of competencies in order to use them, and to use them effectively. The greater use of student-teacher interaction, open-ended questions, non-routine problems and approaches such as investigation also require that teachers have a high level of mathematical and pedagogical competencies. In order to conduct activities involving higher order thinking, the teacher must not only be able to engage in higher order thinking him/herself, but to lead discussions and guide students to enable them to do the same. While setting routine problems that require simple answers is an easy process to control, the use of open-ended questions and non-routine problems creates a greater potential for the class to go in different directions to that planned by the teacher. Teachers with lower levels of competency tend to be more reluctant to allow themselves to get caught in these situations. A higher level of interaction with students and activities to probe student thinking and understanding require not only strong mathematical skills, but also a high level of pedagogical skills. The Video Study results indicate that those teachers with lower levels of knowledge will have particularly difficulty utilizing the more advanced practices. Even if they do adopt the practices, they are more likely to struggle to use them effectively. Not only will teachers require support in learning the new practices and the necessary associated pedagogical knowledge, but they will need to increase their knowledge of their subject as well. The findings from the Video Study indicate that there is a positive relationship between many of the practices encouraged over the past decade and through the 2013 curriculum with student learning outcomes. However, there is a large gap between the practices used and what the new curriculum expects. The results also indicate there are barriers that cannot be resolved through quick training. Change is notoriously difficult Changing teaching practices is notoriously challenging and this can be seen in previous Indonesia curriculum reforms. The 1984 curriculum reform was specifically designed for student-centered learning, but this study as well as others (e.g. Somerset 1997, Hendayana et al., 2003; Karnasih and Soeparno, 1999; Soedjadi, 2000; Fauzan et al., 2010) indicates that the teacher-centered approach held strong. Teachers take the information and interpret it in the context of their existing beliefs and environmental situation. With weak or limited training, teachers will be even less likely to change their practices. With limited support, teachers will often not implement in the intended manner, possibly with detrimental results. 122 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Themes on Teaching Practices and Policies Effective change requires alignment with teacher knowledge and beliefs A useful model for understanding teachers’ use of practices is to also consider two major influences on their practices: their subject and pedagogical knowledge, and their beliefs of how students learn and how to best teach. Teachers will attempt to align their practices to their knowledge and beliefs. The Video Study results indicate that it is when the practices are aligned with knowledge and beliefs that teachers are most effective. When they have to utilize practices for which they do not have sufficient mathematical or pedagogical knowledge, or that do not align with their beliefs on the teaching-learning process, they will not be effective. For example, teachers forced to use a student-centered approach where they are a guide and facilitator rather than an instructor will struggle if they do not know how to use the approach and/or they do not believe this is the way that students learn. Figure 43: Theoretical Model Showing Alignment of Teaching Practices with Knowledge and Beliefs PRACTICES Dissonant Zone PRACTICES Congruent Congruent Zone Zone KN OW KN S EF OW LE LI BE DG LE E S DG EF E LI BE Teacher using many practices not aligned Teacher whose practices are mostly with her knowledge set or beliefs aligned with her knowledge set or beliefs The implications of this model for the 2013 curriculum are crucial. If teachers do not have sufficient knowledge in the new approaches and they do not believe they are effective approaches, then teachers will do one of two things: (i) they will either revert back to the practices that do align with their knowledge and beliefs; or (ii) they will continue to use the practices, but use them ineffectively. This second scenario could be even more detrimental in the short term than if teachers use their normal practices. After the initial hurried national rollout of the 2013 curriculum, it has been put on hold while a proper evaluation can be conducted. The 2013 curriculum has the potential to significantly change what takes place in Indonesia’s classrooms and to improve student learning outcomes, but the extreme gap between teaching practices found before the 2013 curriculum was implemented and what is expected of both students and teachers following its implementation must be bridged. A holistic approach to supporting teachers in improving their knowledge and skills is essential and can be driven by MOEC’s new Teacher Professional Management System, which involves a Main Report 123 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms component on Continuous Professional Development. It involves identification of competencies both through subject competency testing and through classroom observation. These tools can help in identifying where teachers can improve, including with 2013 requirements, and then support teachers in their improvement. 8.4 THE NATIONAL EXAM AND OTHER INFLUENCES ON TEACHING PRACTICES Teaching practices are influenced by a large number of factors. System and policy influences could come in the form of curriculum changes, textbook changes, new training offered in pre-service or in-service programs and activities related to what is emphasized in teacher appraisal. Changes may also come through less direct interventions, such as modifications to the content and level of emphasis of the national exam, influences of teaching peers, or a teaching methodology that happens to be hot or trending. However, while educational ministries and other stakeholders often attempt to change what happens in the classroom, actually achieving this is notoriously difficult. There have been many failed attempts to implement in-service training programs or policies that emphasize the use of certain approaches or practices. Simply learning about a new practice or approach is typically not enough for a teacher to change his or her behavior in the classroom. Education system trends or major reforms may shift the system through multiple influences. Indonesia’s teacher reform effort and the passage of the 2005 Teacher Law was intended to improve teacher quality, with a cornerstone feature of the reform being teacher certification. 8.4.1 WHAT DO THE TRENDS TELL US ABOUT TEACHER REFORM AND OTHER INFLUENCING FACTORS? Indonesia’s Teacher Reform and Teaching Practices Unlike an intervention whose specific goal is to encourage the use of specific practices, Indonesia’s reform did not provide any explicit training that emphasized specific methods. Rather, the reform encouraged teachers to upgrade their qualifications. Since the reform began, a much larger proportion of the teacher workforce now has a 4-year degree (an increase from 29% in 2006 to more than 65% in 2012) and has been certified (with nearly half of all teachers having obtained certification by 2012). Such shifts might possibly reflect a change in what happens in the classroom. The payment of a bonus that effectively doubles the teacher’s salary following certification could also have effects on teaching practices by eliminating the need for a second job, increasing motivation, and so on. A Study to Measure Trends in Teaching Practices The Indonesia Video Study of TIMSS Eighth Grade mathematics classrooms measured multiple types of classroom practices, both in terms of frequency and as a proportion of lesson time. While many practices showed little change, some trends did emerge. Surprisingly, many of the trends went in the opposite direction of what might have been expected. The following brief presents the trends in the period from 2007 to 2011 and considers why they might have occurred. 124 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Themes on Teaching Practices and Policies Trends in Interaction The Video Study explored the interaction between teachers and students in depth in order to understand levels and types of participation. The first, top layer of interaction considered was whether mathematics time took place in whole class (all in the classroom participating together), group work (with multiple groups, typically of three to six students) or individual seatwork (with students conducting tasks on their own). In both phases, the time allocated to whole class activities took up most of the class time, at 59% in 2007, increasing to 64% in 2011. In comparing this proportion to the seven countries that participated in the 1999 TIMSS Video Study, Indonesia’s whole class time was in the middle, with the country average being 63%. Group work made up 21% in 2007, but decreased to 15% in 2011. This result was somewhat surprising, since the general emphasis in pre-service and in-service programs is on encouraging encourage group work and collaboration. Figure 44: Proportion of Mathematics Time by Structure of Interaction 2007 59 21 20 2011 64 15 21 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Whole Class Group Work Individual Seatwork When breaking down the above three components further into the specific types of teacher-student interactions that take place, further interesting patterns emerge. When looking only at the whole-class time (which was 59% in 2007 and 64% in 2011), the trend clearly indicates an increase of teacher only time from 61% to 74%. The increase came mainly at the expense of a large decrease in teacher and student interaction, which fell from 21% to 11%. Figure 45: Proportion of Whole Class Time by Interaction Type 2007 62 38 2011 52 48 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Teacher monitoring/facilitating Students only When looking at the proportion of group work in terms of whether the teacher either monitors and facilitates (going from group to group to check progress and possibly provide feedback and guide the groups) rather than just letting the groups function on their own, the trend is for teachers to do less monitoring and facilitating. While 62% of time in 2007 involved monitoring and facilitating, this dropped to 52% in 2011. Main Report 125 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms A less pronounced but similar trend can be found for individual seatwork, where a slight drop from 40% to 37% was registered. Are these trends good or bad? An important point in examining teaching practice proportions is that there is no right or wrong practice. In fact, teachers will often mix their practices, using different approaches within and across lessons. The “correct” method is context-specific and will depend on the stage of the lesson, the students’ abilities and other factors. Still, relating practices to student learning outcomes provides general clues as to what more effective teachers may utilize or what may generally work well in the Indonesian context. Proportions were analyzed lesson-by-lesson and compared with pre- and post-test scores given at the beginning and end of the eighth grade year to measure student learning outcomes. For proportions of whole class, group and individual seatwork, no statistically significant relationships emerged. Interestingly, however, a clear pattern emerged among all three sub-layers. In whole-class interaction, there was a statistically significant negative relationship with teacher-only interaction and a positive relationship with teacher and student interaction. For group work and individual seatwork, there was a statistically significant relationship with teacher monitoring and facilitating, while there was a negative relationship with student-only time. This general result across all sub-layers appears to indicate that students in classrooms where there is more teacher- student interaction tend to perform better. A teacher-centered approach in which students are less active appears to be relatively less effective than our more student-centered approaches. This finding has important implications for the 2013 curriculum, under which student-centered approaches and more active learning are emphasized. So, in terms of trends, it appears that the extent to which those practices associated with positive learning outcomes decreased, while those associated with negative outcomes increased. Again, the use of particular practices is contextual and cannot be easily labeled as good or bad. Still, the general trends combined with their relationships to student learning outcomes are not promising. Trend in Teaching Method Teaching methods were broken down into various types based on the teaching method or activity involved. The major shift took place in terms of the proportion of time involving exposition or lecturing, which increased from 52% to 51%. The proportion of time involving the practices of discussion, practical work and investigation, on the other hand, all decreased. Figure 46: Proportion of Teaching Methods and Approaches Used 2007 13 53 3 10 21 2011 9 61 1 7 22 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Discussion Exposition Investigation Practical work Problem solving 126 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Themes on Teaching Practices and Policies Again, the relationships of these practices present an unappealing picture. There was a negative relationship between the use of Exposition with student learning outcomes, while there was a positive relationship with the use of investigation. The proportion of time involving the use of Problem solving, which has a positive relationship with student learning outcomes, remained unchanged between the two points in time. Trends in Problem Time and Approach The proportion of time spent on studying mathematics that was allocated to solving problems relative to non- problem time (explaining mathematical concepts and other mathematics-related work not specifically involving problems) decreased significantly over the two points in time, from 76% to 64%. In 2007, this proportion was already lower than the other countries, but by 2011 proportion, the proportion was 18% less than the average for other countries. Practices are interrelated and the increase in time allocated to Exposition tended to involve more lecture time by the teacher. This in part explains the decline in problem time. Teachers tended to spend more time explaining concepts in whole-class time, with less time spent on group and individual seatwork also leading to less problem time. Figure 47: Proportion of Mathematics time as Problem vs. Non-problem time Indonesia 2007 76 24 Indonesia 2011 64 36 Japan 82 18 Hong Kong 85 15 Czech Republic 85 15 Australia 85 15 Switzerland 87 13 United States 89 11 Netherlands 96 4 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Problem Non-problem As with the previously presented results, the trend appears to be going in the wrong direction, given that there is a positive relationship between problem time and student learning outcomes. Mathematical Problems by Type When examining the types of problems and the language used, there is little variation between the two points in time, with problem types being largely routine, closed and using mathematical language rather than non-routine, open and involving real world contexts (applications) at both points. However, a couple of important trends did emerge. Main Report 127 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms Routine versus Non-routine problems While problems could predominantly be classified as routine in 2007, with 88% of problems falling into this category, by 2011, this had increased to 96%. Time spent on non-routine problems, which would tend to involve a greater degree of complexity and higher order thinking, only made up 4% of the total. Figure 48: Proportion of Problem Work as Involving Routine versus Non-Routine Problems 2007 88 12 2011 96 4 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Routine Non-routine Mathematical Language versus real World Contexts When classifying problem time in terms of the problem context, the large majority involved mathematical language and symbols. In 2007, the average time spent on these problems was 89%, increasing to 93% by 2011. These problems tend to be procedural problems. Problems using real world contexts, on the other hand, were rare, making up only 7% of the total by 2011. Figure 49: Proportion of Problem Work as Involving Mathematical Language vs. Real World Problems 2007 89 11 2011 93 7 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Mathematical language and symbols Real world When considering the relationship between these approaches to mathematical problem with student learning outcomes, mixed results emerge. There was a positive relationship between the use of non-routine problems and student learning outcomes. This, combined with the fact that problems of this type would tend to be more complex and involve a greater degree of higher order thinking, indicate that the trend is going in an undesirable direction. On the other hand, there was a negative relationship between the use of real world contexts and student learning outcomes. In this case, the increase in the use of mathematical language and symbols would be a positive trend. However, this goes counter to MOEC policies and training activities, which tend to encourage the use of real-world contexts (Contextual Teaching and Learning). 128 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Themes on Teaching Practices and Policies Interpretation of this category is particularly challenging and raises fundamental issues when consideration is given to what is tested. One important consideration relates to the test used to measure student learning outcomes. The items were drawn from the item bank for the national examination and also included TIMSS, which both tend to emphasize procedural problems. While a few of the problems used in the pre and post tests involved real world contexts, the majority were procedural and could therefore favor students who were exposed to such problems to a greater extent. When examining learning outcomes involving only the real world problems in the test, the students whose teachers tended to use real world problems in their lessons performed better, although the result did not reach the level of statistical significance. At the very least, however, they did not perform less well. Thus, it is important to consider what is emphasized on the test in terms of what sort of outcomes might be obtained. Would Certification Explain the Trends? The general hypothesis posed at the beginning of the study was that the teacher reform could impact what happens in the classroom and that this could be seen through trends in teaching practices. The teaching practices of certified and uncertified teachers may differ because of either varying levels of teacher motivation or fundamental underlying differences between the two groups. Certified teachers may alter their teaching practices because of the higher motivation provided through the incentive than increased salary, an increased in status, or an increased sense of obligation. Such teachers might also dedicate more time to teaching because they do not feel the need for a second job, which could conceivably lead to changes in the way teachers plan and execute their lessons. Figure 50: Differences in Teaching Practices of Certified Teachers vs. Uncertified Teachers in Indonesia TIMSS Video Study of Mathematics for the 2011 Phase Public: teacher and student Private (group or seatwork) Mathematical organization Question and answer Public: student only Public: teacher only Public (whole class) Nonmathematical Problem solving Math language Practical work Mathematical New concept Investigation Nonproblem Assessment Nonroutine Discussion Real world Exposition Rhetorical True-false Problem Practice Routine Review Closed Open 4 Direction and level of statistical signi cance Thresholds of statistical signi cance 3 1% 2 5% 10% 1 0 (1) 10% (2) 5% 1% (3) (4) Note: Bars that pass the 5% threshold are considered to be statistically significant. Main Report 129 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms In the 2011 sample, 53% of teachers were certified, providing a nice balance for comparative purposes. However, in looking at teaching practices, the striking feature is the lack of statistically significant differences between certified and uncertified teachers. Figure 51 shows results in which the teaching practices were placed one at a time into a regression model that controlled for multiple contextual variables. Only one difference emerged: certified teachers tended to use whole-class interaction to a relatively greater extent and private (group and seatwork) interaction to a relatively lesser extent. It appears that certification could in part explain the trend towards increased whole-class time, but would not explain any other trends. These results appear to indicate that certification itself did not have an impact on teaching practices and behavior. Considering that the reform did not emphasize a specific change in teaching approach, this lack of difference should not be considered particularly surprising. Almost every teacher who went through the certification process actually obtained their certificate, so there was no separation of teachers during the process in terms of quality.43 Although approximately two-thirds of the teachers took an additional 90-hour course, such training would be highly unlikely to alter teaching practices in any measurable way. The certification study found that certified teachers were less likely to hold a second job, but it did not identify any other differences in teacher behavior such as number of teaching hours, levels of absenteeism, or student outcomes. The Video Study accords with those results by further indicating that there are no discernible differences in the practices of the two groups. What are the Implications for the 2013 Curriculum Reform? These problem context and approach categories bring up interesting aspects related to testing, raising questions regarding what is truly being measured and how teachers might change their teaching practices to better assist their students to succeed in high-stake assessments, such as Indonesia’s national examination. Teachers seem to have moved towards more routine, formulaic, and possibly rote learning approaches. These actions could be considered “teaching to the test.” The increase in mathematical language and the use of routine approaches and closed problems and questions may have to do with an increased emphasis on performing well in the national examination, where questions typically require a routine approach in their solution. There was a positive relationship between teachers’ self-reported influence of the national examination and their use of routine problems, procedures and mathematical language contexts. Teachers were asked whether the national examination influenced their teaching. Those who said they were more influenced by the national examination also tended to use routine problems, closed problems, and mathematical language contexts, which is evidence that teachers see these practices as being an effective means of preparing students for this examination. In addition, the percentage of teachers who said they were highly influenced by the national examination increased from 70% to 75% in the period from 2007 to 2011, which indicates the national examination is increasingly influencing how teachers prepare and implement their lessons. This finding raises an important contradiction. Many educationalists argue that the teaching approach encapsulated in these routine, closed, and formulaic techniques does not promote true understanding of mathematics. Still, if teachers are “teaching to the test” and these approaches tend to help students score better, is it not a wise strategy? Could it, in fact, lead to better learning, albeit more through a mechanistic, rote-learning approach? The results regarding non-routine and open approaches are an indication that the less mechanistic approach could, in fact, lead to better learning, but it is used infrequently. 43 The queuing criteria such as experience and educational level did mean that differences do exist between certified and uncertified teachers, but these factors were controlled for in the analysis. 130 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning CHAPTER 9 OVERARCHING CONCLUSIONS FROM THE STUDY FINDINGS A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms The Video Study provides many insights into the intricacies of the teaching-learning process, generating significant insights what takes place in Indonesia’s mathematics classrooms, why it takes place, and how it relates to student learning outcomes. The following are the overarching messages and conclusions from this study: 1. Many of the practices encouraged through policies promulgated by the MOEC over the past decade, and particularly through the 2013 curriculum, have a positive relationship with student learning outcomes, indicating that proper implementation of these approaches should improve student learning. 2. However, teachers rarely use these practices, indicating that simply promulgating policies does not lead to teachers utilizing the practices implied by those policies. Rather, it takes a major effort to achieve change in terms of methods and practices. Teachers will need strong support if they are to understand and utilize the practices described by the policies properly. 3. The strong, positive correlation between teacher subject and pedagogical knowledge on the one hand and student learning outcomes on the other, indicates that this knowledge is a critical element in teacher effectiveness. However, the national teacher competency test indicates that in general, levels of teacher subject and pedagogical knowledge are very low. Support to in-service teachers to improve their knowledge and targeted hiring to ensure that only teachers with a certain level of competency are hired would improve the quality of the teacher workforce. 4. Teacher-student interaction is an important element of the student learning process. Students in classrooms with a greater level of teacher-student interaction achieve better learning outcomes. This holds true whether the students are involved in whole-class, group work or individual seatwork activities. There tends to be a negative correlation between the extent to which teacher centered practices such as teacher-only talking time are used and student learning outcomes. 5. The most effective teachers in Indonesia tended to have a flexible and adaptable beliefs structure rather than falling into a single orientation. Generally, teachers who had a more transmissionist belief system, where the role of the teacher is to be the source of knowledge and traditional practices of lecturing and rote learning, also tended to have lower learning outcomes of their students. Teachers who tended to have a more connectionist belief system, where the teacher’s role as facilitator and student-centered learning approaches are dominant, tended to have higher learning outcomes of their students. But the most striking finding on beliefs was that teachers who had a flexible belief system and saw the need to adapt teaching practices to each unique context were the teachers whose students tended to have the highest learning outcomes. These teachers saw the need for more traditional methods in some instances and more progressive methods in other instances. These teachers tended to use a wider range of practices in their lessons. They were highly aware of their own students’ needs and attempted to tailor their lessons in accordance with those needs. 6. In the period from 2007 to 2011, there has been a decline in the extent to which many practices that have a positive relationship with student learning outcomes are utilized. The National Exam and the curriculum have had an increased influence on the choice of teachers in classroom practices and appear at least to some extent to be driving the negative trends. 132 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Overarching Conclusions from the Study Findings 7. According to the interactive model of beliefs-knowledge-practices produced through this study, teachers are most effective when their teaching practices align with their levels of mathematical knowledge in teaching (MKiT) and mathematical beliefs, so that they are operating in the “congruence zone”. They are least effective when they are pushed to use practices that are not aligned with their knowledge or beliefs, or when they are operating in the “dissonance zone”. This has two important implications. First, the concept of “best practices” is relative rather than absolute. What would be considered the best teaching approach may not be the same for all teachers. Second, if policies encourage and require teachers to use specific practices, such as active learning, these policies must be accompanied with the provision of professional development support that helps teachers to build the knowledge required to effectively implement these practices and to understand their merit so that it develops and engrains a belief in teachers that it is an appropriate practice. 8. On the basis of this study, it can be seen that teachers need support to develop skills in the following priority areas: • Engaging students actively in the learning process; • Paying attention to the language used; • Paying attention to the accuracy of the mathematical content (e.g. What is volume? What is 3D space?); • Enriching teachers’ roles. Teachers need to move from direct teaching to posing “strategic questions”; to listening to students carefully; and to eliciting students’ understanding; • shifting the emphasis of their teaching from a very procedural pattern, to a more investigative and meaningful pattern; • Incorporating non-routine tasks into lessons to make these tasks more challenging and more productive of higher-order thinking. 9. Teachers require support if they are to change their practices. Results from a pilot geometry module (presented in the next section) implemented through KKG/MGMP indicate that continual support through the provision of “in-on-in” activities and practical applications appear to be an effective way to change teacher beliefs; to improve levels of teacher knowledge; and to change practices in the classroom. Main Report 133 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms 134 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning CHAPTER 10 TURNING STUDY FINDINGS INTO PRACTICE: VIDEO STUDY RESULTS CONVERTED INTO A CPD MODULE A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms The insights gained through the Video Study highlighted the general importance of strengthening both teacher subject knowledge and pedagogical knowledge. It also identified specific areas where teaching practices should be enhanced. The study has also become particularly relevant in the context of the 2013 curriculum, which emphasizes teacher practices such as discovery learning, problem-based teaching and higher order thinking. A natural extension of the study was to incorporate the lessons learned into modules that teachers could use in their Continuous Professional Development (CPD) activities. More than 600 hours of video footage captured activities conducted in natural classroom time, providing useful real-life examples of what teachers do in their classroom. This unique and invaluable data provided a perfect platform upon which to develop teacher professional development activities. 10.1 USING KEY STUDY FINDINGS TO IDENTIFY AREAS FOR IMPROVEMENT The impetus for the creation of a module came from the findings of the Video Study and the desire to utilize these findings to better practice. Through an understanding of relationships with student learning outcomes and by witnessing how teachers use the practices through in-depth analysis of the videos, the key challenges and opportunities were identified. The challenge(s): Indonesian teachers typically possess low levels of mathematics subject knowledge which impacts on what they teach and the way they teach. Teachers spend considerable time introducing new concepts; less time on practicing the concepts; and very little time on assessment and reflection. They are very teacher-centered in their approach to teaching. They rarely situate mathematics within real-life situations and they tend to use routine problems to demonstrate concepts, and hence provoke minimal higher order thinking, which is usually associated with solving non-routine problems. Teachers often use language that does not accurately convey mathematical concepts. Indonesian teachers rely on question/answer type questions, where the intent of the questioning is to test if a student knows the answer, rather than initiating discussion or seeking information related to student thinking. The opportunities: Video is a powerful medium for promoting discussion and discourse. It has the distinctive ability to capture the power and intricacy of classroom practice, since it encourages teachers to engage in comparative reflection; descriptive reflection; and critical reflection. It also affords opportunities to analyze teaching from different perspectives, including from the teacher, the student, the content (discipline) and the interactions among participants. Teachers are able to gain new insights into their own teaching practice by both experiencing the challenges of colleagues and identifying new ideas for their own practice, such as questioning techniques and new activities (Zhang et al., 2011). In order to overcome the challenges faced by Indonesian teachers, new levels of professional awareness and theorizing need to be established to encourage critical thinking and personal theorizing. The findings of the video study and the desire to put these findings into practice served as the impetus for creation of the module, which focused on developing the skills described in the introduction to this section. 10.2 AN INNOVATIVE APPROACH TO CPD THROUGH THE USE OF VIDEO The central goal of the study was to determine the level of success of a professional development initiative aimed at enhancing teaching and learning opportunities for teacher educators, classroom teachers and their students in Indonesian schools. The professional development initiative was designed around the use of classroom video data, 136 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Turning Study Findings into Practice: Video Study Results Converted into a CPD Module which was used as a catalyst to promote new pedagogical practices. The video data, and related learning framework and tools, was designed to encourage classroom teachers to reflect more deeply on their practices. This involved the establishment of teacher working groups to develop new levels of professional engagement. The main objective was to conduct a comprehensive analysis of a pilot that will determine the effectiveness of a new approach to promote new and contextually innovative teaching practices. 10.2.1 IN-ON-IN APPROACH With the stated goal of achieving behavioral change, the module was designed with an in-on-in approach, in which teachers first engage in learning activities in a working group session. Following this they are assigned with KEY DESIGN PRINCIPLES FOR BEHAVIORAL CHANGE IN TEACHING PRACTICES Behavioral change is most likely to occur when: 1. Activities to facilitate this change are conducted over an extended period of time; 2. There is an iterative cycle of learning, practice, reflection and modification; 3. There are opportunities to engage in discussions with peers; 4. The content and subject of the activities relate directly and are relevant to the teacher’s classroom context; 5. The content is practical and has practical applications, rather than being purely theoretical; 6. The approach strengthens both the teacher’s knowledge and beliefs in relation to the practices; 7. Activities relate to both subject matter and pedagogical content, so that the teacher strengthens both simultaneously. “homework” tasks to complete in their own classrooms. In the subsequent working group session, teachers share their experiences and reflect on what worked, what didn’t work, and how they could do better in the future. The in-on-in approach dictates that what is learned has to be practical and to have practical applications, rather than being purely theoretical. Through the learning process, both subject matter and pedagogical aspects are addressed simultaneously through the activities in order to build teacher knowledge and to enable them to apply this knowledge in the pedagogical context of the classroom. Teachers must be provided with reasons and a rationale for the activities they engage in, with measures to develop a foundational understanding that helps convince teachers that the practices are useful and beneficial. 10.2.2 ELPSA FRAMEWORK The ELPSA (Experiences, Language, Pictures, Symbols and Application) learning framework is embedded in the design of the module. The ELPSA framework is built on the basis of constructivist and social learning theories. The ELPSA framework views learning as an active process in which students construct their own ways of knowing and developing understanding through both individual reflection and through social interactions with others. It should be emphasized that form of learning defined by the ELPSA is not a linear process. Learning is a complex and unpredictable process that does not take place in a linear sequence. Thus, the various elements of the model should be considered as interrelated and overlapping. While the components of the ELPSA framework can be discussed individually, these components cannot be implemented in isolation. Rather, the various components are incorporated and intertwined with each other throughout the lesson. Main Report 137 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms Figure 51: Example of ELPSA for the Topic of Right Triangles Experience Students construct right-angled triangles to scale and label Language Terminology including: right angle, hypotenuse, 900 Pictures 5 reinforced 3 4 c a Students encounter triangles that b do not have a right angle as an experience to move toward trigonometry Formula c2 = b2 + a2 reinforced Application Students are given task to determine the area of the oor Symbols in their classroom Introduction of formula for area of right-angled triangle A = 1/2 bh 10.2.3 INCORPORATION OF VIDEOS INTO THE MODULE One of the advantages of developing the module on the basis of the video study is that those designing the module could draw upon more than 600 hours of footage containing real-life examples of classroom practices. Example clips were incorporated into the session activities for tools to stimulate discussion. These examples were not necessarily intended to portray best practices, but rather to serve as stimulants to provoke discussion on specific issues and subjects. 10.2.4 FACILITATORS AS AN INTEGRAL PART OF SESSION ACTIVITIES The least costly method of delivering CPD through teacher working groups would be to design the module so that it could be utilized without any facilitation. If this approach were adopted, the materials would need to be completely self-explanatory. In addition, teachers would need to have a sufficient level of motivation to conduct the activities in a fairly self-directed manner. While this option was considered, it would have severely limited the complexity of the activities included in the module. Also, it would have few if any regulators to control the quality and consistency of delivery. It was decided that the module could be developed much more effectively if it involved the selection of two members within the working group to serve facilitators and leaders, together with measures to develop the capacities of the selected two facilitators. 138 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Turning Study Findings into Practice: Video Study Results Converted into a CPD Module 10.3 IMPLEMENTATION IN TEACHER WORKING GROUPS Teacher working groups were selected as the venue for the delivery of the module. These groups were selected for this purpose primarily on the basis of the benefits of their use from the point of view of both effectiveness and sustainability. This section briefly examines the theory and core concepts of teacher professional development in order to provide a context for the implementation of the module. Teacher working groups, known in Indonesian as Kelompok Kerja Guru (KKG) when they involve primary school teachers, and Musyawarah Guru Mata Pelajaran (MGMP) when they involve secondary school teachers, are clusters of teachers within a defined geographical area. For secondary school working groups, the number of schools within a cluster tend to range from three to six, and involve an average of approximately 20 teachers. In the secondary school groups, the groups are subject specific, so, for example, a single group may consist of 20 mathematics teachers. The frequency of meetings varies, but they tend to be held every second week. Teacher working groups are intended to provide a forum for teachers to discuss issues and share ideas. They are also intended to be vehicles for professional development activities. Teacher working groups provide advantages over more traditional professional development in-service training. Working groups can provide a very different type of learning and professional development experience. They have an advantage over traditional in-service training in that: • Training activities can be performed on a regular basis (every two weeks rather than once a year); • Regular meetings allow for the reinforcement of concepts and multiple iterations of training and professional development activities; • The structure of training involves a “for teachers, by teachers” approach, which has been shown to be the most successful method for professional development; • Training activities can be more easily modified to meet the needs of a local context, particularly since all teachers come from the same general location; • Teachers form a support system upon which they can rely on and to which they have easy access; • The teacher working group methodology conforms to principles of adult learning, in that: –– Teachers are able to structure learning around their interests and needs; –– Teachers are able to apply life experience and prior knowledge to the learning situation; –– Teachers are able to be actively involved in the learning process; –– The learning style can be structured to adapt to the needs of working group members; –– Teachers are able to be self-directed; –– The working group provides an affiliation for its members; • Learning activities tend to be participatory and experiential. Main Report 139 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms 10.4 IMPACT ON TEACHER BELIEFS, KNOWLEDGE AND PRACTICES The quantitative analysis examined three areas: (i) teacher beliefs; (ii) teacher knowledge; and (iii) teaching practices. For each of these areas, data was gathered before the module began and after it ended. • Beliefs: A survey was used that asked teachers to rate statements on a scale of 1 to 5 in their level of agreement. The items covered: (i) teaching mathematics; (ii) the process of learning mathematics; and (iii) learning activities. • Knowledge: Teachers were asked to participate in pre and post-tests. The tests were different, but contained the same general question types and contained anchor items (which appeared on both the pre and post tests) and parallel items (which were similar, but were changed so that the answer would not be the same). It also contained groupings of questions, as were either directly related, indirectly related or unrelated to what was learned in the course module. For example, open-ended questioning was one of the main topics of the module. Thus, the tests contained items that would test teachers’ ability to create open-ended questions and answer open-ended questions. This would be considered directly related topics of the module. Items that are indirectly related touched slightly upon what was learned in the module, while indirect items had no relationship to the module. The mix of these questions enables an analysis to determine the direct and indirect impacts of the module, while also containing control questions, which should not change as a result of the intervention. • Practices: Students were asked to participate in a practices survey to rate on a scale of 1 to 5 the extent to which certain practices took place or to the extent to which their teacher uses techniques or be described in a certain manner. For all of these instruments, the pre result was compared to the post-result to determine whether the behavior of the treatment group had changed in relation to the control group and, if so, to what extent. Another end goal is to gain an understanding of the relationship between beliefs-knowledge-practices. • Teacher Beliefs: Treatment teachers showed a statistically significant increase in their positive responses to statements regarding hands-on activities; inquiry-oriented activities; performance-based assessment; and the use of informal questioning to assess student understanding. All areas except the performance-based assessment were part of the pilot module. A statistically significant increase was recorded amongst teachers in the treatment group for statements related to students participating in student-led discussions and working in cooperative learning groups, as well as for statements related to students working on extended mathematics investigations or projects. Again, these items relate to what was emphasized in the module. • Teacher Knowledge: The treatment group had a clear and significant increase in the teacher competency assessment, indicating that the module accomplished the goal of simultaneously developing both teacher subject knowledge and teacher pedagogical practices. Of particular interest was that teachers were much more able to address items related to open-ended questions. This improvement in terms of teachers being more comfortable with, and able to correctly answer, open-ended items is an important achievement of the module. • Teaching Practices: The results appear to indicate that teachers in the treatment group changed their practices in ways that students interpreted to be related to: (i) asking students to explain their answers and explain what they think; (ii) giving students time to explain their ideas and respecting students’ ideas and suggestions; (iii) expecting the students to work through the hard problems; and (iv) providing more feedback and using multiple ways to explain the topics. These suggest that the module was effective in increasing student involvement and in encouraging teachers to use questioning and rich problems. 140 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Turning Study Findings into Practice: Video Study Results Converted into a CPD Module The quantitative analysis indicates that the module was successful in changing beliefs; improving teacher subject knowledge; and, most importantly, changing behavior in the classroom. The qualitative data gathered through observations, interviews and surveys also provided many insights into the effectiveness of the module, both in terms of strengths and deficiencies and areas where improvements could be made. There was clearly an appreciation on the part of participants and facilitators for the content of the module. The ELPSA framework was very well received, with teachers expressing the belief that it was very relevant and helpful for them in transforming their classroom practices. The sessions on the use of questioning in the classroom and of hands-on experience were also particularly well received. Challenges also arose and many lessons were learned. 10.5 KEY LESSONS LEARNED AND CONSIDERATIONS Many lessons were learned through the implementation of the module in terms of its content, implementation and effectiveness. These lessons learned also raise questions regarding the provision of continuous professional development in general. The following is a summary of the key lessons and considerations, as well as a discussion of sustainability in terms of scaling up and expanding the practices covered in the module in the area of mathematics and for other subjects. CONTENT The content of the modules was generally very well received. There were almost no comments expressing disagreement with the module’s approach or topics and there was near universal agreement that the content was beneficial. In general, participants expressed the belief that the ELPSA framework and the topics of the module were highly relevant to their own practice. There were mixed responses regarding the level of difficulty of the content. Some participants indicated that the concepts and approaches were not entirely new, but that they still found it valuable to develop their ability to implement them in the classroom. Other teachers indicated that the content was challenging and felt that they did not acquire a sufficient level of knowledge to be confident about implementing the new practices in their own classroom. These teachers tended to regard the approaches as valuable, but indicated that they required more support if they were to implement them effectively. Some teachers also indicated that they would like to go into more depth on some of the topics. In particular, some teachers felt that it would be beneficial to spend more time on the topic of questioning and ways in which a culture of questioning could be created in the classroom. On the basis of this feedback, it might be desirable to consider expanding the module to include a second session on questioning. Another approach would be to develop a separate module consisting of three sessions that could go into much greater depth on this topic. The issue of the time required both to conduct the working group sessions and to implement the sample lesson plan also arose. It appears that in some of the sessions, the content was too dense and could not be completed effectively within the 3-4 hour duration of the working group sessions. This led to some activities not being conducted or being rushed. Revisions to the module should reduce the content to ensure that all activities can be implemented appropriately. Main Report 141 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms IMPLEMENTATION Using Working Groups for the Delivery of CPD Teacher working groups have both advantages and disadvantages as a system of delivery for CPD. Some specific issues that the pilot revealed that included the following: • The level of attendance at working groups can vary significantly: While in some working groups, teachers from all schools in the cluster attend, others rotate teachers within the school. In many districts, there is a policy of having a day dedicated to working groups for particular subjects. For example, working groups for mathematics teachers may be conducted on Thursdays, so no mathematics classes are scheduled for that day. However, in Jakarta, this is not the case, so teachers need to find time in their schedules to attend. This negatively affected the level of attendance. Attendance was also affected by conflicting obligations, as discussed in the next point. • A module implemented through a working group may be affected by teachers’ conflicting obligations: Teachers have many conflicting obligations, being required to engage in training on the new curriculum, to prepare for national examinations, and to conduct many other activities. This creates significant challenges in terms of maintaining the attendance of participants in working groups throughout the semester. • Teachers have different interests and needs, which may not all be addressed through a single module: While some topics may be of interest to most or all teachers, different teachers will have different professional development interests and needs. It may be worth considering adopting a certain degree of flexibility to enable teachers to engage only in the specific activities that meet their needs or to enable multiple sub-groups to conduct separate activities/modules. • The quality of the facilitator played a critical role in ensuring the effectiveness of working group activities: This point is discussed in greater detail below. • The support provided to working groups is not consistent across different schools and districts: School and educational agency authorities have varying perceptions of the degree to which working groups are the responsibility either of the district, the school, or the teachers themselves. One district commented that working groups are out of their jurisdiction and that the district could not oblige teachers to attend working groups. Importance of Facilitators At the design stage, consideration was given to the idea of developing stand-alone modules that would not require the training and use of facilitators, but could be used in a self-directed manner by teachers. While this approach would be less expensive and could be more easily and widely distributed to working groups, the content itself would have had to be simplified and include even more detailed step-by-step instructions. There would have also been a greater reliance on the motivation of the members of the working groups to implement the module on their own and to engage in all the associated activities. Through the piloting of the module, it became clear that the decision to train and deploy facilitators was correct, with facilitators playing a vital role in ensuring the effectiveness of the module. Training and deploying facilitators provided the necessary leadership to conduct the activities and to facilitate an understanding of the content. It also allowed for the establishment of a support network through which facilitators were able to share their experiences and pose questions with facilitators in other working groups. 142 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Turning Study Findings into Practice: Video Study Results Converted into a CPD Module In addition, the level of commitment and understanding of the facilitators were important in determining how effectively the sessions were conducted. In some cases, facilitators did not have a sufficient level of understanding of the materials. In these cases, it was found that the sessions were much less effective. This indicates the importance of facilitator training, underscoring the need to allow sufficient time to go through all key activities. The most effective sessions tended to be those in which simulations were conducted during the training process. This indicates that additional practice and a greater use of simulations in the training process would be beneficial. Training for facilitators should be more practical. Facilitators should engage in simulation sessions so they clearly understand their role in facilitating MGMP meetings, especially in providing input for the development and implementation of lesson plans. Facilitators have to prepare themselves beforehand so they understand the topic and are able to manage the meetings well. It would be good if facilitators could provide some other examples to enrich the modules. Responsibility for the Provision of CPD One challenge created by the decentralization of the responsibility for the provision of educational services to district governments is that continuous professional development is now the responsibility of these governments, through district-level educational agencies. The capacity of districts to provide high quality CPD varies, but is generally very low. How local governments perceive their role in the provision of CPD also varies. In the case of the pilot, the study team was able to meet with district-level officials and propose that teachers receive credit for successful completion of the module. But how could this process be scaled up? Under current arrangements, if a working group or set of teachers were to want to conduct the module, they would need to reach an agreement with the district to receive credit. Alternatively, should CPD be a supply-driven process in which districts offer the CPD activity to the working groups? Should it be voluntary or obligatory for all teachers to participate? The answer to these questions is beyond the scope of this study. However, challenges in the implementation of the module did underscore the need to clarify the role of the MOEC, provincial and district level agencies, and of schools and teachers in the provision of CPD, particularly in terms of how teachers will receive credit for their participation. For the delivery of CPD through online courses, such as courses offered by P4TK Mathematics, it appears that there is already a mechanism for teachers to receive credit. However, it is important for the rules and processes to be clearly defined. Educational agencies, school authorities and other stakeholders should support and opportunities for teachers to attend the MGMP activities and to create opportunities for them to do so. At present, teachers need a letter of permission from their principal to attend MGMP meetings. Teachers’ PKB activities should be mandatory and scheduled. EFFECTIVENESS The results of the impact evaluation indicate that the module led to statistically significant changes in teacher beliefs, improvements in teacher knowledge, and changes in the classroom. The qualitative results also indicate that teachers valued the professional development activities and felt that they benefited from these activities. Input from the facilitators, teachers and observers have factors to identify factors that lead to effective implementation, but also indicate that there were aspects that hindered the effectiveness of the module. Main Report 143 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms Working group meetings tend to be most effective when these conditions are present: (i) the topics meet with the needs of the participants; (ii) the facilitators prepare and perform the activities well; (iii) learning tools, stationeries, laptops, projector, and other facilities are available; (iv) all participants are actively involved in discussions, questioning, engaging in practice, and exploring through trial and error; (v) the activities can be effectively implemented within the allocated time (180-240 minutes). A key area for revision relates to be content load, given the time constraints that apply to working group sessions. In particular, the content of the module has to be adjusted to achieve a better fit with the time available so that the teacher learning process is more effective. Most RPP analysis, which was intended to be conducted during the sessions, had to be assigned as additional homework because of the lack of time. Each module required 3-4 hours of intensive activities, with 15 minutes break each session. Regular MGMP meetings usually run for four hours, but with a slower pace of activities. Specifically, Session 2, on questioning, could be conducted over two meetings, since the activities required more time than most others. Session 5, on enquiry learning, could also be conducted over two meetings. SCALING UP AND SUSTAINABILITY There is significant interest in CPD modules of this type. Teachers who participated in the pilot were generally very enthusiastic and requested additional support. Teachers in the control group were also very interested in utilizing the module in their working group in the future. Many of the educational agencies in the participating districts are developing plans to support the rollout of the module to the rest of the MGMP in their district. In dissemination events with district officials and teachers, there were many requests for support. However, despite this enthusiasm, scaling up the implementation of the module requires careful consideration. 10.6 WIDER POLICY IMPLICATIONS EMERGING FROM THE PILOT Policies related to teaching practices over the past decade have tended to emphasize more modern, student-centered teaching approaches, with these approaches involving a higher level of student interaction and a more intense focus on reasoning and higher order thinking. For example, Regulation 19/2005 states: “The process of learning in the educational unit should be organized to be interactive, inspiring, fun, challenging, motivating learners to actively participate and provide enough space for innovation, creativity, and independence in accordance with their talents, interests, and physical and psychological development of learners.” A similar statement was also included in Regulation 32/2013. The video study results indicate that eighth grade mathematics teachers tend to use traditional methods and that the approaches tend to be procedural, rather than encouraging reasoning and higher order thinking. Importantly, the results of the video study indicate that when teachers used more interactive methods that encourage reasoning, students tend to have better learning outcomes. This shows that there are benefits to the teaching methods that have begun to be encouraged over the past decade. However, it is also clear that in mathematics classrooms (and likely with other subjects), there is a gap between the methods that are encouraged and methods that are actually put into practice. How can teachers incorporate more progressive methods into their arsenal of practices? Teachers clearly require support, and there are no quick fix solutions. Changing practices is notoriously difficult and requires long-term, continuous, and intensive support. 144 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Turning Study Findings into Practice: Video Study Results Converted into a CPD Module This pilot was specifically designed to strengthen teacher knowledge and to improve teaching practices in the areas that encourage greater student interaction and that utilize inquiry-based methods that promote student reasoning. The results indicate that the approach used in this model changed teacher beliefs: improved teacher knowledge; and led to changes in practices used in the classroom. Although the results were positive, a module of this type must be seen merely as a seed for the development of a much larger support system. Teachers will need a much higher level of support provided on an ongoing basis in order to fully build and strengthen their subject and pedagogical knowledge. It is clear that the system for the provision of CPD at present is not working effectively in Indonesia. Since decentralization, teachers have received less CPD and the quality and relevance of the offerings has declined. While district governments are responsible for the provision of CPD, the majority do not have sufficient capacity or the willingness to provide opportunities for effective CPD to teachers. In some cases, local governments do not even necessarily consider it their obligation to provide CPD. Even for those that do see it as their obligation, in many cases, the offerings tend to be haphazard, of low quality and of limited relevance. Very often, they consist of short workshops in which a speaker is brought in to give a presentation. While teachers receive credit for participation in these workshops, they often fail to meet the needs of teachers and have little impact on classroom practices. At least in the short-term, it is not realistic to expect local governments to be able to develop high quality CPD activities. In addition, a larger framework is required to give structure to CPD as a whole, rather than having patchy, poorly integrated offerings. The P4TK Mathematics centers do tend to offer good courses and programs, but these benefit only a small number of teachers each year. With a country as large and diverse as Indonesia, in-house P4TK training at their centers for the majority of teachers is simply not a feasible large-scale solution. Online methods may provide a way to reach a larger number of teachers, with the pilot geometry module serving as an example of how online methods may be effectively utilized. This methodology certainly has good prospects, but it cannot be relied upon as the only method for the provision of CPD. Teacher working groups offer many advantages, as specified in detail in this report. Among other advantages, they can be a very effective means of building skills through the in-on-in process, which ensures that what is learned can be applied in the classroom and then reflected upon with fellow teachers. However, there are many challenges to the use of teacher working groups as a large-scale method of providing CPD. Working groups function very differently from district to district, or even from school to school. This makes it difficult to develop and implement a program consistently. It may also necessitate the development of high quality modules that can be run by teachers themselves. The development of the geometry module used in this pilot is required extensive time and resources. However, the payoff is that it can have a significant positive impact in terms of improving the level of teachers’ subject knowledge and pedagogical knowledge. Most importantly, it may result in real positive changes to what occurs in the classroom. There can be no single method for the provision of CPD. A holistic approach, with many different offerings that utilize a range of different methodologies, is necessary. Within Indonesia’s teacher professional management system, individual teachers are required to develop an annual CPD plan. To achieve this effectively, they should have a catalog of offerings that can be used to address their specific needs. This requires a system that provides many options under a holistic dynamic pedagogies framework. This CPD module has planted a seed for the professional development of mathematics teachers. The approach adopted through the module will be continued through a project in NTB under a DFAT Global Partnership for Development (GPFD) grant. However, such approaches need to be applied for teachers of other subjects and for every level of schooling, from early childhood through senior secondary. Main Report 145 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms The central government is best positioned to develop the necessary framework and to produce high quality activities that can be implemented across Indonesia. However, ultimately, it is districts that are responsible for the provision of CPD. In this capacity, they must provide support for their teachers by establishing mechanisms that will give access to and credit for CPD activities. None of this is easy, requiring inputs and support from many stakeholders. However, it is critically important if Indonesian students are to benefit from improved learning outcomes. For more than a decade, Indonesian teachers have been provided with very little support and only limited opportunities to develop the skills necessary to improve Indonesia’s human capital. The level and quality of teachers’ subject knowledge and pedagogical knowledge is still generally very low, which ultimately impacts the ability of Indonesian students to grow up and become productive citizens. It is vitally important for the nation’s future to address this. 146 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning REFERENCES A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms Abelson, R.P. (1979). Differences between belief and knowledge systems. Cognitive Science, 3 (4), pp. 355-366. Ainley, J. (1988). Perceptions of teachers’ questioning styles. Proc. 12th Conf. of the Int. Group for the Psychology of Mathematics Education, pp. 92. Ainley, J. (1987). Telling questions. Mathematics Teaching, 118, pp. 24-26. Ainley, M. (2006). Connecting with learning: Motivation, affect and cognition in interest processes. Educational Psychology Review, 18 (4), pp. 391-405. Ali, R.M. (2007). Teacher talk in mathematics classrooms: Questioning to establish procedural competence. Proceedings of the Second International Conference on Science and Mathematics Education, pp. 342. Anderson, D.S. and Piazza, J.A. (1996). Changing Beliefs: Teaching and Learning Mathematics in Constructivist Preservice Classrooms. Action in Teacher Education, 18 (2), pp. 51-62. Andrews, P. (2010). Comparing Hungarian and English mathematics teachers’ professional motivations. In V. Durand-Guerrier, S. Soury-Lavergne & F. Arzarello (Eds.), Proceedings of the Sixth European Conference on Research in Mathematics Education (CERME 6, January 2009), (pp. 2452-2462). Lyon, France: Institut National de Recherche Pedagogique. Andrews, P. (2009). Comparative studies of mathematics teachers’ observable learning objectives: validating low inference codes. Educational Studies in Mathematics, 71 (2), pp. 97-122. Andrews, P. (2007a). Mathematics teacher typologies or nationally located patterns of behaviour. International Journal of Educational Research 46, no. 5: 306-18. Andrews, P. (2007b). The curricular importance of mathematics: A comparison of English and Hungarian teachers’ espoused beliefs. Journal of Curriculum Studies 39, no. 3: 317-38. Andrews, P. and Hatch, G. (2000). A comparison of Hungarian and English teachers’ conceptions of mathematics and its teaching. Educational Studies in Mathematics, 43 (1), pp. 31-64. Arvold, B. and Albright, M. (1995). Tensions and Struggles: Prospective Secondary Mathematics Teachers Confronting the Unfamiliar. Askew, M., Brown, M., Rhodes, V., Johnson, D. and Wiliam, D. (1997). Effective teachers of numeracy. London, King’s College London. Atweh, B.D. and Abadi (2012). Investigating Teachers’ Pedagogical Beliefs in Indonesia and Australia, The Asia- Pacific Education Researcher, Vol 21, No 2 Aubrey, C. (1993). An investigation of the mathematical knowledge and competencies which young children bring into school. British Educational Research Journal, 19 (1), pp. 27-41. 148 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning References Ajzen, I. and Fishbein, M. (1980). Understanding attitudes and predicting social behaviour. Prentice-Hall Englewood Cliffs, NJ. Ball, D.L. (1991). Research on teaching mathematics: Making subject matter knowledge part of the equation. Advances in research on teaching, 2, pp. 1-48. Ball, D.L. (1991). Research on teaching mathematics: Making subject matter knowledge part of the equation. Advances in research on teaching, 2, pp. 1-48. Ball, D.L. (1990). Prospective elementary and secondary teachers’ understanding of division. Journal for Research in Mathematics Education, , pp. 132-144. Ball, D.L. (1988). Unlearning to teach mathematics. For the Learning of Mathematics, , pp. 40-48. Ball, D.L., Lubienski, S.T. and Mewborn, D.S. (2001). Research on teaching mathematics: The unsolved problem of teachers’ mathematical knowledge. Handbook of research on teaching, 4, pp. 433-456. Ball, L., Thames, M. and Phelps, G. (2008). Content Knowledge for Teaching: What Makes It Special? Journal of Teacher Education, 59 (5), pp. 389. Barkatsas, A. and Malone, J. (2005). A Typology of mathematics teachers’ beliefs about teaching and learning mathematics and instructional practices. Mathematics Education Research Journal, 17 (2), 69-90. Baumert, J., Kunter, M., Blum, W., Brunner, M., Voss, T., Jordan, A., Klusmann, U., Krauss, S., Neubrand, M. and Tsai, Y.M. (2010). Teachers’ mathematical knowledge, cognitive activation in the classroom, and student progress. American Educational Research Journal, 47 (1), pp. 133. Bennett, N., Summers, M. and Askew, M. (1994). Knowledge for teaching and teaching performance. Look before you leap, pp. 23-36. Bereiter, C., Scardamalia, M., De Corte, E., Verschaffel, L., Entwistle, N. and van Merriënboer, J. (2003). Powerful Learning Environments: Unravelling Basic Components and Dimensions. Berk, R.A. (1988). Fifty reasons why student achievement gain does not mean teacher effectiveness. Journal of Personnel Evaluation in Education, 1 (4), pp. 345-363. Berkowitz, S. (1997). Analyzing qualitative data. In J. Frechtling and L. Sharp (Eds.), User-friendly handbook for mixed method evaluations. Arlington, VA: Division of Research, Evaluation and Communication, National Science Foundation. Retrieved February 10, 2013, from http://www.nsf.gov/pubs/1997/nsf97153/start.htm Beswick, K. (2012). Teachers’ beliefs about school mathematics and mathematicians’ mathematics and their relationship to practice. Educational Studies in Mathematics, 79(1), 127-147. Beswick, K. (2008). Middle School Mathematics Teachers’ Knowledge for Teaching Percent. 11th Annual Congress on Mathematical Education, Mexico City, Mexico. Main Report 149 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms Beswick, K. (2007). Teachers’ beliefs that matter in secondary mathematics classrooms. Educational Studies in Mathematics, 65 (1), 95-120. Beswick, K. (2005). The beliefs/practice connection in broadly defined contexts. Mathematics Education Research Journal, 17 (2), 39-68. Bjork, C. (2005). Indonesian education: Teachers, schools, and central bureaucracy. Routledge. Borko, H. (2004). Professional development and teacher learning: Mapping the terrain. Educational Researcher, 33 (8), pp. 3-15. Blum, W., Baumert, J., Neubrand, M., Krauss, S., Brunner, M., Jordan, A. and Kunter, M. (2005). COAKTIV–A Project for Measuring and Improving the Professional Expertise of Mathematics Teachers. 15th ICMI Conference on “The Professional Education and Development of Teachers of Mathematics”(2005, 15-20 May, Brazil). Boaler, J. (1998). Open and closed mathematics: Student experiences and understandings. Journal for research in mathematics education, pp. 41-62. Borich, G.D. (1996). Educational testing and measurement: Classroom application and practice. Harper Collins. Brophy, J. and Good, T.L. (1986). Teacher behaviour and student achievement. Handbook of research on teaching, 3, pp. 328-375. Brown, D.F. and Rose, T.D. (1995). Self-Reported Classroom Impact of Teachers. Action in Teacher Education, 17 (1), pp. 20-29. Bruns, B. and Luque, J. (2014), Great Teachers: How to Raise Student Learning in Latin America and the Caribbean. World Bank, Washington, D. C. Butt, R.L. and Raymond, D. (1989). Studying the nature and development of teachers’ knowledge using collaborative autobiography. International Journal of Educational Research, 13 (4), pp. 403-419. Calderhead, J. (1996). Teachers: Beliefs and knowledge. Carroll, J.B. (1963). A Model of School Learning. Teachers College Record 64 (8): 723–33. Chan, C. C. and Wong, A. K. C. (1991). Knowledge Discovery in Data Bases: 107-123. Clark, C. and Peterson, P. (1986). Teachers’ thought processes’ in MC Wittrock (ed.): Handbook of Research on Teaching. Clarke, D., Keitel, C. and Shimizu, Y. (2006). The learners’ perspective study. D.Clarke, C.Keitel and Y.Shimizu (eds.) Mathematics Classrooms in Twelve Countries: The Insider’s Perspective. Rotterdam: Sense Publishers 150 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning References Clarke, N. (2011). With good reason?: students with low prior attainment reasoning in classroom mathematical activity, DPhil Thesis, University of Oxford Cobb, P., Wood, T. and Yackel, E. (1990). Classrooms as learning environments for teachers and researchers. In R. B. Davis, C. A. Maher & N. Noddings (Eds.), Constructivist views on the teaching and learning of mathematics (pp. 125–146). Reston, VA: National Council of Teachers of Mathematics. Cohen, D. (1990). A revolution in one classroom: The case of Mrs. Oublier. Educational Evaluation and Policy Analysis, 12(3), 311—329. Cooney, T.J., Shealy, B.E. and Arvold, B. (1998). Conceptualizing belief structures of preservice secondary mathematics teachers. Journal for Research in Mathematics Education, 29 (3), pp. 306-333. Correa, C. A., Perry, M., Sims, L. M., Miller, K. F. and Fang, G. (2008). Connected and culturally embedded beliefs: Chinese and US teachers talk about how their students best learn mathematics. Teaching and Teacher Education, 24(1), pp. 140-153. Davis, B. and Simmt, E. (2006). Mathematics-for-teaching: An ongoing investigation of the mathematics that teachers (need to) know. Educational Studies in Mathematics, 61 (3), pp. 293-319. Davis, B. and Simmt, E. (2003). Understanding learning systems: Mathematics education and complexity science. Journal for Research in Mathematics Education, 34 (2), pp. 137-167. Denscombe, M. (2008). Communities of Practice A Research Paradigm for the Mixed Methods Approach. Journal of mixed methods research, 2(3), 270-283. Dewey, J. (1938). Education and experience. Touchstone, New York, NY. Dictionary, O. E. (1989). OED online. Oxford University Press. http://www. oed. com, Accessed Dec, 1, 2012. Doyle, W. (1990). Classroom Management Techniques.” In Student Discipline Strategies: Research and Practice, edited by O. C. Moles, 113–27. New York: New York University Press. Dunkin, M.J. (1976). Problems in the accumulation of process-product evidence in classroom research. Journal of Education for Teaching, 2 (2), pp. 175-187. Eisenhart, M., Borko, H., Underhill, R., Brown, C., Jones, D. and Agard, P. (1993). Conceptual knowledge falls through the cracks: Complexities of learning to teach mathematics for understanding. Journal for Research in Mathematics Education, 24 (1), pp. 8-40. da Ponte, J. P. (1994). Mathematics teachers’ professional knowledge. In J. P. da Ponte and J. F. Matos (eds), Proceedings of the 18th International Conference for the Psychology of Mathematics Education, Vol. 1 (Lisbon: University of Lisbon, Program Committee of the 18th PME Conference), 195–210. ERIC ED 383 537. Main Report 151 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms Dickinson, P. (2000). Teachers ‘questions. Mathematics Teaching, 171, pp. 4-7. Ernest, P. (1991). The philosophy of mathematics education. Falmer. Ernest, P. (1989). The impact of beliefs on the teaching of mathematics. Mathematics teaching: The state of the art, pp. 249-254. Ernest, P. (1989). The knowledge, beliefs and attitudes of the mathematics teacher: A model. Journal of Education for Teaching, 15 (1), pp. 13-33. Fang, Z. (1996). A review of research on teacher beliefs and practices. Educational Research, 38 (1), pp. 47-65. Fauzan, A. (2002). Applying Realistic Mathematics Education (RME) in teaching geometry in Indonesian primary schools. Thesis, University of Twente. Fennema, E., Carpenter, T. P., Franke, M. L., Levi, L., Jacobs, V. R. and Empson, S. B. (1996). A longitudinal study of learning to use children’s thinking in mathematics instruction. Journal for Research in Mathematics Education, 27, 403-434. Fennema, E. and Franke, M. L. (1992). Teachers’ knowledge and its impact. Chapter in Handbook of research on mathematics teaching and learning: A project of the National Council of Teachers of Mathematics. , (pp. 147- 164). New York, NY, England: Macmillan Publishing Co, Inc, xi, 771 pp. Filmer, D. (2006). A Millennium Learning Goal: Measuring Real Progress in Education. CGD Working Paper 97. Fisher, C. W., D. C. Berliner, N. N. Fully, R. S. Marliave, L. S. Cahen, and M. M. Dishaw (1980). “Teaching Behaviors, Academic Learning Time, and Student Achievement: An Overview.” In Time to Learn, edited by C. Denham and A. Lieberman, 7–32. Washington, DC: National Institute of Education. Fleischman, Howard L., et al. (2010). Highlights from PISA 2009: Performance of US 15-Year-Old Students in Reading, Mathematics, and Science Literacy in an International Context. NCES 2011-004. National Center for Education Statistics Goldin, G. (2003). Affect, meta-affect, and mathematical belief structures. Beliefs: a hidden variable in mathematics education?, pp. 59-72. Gondoseputro, T. T. (1999). The cross-cultural perspective of teachers’ beliefs and their influence on teaching practices: A case study of two teachers teaching secondary mathematics in Australia and Indonesia. In Making the difference: Proceedings of the Twenty-second Annual Conference of The Mathematics Education Research Group of Australasia Incorporated (pp. 494-501). Greeno, J.G., Collins, A.M. and Resnick, L.B. (1996). Cognition and learning. Handbook of educational psychology, pp. 15-46. 152 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning References Grossman, P.L. and Stodolsky, S.S. (1995). Content as context: The role of school subjects in secondary school teaching. Educational Researcher, 24 (8), pp. 5. Guba, E.G. and Lincoln, Y.S. (1989). Fourth generation evaluation. Sage Publications, Inc. Guskey, T.R. (1986). Staff development and the process of teacher change. Educational Researcher, 15(5), pp. 5-12. Handal, B. (2003). Teachers’ mathematical beliefs: A review. The Mathematics Educator, 13 (2), pp. 47-57. Handal, B. (2002). Teachers’ beliefs and gender, faculty position, teaching socio-Economic area, teaching experience and academic qualifications. Proceedings of the 2002 International Biannual Conference of the UWS Self Research Centre, pp. 1–6. Hanushek, E. A., & Rivkin, S. G. (2010). Generalizations about using value-added measures of teacher quality. The American Economic Review, 267-271. Hanushek, E. A., & Rivkin, S. G. (2010). Using Value-Added Measures of Teacher Quality. Brief 9. National Center for Analysis of Longitudinal Data in Education Research. Harbison, R.W. and Hanushek, E.A. (1992). Educational performance of the poor: lessons from rural Northeast Brazil. World Bank, Office of the Publisher, 1818 H Street, NW, Washington, DC 20433 Hattie, J. (2003). Teachers make a difference: What is the research evidence? Research conference of the Australian Council for Educational Research, Melbourne, VIC, Australia. Retrieved May, pp. 2007. Hattie, J. (1999). Influences on student learning. Inaugural lecture given on August, 2, pp. 1999 Hiebert, J. (1997). Making Sense: Teaching and Learning Mathematics with Understanding. Heinemann, 361 Hanover Street, Portsmouth, NH 03801-3912. Hiebert, J. (2003). Teaching mathematics in seven countries: Results from the TIMSS 1999 Video Study. DIANE Publishing. Hiebert, J., Gallimore, R., Garnier, H., Givvin, K.B., Hollingsworth, H., Jacobs, J., Chiu, A., Wearne, D., Smith, M. and Kersting, N. (2003). Teaching mathematics in seven countries: Results from the TIMSS 1999 video study (NCES 2003-013). US Department of Education. Washington, DC: National Center for Education Statistics. Hiebert, J. and Stigler, J.W. (2000). A proposal for improving classroom teaching: Lessons from the TIMSS video study. The Elementary School Journal, pp. 3-20. Hill, H.C., Schilling, S.G. and Ball, D.L. (2007). Developing measures of teachers’ mathematics knowledge for teaching. The Elementary School Journal, 105 (1). Main Report 153 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms Hill, H.C., Rowan, B. and Ball, D.L. (2005). Effects of teachers’ mathematical knowledge for teaching on student achievement. American Educational Research Journal, 42 (2), pp. 371. Hill, H.C., Schilling, S.G. and Ball, D.L. (2007). Developing measures of teachers’ mathematics knowledge for teaching. The Elementary School Journal, 105 (1). Howe, K. R. (1988). Against the quantitative-qualitative incompatibility thesis, or, Dogmas die hard. Educational Researcher, 17, 10–16. Hoyles, C. (1992). Mathematics teaching and mathematics teachers: A meta-case study. For the Learning of Mathematics, 12(3), 32-44. Jalil, A. (2011). Comments made in an informal discussion during a case study visit for this research. Kagan, D.M. (1992). Implication of research on teacher belief. Educational psychologist, 27 (1), pp. 65-90. Karnasih, I and Soeparno. (1999). Teaching mathematics has to focus on logic. Kompas, Indonesia, May 17, 2000. Knipping, C. (2003). Learning from comparing: A review and reflection on qualitative oriented comparisons of teaching and learning mathematics in different countries. ZDM, 35(6), 282-293. Kounin, J. S. (1970). Observing and Delineating Technique of Managing Behavior in Classrooms. Journal of Research and Development in Education 4 (1): 62–72. Krauss, S., Baumert, J. and Blum, W. (2008). Secondary mathematics teachers’ pedagogical content knowledge and content knowledge: validation of the COACTIV constructs. ZDM, 40 (5), pp. 873-892. Kuhs, T.M. and Ball, D.L. (1986). Approaches to teaching mathematics: Mapping the domains of knowledge, skills, and dispositions. East Lansing: Michigan State University, Center on Teacher Education. Lerman, S. (2002). Cultural, discursive psychology: A sociocultural approach to studying the teaching and learning of mathematics. Learning discourse, pp. 87-113. Lerman, S. (2001). A review of research perspectives on mathematics teacher education. Making sense of mathematics teacher education, pp. 33-52. Lerman, S. (2000). The social turn in mathematics education research. Multiple perspectives on mathematics teaching and learning, pp. 19-44. Lerman, S. (1990). Alternative perspectives of the nature of mathematics and their influence on the teaching of mathematics. British Educational Research Journal, 16 (1), pp. 53-61. Lester, F. (2003). Implications of research on students’ beliefs for classroom practice. Beliefs: a hidden variable in mathematics education?, pp. 345-353. 154 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning References Leung Frederick K.S. and Ragatz, A. (2010). Inside Indonesia’s Mathematics Classrooms: A TIMSS video study of teaching practices and student achievement, World Bank, Washington D.C. Lindmeier, A. and Ufer, S. (2010). Modelling and Measuring Components of Mathematics Teacher Knowledge and Competencies. Proceedings of the 34th Annual Conference of the International Group for the Psychology of Mathematics Education. (Vol. 3, pp. 209-216). Belo Horizonte, Brazil: PME. Johnson, R. B. and Onwuegbuzie, A. J. (2004). Mixed methods research: A research paradigm whose time has come. Educational researcher, 33(7), 14-26. Ma, L. (1999). Knowing and teaching elementary mathematics: Teachers’ understanding of fundamental mathematics in China and the United States. Lawrence Erlbaum Associates. Mullis, I. V., Martin, M. O., Foy, P. and Arora, A. (2012). TIMSS 2011 international results in mathematics. Martino, A.M. and Maher, C.A. (1999). Teacher questioning to promote justification and generalization in mathematics: What research practice has taught us. The Journal of Mathematical Behaviour, 18 (1), pp. 53-78. Mason, J. (2002). Minding your Qs and Rs: effective questioning and responding in the mathematics classroom. Moyer, P.S. and Milewicz, E. (2002). Learning to question: Categories of questioning used by preservice teachers during diagnostic mathematics interviews. Journal of Mathematics Teacher Education, 5 (4), pp. 293-315. Muijs, D. and Reynolds, D. (2002). Classroom Factors and Student Achievement in Primary Math. American Education Research Association. Mullis, I.V.S. (2007). PIRLS 2006 international report: IEA’s progress in international reading literacy study in primary schools in 40 countries. TIMSS & PIRLS International Study Center, Lynch School of Education, Boston College. Mullis, I., Martin, M. and Foy, P. (2008). TIMSS 2007 international mathematics report. Achievement. Journal of educational psychology, 96 (1), pp. 174. Pajares, M.F. (1992). Teachers’ beliefs and educational research: Cleaning up a messy construct. Review of educational research, 62 (3), pp. 307. Nespor, J. (1987). The role of beliefs in the practice of teaching. Journal of curriculum studies, 19 (4), pp. 317-328. OECD (2014), Measuring Innovation in Education: A New Perspective, Educational Research and Innovation, OECD Publishing. http://dx.doi.org/10.1787/9789264215696-en 
 OECD (2007). PISA 2006 science competencies for tomorrow’s world, Volume 1: Analysis. Main Report 155 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms O’Neill, G.P. (1988). Teaching Effectiveness: A Review of the Research. Canadian Journal of Education/Revue canadienne de l’éducation, 13 (1), pp. 162-185. Op‘t Eynde, P., Corte, E. and Verschaffel, L. (1999). A Socio-constructivist Perspective on the role of Emotions in Mathematical Problem Solving. 8th EARLI-conference, Göteborg. Ornstein, A.C. (1976). Can We Define a Good Teacher? Peabody Journal of Education, 53 (3), pp. 201-207. Pajares, M.F. (1992). Teachers’ beliefs and educational research: Cleaning up a messy construct. Review of educational research, 62 (3), pp. 307. Philipp, R.A., Sowder, J.T., Flores, A. and Schappelle, B.P. (1995). A responsible mathematics teacher and the choices she makes: A case study. Providing a foundation for teaching mathematics in the middle grades, pp. 301-325. Porter, A.C. (2002). Measuring the content of instruction: Uses in research and practice. Educational Researcher, 31 (7), pp. 3-14. Pratt, M. L. (1991). Arts of the contact zone, Profession, 91, 3–20. Ragatz, A. (2013a), Teaching Practices of Indonesia’s Eighth Grade Mathematics Teachers: Formation, Execution and Effect, Dissertation Thesis, University of Oxford, Department of Education Ragatz, A. (2013b). Chapter 5 In: Chang, M.C. of The Role of Politics and Evidence-based Policy Making: The Case of Teacher Reform in Indonesia, World Bank, Washington DC Ragatz, A. and Chen, D. (2009). Transforming Indonesia’s Teaching Force, World Bank, Washington DC Raudenbush, S.W. (2004). What are value-added models estimating and what does this imply for statistical practice? Journal of Educational and Behavioral Statistics, 29 (1), pp. 121-129. Renne, C. (1992). Elementary school teachers’ views of knowledge pertaining to mathematics. Annual Meeting of the American Research Association, San Francisco, CA. Ragatz, A. (2014). Teaching Practices in Indonesia’s 8th Grade Mathematics Classrooms: Formation, Execution and Effect. Doctoral thesis, University of Oxford. Rowan, B., Correnti, R. and Miller, R. (2002). What Large-Scale Survey Research Tells Us About Teacher Effects on Student Achievement: Insights from the Prospects Study of Elementary Schools. The Teachers College Record, 104 (8), pp. 1525-1567. Rowland, T., Thwaites, A. and Huckstep, P. (2005). The knowledge quartet: A framework for reflection, discussion, and professional development. 15th ICMI study: The professional education and development of teachers of mathematics. Retrieved October, 31, pp. 2006. 156 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning References Sanders, W.L. and Rivers, J.C. (1996). Cumulative and residual effects of teachers on future student academic achievement. Knoxville: University of Tennessee Value-Added Research and Assessment Center. Scheerens, J. (1993). Effective schooling: Research, theory and practice. School Effectiveness and School Improvement, 4 (3), pp. 230-235. Schoenfeld, A.H. (2011). Toward professional development for teachers grounded in a theory of decision making, ZDM Mathematics Education (2011) 43:457–469 Schoenfeld, A.H. (2010). How We Think: A Theory of Goal-Oriented Decision Making and Its Educational Applications. Schoenfeld, A.H. (1992). Learning to think mathematically: problem solving, metacognition and sense-making in mathematics. In D. A. Grouws (Ed.), Handbook for Research on Mathematics Teaching and Learning (pp. 334-370). New York: MacMillan. Schoenfeld, A.H. (1989). Explorations of students’ mathematical beliefs and behavior. Journal for Research in Mathematics Education, 20(4), 338-355. Schmidt, W.H., Houang, R.T., Cogan, L., Blömeke, S., Tatto, M.T., Hsieh, F.J., Santillan, M., Bankov, K., Han, S.I. and Cedillo, T. (2008). Opportunity to learn in the preparation of mathematics teachers: its structure and how it varies across six countries. ZDM, 40 (5), pp. 735-747. Schwab, J.J. (1978). Science, curriculum and liberal education: Selected essays, Joseph J. J.Schwab. Chicago: University of Chicago Press. Schmidt, W.H., Houang, R.T., Cogan, L., Blömeke, S., Tatto, M.T., Hsieh, F.J., Santillan, M., Bankov, K., Han, S.I. and Cedillo, T. (2008). Opportunity to learn in the preparation of mathematics teachers: its structure and how it varies across six countries. ZDM, 40 (5), pp. 735-747. Shulman, L.S. (1999). Knowledge and teaching: Foundations of the new reform. Learners and pedagogy, pp. 61-77. Shulman, L.S. (1987). Knowledge and teaching. Harvard Educational Review, 57 (1), pp. 1-22. Shulman, L.S. (1986). Those who understand: Knowledge growth in teaching. Educational researcher, 15 (2), pp. 4. Skemp, R. (1976). Instrumental understanding and relational understanding. Mathematics Teaching, 77, pp. 20–26. Skott, J. (2011). Beliefs vs. Patterns of participation–towards coherence in understanding the role the teacher. Skott, J. (2010). Shifting the direction of belief research: From beliefs to patterns of participation. Skott, J. (2009). Contextualising the notion of ‘belief enactment’. Journal of Mathematics Teacher Education, 12 (1), pp. 27-46. Main Report 157 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms Skott, J. (2004). The forced autonomy of mathematics teachers. Educational studies in Mathematics, 55 (1), pp. 227-257. Skott, J. (2001). The emerging practices of a novice teacher: The roles of his school mathematics images. Journal of Mathematics Teacher Education, 4(3), 3-28. Snow, R.E. and Lohman, D.F. (1984). Toward a theory of cognitive aptitude for learning from instruction. Journal of educational psychology, 76 (3), pp. 347-376. Soedjadi. (2000). Teaching mathematics has to focus on thinking process. Kompas, Indonesia April 17, 2000. Somerset, A. (1997). Strengthening quality in Indonesian junior secondary school: An overview of issues and initiatives. Ministry of Education and Culture, Jakarta. Srivastava, P. and Hopwood, N. (2009). A practical iterative framework for qualitative data analysis. International journal of qualitative methods, 8 (1), pp. 76-84. Steadman, S. (2005). Practically perfect? Excellence in practice-based educational research. A discussion paper for seminar 3 in the ESRC TLRP seminar series: Quality in Educational Research. Sussex: University of Sussex. Stillman, G. (2000). Impact of prior knowledge of task context on approaches to applications tasks. The Journal of Mathematical Behaviour, 19 (3), pp. 333-361. Swan, M. (2000). GCSE mathematics in further education: challenging beliefs and practices. Curriculum Journal, 11 (2), pp. 199-223. Swan, M., (2006). National research and development center for adult literacy and numeracy and National institute of adult continuing education (England/Wales), Collaborative learning in mathematics: a challenge to our beliefs and practices. National Institute of Adult Continuing Education. Tatto, M.T., Schwille, J.S., Senk, S., Ingvarson, L.C., Peck, R. and Rowley, G.L. (2009). Teacher education and development study in mathematics (TEDS-M): Conceptual framework. Dr. Lawrence Ingvarson, pp. 150. Thompson, A.G. (1992). Teachers’ beliefs and conceptions: A synthesis of the research. Handbook of research on mathematics teaching and learning, 127, pp. 146. Thompson, A.G. (1985). Teachers’ conceptions of mathematics and the teaching of problem solving. In E. A. Silver (Ed.), Teaching and learning mathematical problem solving: Multiple research perspectives (pp. 281-294). Hillsdale, NJ: Lawrence Erlbaum. Thompson, A.G. (1984). The relationship of teachers’ conceptions of mathematics and mathematics teaching to instructional practice. Educational studies in mathematics, 15 (2), pp. 105-127. 158 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning References Udin, S. and Johnston, M. (2006). Cross-Cultural Influences on Teacher Education Reform: Reflections on Implementing the Integrated Curriculum in Indonesia, Journal of Education for Teaching: International Research and Pedagogy, Vol.32 (1), p.3-20 Vegas, E., and I. Umansky. (2005). “Improving Teaching and Learning through Effective Incentives.” In Incentives to Improve Teaching: Lessons from Latin America, edited by E. Vegas, 21–62. Washington, DC: World Bank.
 Warfield, J., Wood, T. and Lehman, J. D. (2005). Autonomy, beliefs and the learning of elementary mathematics teachers. Teaching and Teacher Education, 21, 439-456. Watson, A. (2008a). Developing and deepening mathematical knowledge in teaching: Being and knowing. In MKiT 6, Nuffield Seminar Series, 18th March, at University of Loughborough. Watson, A. (2008b). How secondary teachers structure the subject matter of mathematics. Proceedings of the British Society for Research into Learning Mathematics, 28, pp. 126-131. Watson, A. (2007). Developing and deepening mathematical knowledge in teaching: being and knowing. Mathematical Knowledge in Teaching Seminar. Watson, A. (2002). Working with students on questioning to promote mathematical thinking. Mathematics Education Review, 15, pp. 32-42. Watson, A. and Geest, E.D. (2005). Principled teaching for deep progress: Improving mathematical learning beyond methods and materials. Educational Studies in Mathematics, 58 (2), pp. 209-234. Watson, A. and Mason, J. (1998). Questions and Prompts for Mathematical Thinking, ATM, Derby. Weber, R. P. (Ed.) (1990). Basic content analysis (No. 49). Sage. Weidle, R. and Wagner, A. (1994). Die Methode des Lauten Denkens. - In: Huber, G.; Mandl, H. (Eds.), Verbale Daten. 2. Aufl. Weinheim: Psychologie Verlags Union, p. 81-103 Wilson, M., Cooney, T., Leder, G., Pehkonen, E. and Toerner, G. (2002). Beliefs: A hidden variable in mathematics education? Winne, P.H. and Marx, R.W. (1980). Matching students’ cognitive responses to teaching skills. Journal of educational psychology, 72 (2), pp. 257-264. Wragg, E. C. (1993). Multimedia in Education: Bane or Boon?” Journal of Educational Television 19 (2): 73–79. Main Report 159 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms 160 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning APPENDICES A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms While a few key appendices are included in this report, many important pieces of information are captured in a separate technical report. For more extensive details on the instruments, processes and analysis, please see the technical report. CODING LAYERS AND DEFINITIONS Description Definition STRUCTURE OF TIME Mathematical time Time spent on mathematical content presented either through a mathematical problem or outside the context of a problem, e.g., talking or reading about mathematical ideas, solving mathematical problems, practicing mathematical procedures, or memorizing mathematical definitions and rules. Mathematical At least 30 continuous seconds devoted to preparing materials or discussing information related to organization time mathematics, but not qualifying as mathematical work (e.g., distributing materials used to solve problems, discussing the grading scheme to be used on a test, distributing a homework assignment). Non-mathematical time At least 30 continuous seconds devoted to non-mathematical content, e.g., talking about a social function, disciplining a student while other students wait, or listening to school announcements on a public- address system. LESSON SEGMENT PURPOSE Review This category, more technically called “addressing content introduced in previous lessons,” focused on the review or reinforcement of content presented. These segments typically involved the practice or application of a topic learned in a prior lesson, or the review of an idea or procedure learned previously. Examples include: • Warm-up problems and games, often presented at the beginning of a lesson; • Review problems intended to prepare students for the new content; • Teacher lectures to remind students of previously learned content; • Checking the answers for previously completed homework problems; and • Quizzes and grading exercises. Introducing New This category focused on introducing content that students had not worked on in an earlier lesson. Content Examples of segments of this type included: (New content) • Teacher expositions, demonstrations, and illustrations; • Teacher and student explorations through solving problems that were different, at least in part, from problems students had worked previously; • Class discussions of new content; and • Reading textbooks and working through new problems privately. Practice This category focused on practicing or applying content introduced in the current. These segments only occurred in lessons where new content was introduced. They typically took one of two forms: the practice or application of a topic already introduced in the lesson, or the follow-up discussion of an idea or formula after the class engaged in some practice or application. Examples of segments include: orking on problems to practice or apply ideas or procedures introduced in an earlier lesson; • W Class discussions of problem methods and solutions previously presented; and • Teacher lectures summarizing or drawing conclusions about the new content presented earlier. • Evaluation/Assessment Tasks or questions that are given to students evaluate the knowledge of students (through exam or quiz). PUBLIC AND PRIVATE INTERACTION (1) Full class/Public Public presentation by the teacher or one or more students intended for all students interaction • Teacher interaction The teacher lectures to all students. • Teacher and student Presentation made by both teachers and students (in intervals), for all students. interaction • Student interaction Presentation made by students, aimed at the teacher and all students 162 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Appendices Description Definition (2) Individual/Private All students work at their seats, either individually, in pairs, or in small groups, while the teacher often interaction circulates around the room and interacts privately with individual students a. Teacher and group Students work in groups or have discussions with the teacher going from group to group to provide interaction guidance. b. Group interaction Students work in groups or have discussions without teacher’s guidance. c. Teacher and Teachers provide individual counseling to students. individual interaction d. Student interaction Each student works alone with no interaction with the teacher. (3) Student presents A student presents information publicly in written form, sometimes accompanied by verbal interaction information between the student and the teacher or other students about the written work; other students may attend to this information or work on an assignment privately PROBLEM SOLVING STRATEGY Exposition The teacher lectures while students listen and answer closed questions (with no discussion). Discussion The teacher and student or students discuss their own ideas about mathematics. Problem solving The teacher provides a problem / situation as a basis to discuss ideas in mathematics. Practical Equipment or situations in the real world are used to get ideas in mathematics. Investigation Students explore the issues (problems) in various mathematical situations. PROBLEM VS. NOT PROBLEM Problem Events that contained a statement asking for some unknown information that could be determined by applying a mathematical operation. Simple questions asking for immediately accessible information was not counted as problems. Examples of mathematical problems included: • A dding, subtracting, multiplying, and dividing whole numbers, decimals, fractions, percents, and algebraic expressions; • Solving equations; • Measuring lines, areas, volumes, angles; • Plotting or reading graphs; and • Applying formulas to solve real-life problems. Non-problem A non-problem segment is defined to be mathematical work outside the context of a problem. Without presenting a problem statement, teachers (or students) sometimes engaged in: • resenting mathematical definitions or concepts and describing their mathematical origins; P • Giving an historical account of a mathematical idea or object; • Relating mathematics to situations in the real world; • Pointing out relationships among ideas in this lesson and previous lessons; • Providing an overview or a summary of the major points of the lesson; and • Playing mathematical games that did not involve solving mathematical problems (e.g., a word search for mathematical terms). METHOD OF PROBLEM SOLVING Using Procedures Problem statements that suggested the problem was typically solved by applying a procedure or set of procedures. These include arithmetic with whole numbers, fractions, and decimals, manipulating algebraic symbols to simplify expressions and solve equations, finding areas and perimeters of simple plane figures, and so on. Example: “Solve for x in the equation 2x + 5 = 6 - x” were classified as using procedures. Stating Concepts Problem statements that called for a mathematical convention or an example of a mathematical concept. Problem statements. Examples: “Plot the point (3, 2) on a coordinate plane” or “Draw an isosceles right triangle” were classified as stating concepts. Main Report 163 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms Description Definition Make a connection Problem statements that implied the problem would focus on constructing relationships among mathematical ideas, facts, or procedures. Often, the problem statement suggested that students would engage in special forms of mathematical reasoning such as conjecturing, generalizing, and verifying. Example: “Graph the equations y = 2x + 3, 2y = x - 2, and y = -4x, and examine the role played by the numbers in determining the position and slope of the associated lines” were classified as making connections. Show A statement concerning the word "buktikan" or "show". PROBLEM CONTEXT Real world Math problems presented within a real-life context. Examples: “Estimate the surface area of the frame in the picture below,” and “Samantha is collecting data on the time it takes her to walk to school. A table shows her travel times over a two-week period; find the mean.” Math language Problems presented only with mathematical language, Examples: “Graph the equation: y = 3x + 7,” and “Find the volume of a cube whose side measures 3.5 cm.” PROBLEM CONTEXT Open The correct answers from more than one problem. Closed The correct answer from a matter of just one. PROBLEM CONTEXT Routine Problem can be solved directly using the formula, definition or proposition Non-routine Problem that cannot be solved with a usual procedure (directly using the formula, definition or proposition), but instead must be solved using a non-routine strategy (options specified below) 164 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Appendices GRAPHICAL MAPPING OF FINDINGS FOR EACH STAGE OF ANALYSIS The following map highlights key points that were found in each of the steps of the analysis. The blue boxes contain key findings at the component level of Beliefs, Knowledge and Practices. The red boxes contain findings of the dual relationship analysis of beliefs-knowledge, beliefs-practices and knowledge-practices. The green box contains analysis of the beliefs-knowledge-practice interaction. PRACTICES D E HIgher MKiT teachers varied practices Use of Questioning in the classroom General relationship clearly exists more (greater variety) Wider range for some teachers Clearest in “Role” concept Less variation on MKiT assessment than Handling of “pivot moment” is a Teachers tended to see their teaching as teacher degree di erentiating aspect of teachers more connectionist than in actual practice Striking lack of di erence in frequency of Strategies Tensions exist in practices-beliefs practices likely about how done Interaction/Approach/Problem (environment; real vs. perceived) Lower knowledge > fewer practices Practices related to student learning Narrower beliefs > fewer practices G Teachers whose beliefs-knowledge and practices appear to be most e ective Teachers with higher knowledge and a wider range of beliefs are also able to utilize a greater range of practices Concept of relative e ectiveness Context plays a role in choice of practices based in part on beliefs and knowledge at teacher’s disposal MATHEMATICAL MKiT F BELIEFS Wide range in Indonesian teachers Higher MKiT > wider range Flexibility concept emerges Many teachers with minimal level Higher MKiT > more exibility Teachers have range based on context Sub/ped correlation Cases of Beliefs-Knowledge opposite Perception on factors Some sub-ped di erences Beliefs Learning Beliefs as a range rather than a point Teacher’s degree is important distinction Concept of relative e ectiveness Link to learning outcomes Strong link to Learning Outcomes Beliefs expressed in discussing scenarios Scenarios approaches di erent Main Report 165 DISTRIBUTION OF RESPONSES FOR EACH STATEMENT IN THE MATHEMATICAL BELIEFS SURVEY 166 LIKERT SCALE RESPONSES (SCALE OF 1-5) How Students Learn Mathematics Strongly Disagree Disagree Neutral Agree Strongly Agree PRACTICES FOR LEARNING Students learn mathematics best in classes with students of similar abilities 6% 61% 2% 27% 4% Telling the children the answer is an e cient way of facilitating their mathematics learning 18% 56% 0% 23% 2% Listening carefully to the teacher explain a mathematics lesson is the most e ective way to learn mathematics 4% 37% 5% 43% 12% Discovery methods of learning mathematics are relevant for the earliest concepts only, e.g. addition, volume, etc 16% 3% 54% 27% Discovery methods would be useful for older pupils if time and syllabus permitted 3%5% 74% 18% Expecting pupils to be creative in mathematics is unreasonable and doesn’t warrant the use of precious lesson time 4% 57% 8% 31% 0% Mathematics is learnt well by working individually on mathematical problems 5% 48% 4% 37% 7% The more students practice mathematical exercise given in the textbooks, they learn mathematics with understanding 9% 0% 50% 42% PROCESS OF LEARNING Allowing a child to struggle with a mathematical problem, even a little tension, can be necessary for learning to occur 1% 36% 6% 47% 10% It is important that mathematics content be presented to children in the correct sequence 4% 2% 51% 43% A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms Mathematical materials is best presented in an expository style: demonstrating, explaining and describing concepts and skills 8% 2% 55% 35% Learning mathematics in school is essentially an active process 2% 2% 59% 37% Students learn mathematics by nding the correct answer to mathematical problems 12% 1% 69% 19% Students can solve most mathematical problems on the basis of memorized formula 5% 44% 2% 47% 3% Use of a calculator hinders student’s understanding of mathematics 5% 33% 8% 43% 12% Students learn most e ectively when learning tasks are broken down into small sequential steps 5%4% 76% 15% Children always bene t by discussing their solutions to mathematical problems with each other 3%1% 57% 39% Persistent questioning has a signi cant e ect on children’s mathematical learning 9% 6% 62% 23% Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Teaching Mathematics Strongly Disagree Disagree Neutral Agree Strongly Agree Main Report ROLE OF TEACHER The teacher is the main source of mathematical knowledge 5% 53% 3% 30% 9% I guide students in nding their own answers to academic problems 3%2% 76% 19% It is more e ective to provide students with the information they need to know, rather than encouraging them to experiment 2% 51% 6% 37% 4% It is the teacher’s responsibility to provide children with clear and concise solution methods for mathematical problems 1%11% 65% 23% I function in my classroom as a learner and partner in learning with my students 6%1% 72% 21% Teachers should help pupils to develop mental strategies 64% 35% Teachers can create, for all children, a non-threatening environment for learning mathematics 3%2% 50% 45% I wait for students to approach me before o ering extra help 10% 62% 3% 25% 1% APPROACH AND SEQUENCING I teach subjects separately, although I am aware of the overlap of content and skills 5% 44% 15% 36% 0% The most e ective way to teach mathematics is to rst explain the topic, give example(s) and then ask students to solve mathematical tasks 18% 1% 52% 28% ASSESSMENT AND EVALUATION I base student grades primarily on homework, quizzes, and tests 1% 30% 0% 60% 9% I adjust my lesson plan based on results of homework assignments 5% 57% 8% 29% 1% I prefer to assess students informally through observations and conferences 43% 7% 44% 6% It is important that I establish classroom control before I become too friendly with students 11% 5% 64% 20% Paper pencil test is the best way to examine student’s mathematical learning 2% 33% 9% 50% 6% I prefer to cluster student’s desks or use tables so they can work together 1% 14% 6% 59% 20% TOOLS AND RESOURCES Textbooks are the main resource to teach mathematics 3% 28% 4% 53% 12% It is important to cover all the topics in the mathematics curriculum in the textbook sequence 1% 42% 5% 47% 5% To be sure that I teach students all necessary content and skills, I follow a textbook or workbook 3% 25% 3% 65% 4% I generally use the teacher’s guide to lead class discussions of a story or text 15% 15% 69% 1% I nd that textbook and other published materials are the best sources for creating my curriculum 2% 22% 11% 59% 6% I encourage parents to follow up on classroom activities with students at home 7% 8% 54% 31% CURRICULUM, ASSESSMENT AND POLICY The examination syllabus should not be the main factor in determining the school syllabus 1% 30% 8% 57% 4% Teachers should be able to in uence public examination boards in the style of examination, syllabus, content, etc. 19% 23% 50% 8% Public examination syllabus, style, etc. should be decided by experts 3% 21% 20% 39% 17% Commerce and industry’s needs should determine the mathematics curriculum 6% 32% 18% 33% 11% The testing program in my state/district dictates what mathematics content I teach 18% 11% 62% 9% I believe that expanding on students ideas is an e ective way to build my curriculum 1%7% 15% 65% 12% 167 Appendices A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms FREQUENCY OF TEACHING PRACTICES IN THE TEACHER’S LESSONS Frequency of Teacher’s Own Teaching Practices Almost never Occasionally Half the time Most of the time Almost always Students learn through doing exercises 15% 18% 20% 47% Students work on their own, consulting a neighbour from time to time 1% 36% 17% 31% 16% Students use only method I teach them 8% 23% 14% 17% 38% Students start with easy questions and work up to harder questions 3% 17% 5% 24% 50% Students choose which questions they tackle 16% 36% 10% 16% 23% I encourage students to work more slowly 7% 16% 11% 23% 44% Students compare di erent methods for doing questions 5% 38% 14% 16% 28% I teach topic from the beginning, assuming they know nothing 19% 35% 10% 17% 20% I teach the whole class at once 9% 11% 5% 23% 53% I try to cover everything in a topic 21% 23% 13% 24% 21% I draw links between topics and move back and forth between topics 8% 42% 13% 19% 19% I avoid students making mistakes by explaining things carefully rst 6% 18% 11% 32% 33% I tend to follow the textbook or worksheet closely 3% 46% 18% 18% 16% Students learn through discussing their ideas 7% 52% 18% 16% 8% Students work collaboratively in pairs or small groups 4% 38% 25% 16% 18% Students invent their own methods 18% 56% 10% 11% 6% Students work on substantial tasks that can be worked on at di erent levels 3% 29% 20% 24% 25% I tell students which questions to tackle 4% 26% 19% 24% 27% I nd myself encouraging students to work more quickly 9% 30% 22% 16% 24% I only go through one method for doing each question 24% 51% 11% 8% 7% I nd out which parts students already understand and don’t teach those parts 17% 25% 11% 25% 23% I teach each student di erently according to individual needs 23% 37% 8% 13% 20% I only cover important ideas in a topic 15% 32% 20% 20% 14% I tend to teach each topic separately 13% 30% 16% 19% 23% I know exactly what maths the lesson will contain 5% 14% 5% 24% 53% I encourage students to make and discuss mistakes 16% 30% 19% 13% 23% I jump between topics as the need arises 24% 48% 9% 12% 8% 168 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Appendices RANKING RESPONSES Ranking of Preferred Teaching Style from 1 (most) to 7 (least) 1 2 3 4 5 6 7 Giving clear explanations of de nitions and procedurs to follow in di erent exercise and problems of application, in the topics studied 42% 15% 14% 11% 6% 7% 7% Organizing problem-based activities for the pupils to work in small groups where they can present their ideas to the whole class for discussion 15% 24% 23% 17% 11% 3% 7% Testing pupils at the end of each activity or topic, in order to have 15% 18% 10% 11% 17% 20% 9% su cient marks for assessment in each Attainment Target Designing activities that provide space for pupils’ self-paced learning 12% 19% 17% 17% 11% 9% 16% Providing opportunities for pupils to develop their communication skills so that they can express their mathematical ideas with con dence 11% 21% 17% 18% 17% 11% 5% Giving pupils lots of exercises for application as homework 9% 12% 16% 9% 13% 12% 29% Designing classroom work that promotes connections between di erent mathematical topics studied 7% 8% 10% 11% 22% 24% 20% Ranking of Priorities in Teaching from 1 (most) to 5 (least) 1 2 3 4 5 Strategies for investigation and problem solving 23% 42% 21% 7% 8% Interpretation for concepts and representations 23% 20% 29% 15% 14% Fluency in recalling facts and performing routine skills 21% 21% 17% 26% 16% Awareness of the nature and values of the educational system (eg ‘exam technique’) 13% 12% 15% 27% 34% Main Report 169 A Video Study of Teaching Practices in TIMSS Eighth Grade Mathematics Classrooms SUMMARY OF TEACHER RESPONSES TO MATHEMATICAL SCENARIOS Identify issues (error, Conjectures of student Mathematics Develop a strategy misconception) thoughts CS2 Multiple errors Missed some issues More focused on the maths, “I would explain…” “I would less on student thinking repeat…” Confidently discussed maths, but made mistakes CS3 No mathematical errors, Quick to spot most issues, Struggled to make Would try to connect to daily but chose some answers but got confused with issues conjectures about students’ life “Step by step” that would be considered regarding student thinking thoughts, but was fine with Would tap into his personal incorrect according to the mathematical errors. item bank for new problems answer key Constant theme of trying and approaches Ease in discussing maths to help students connect to real world and how students have difficulty with abstract concepts CS4 Minor errors Took time to spot issues, Sometimes discussed Work in groups but generally did and then students’ thoughts Some difficulty in discussing Students work out for elaborated more than other maths themselves teachers. CS5 No errors Quick to spot errors Sometimes discussed Strategies focused on students’ thoughts students discovering for Ease in discussing maths themselves CS6 Minor errors Very quick to spot issues Would often discuss students’ Went to mathematical thoughts explanations Ease in discussing maths Would connect to own experiences CS7 Generally very good, but Quick to spot errors, Would often discuss students’ Game for integer problem could not answer the elaborated on roots thoughts Come up with new inequality scenario Would connect to own perspective Ease in discussing most experiences maths (except inequality) CS9 No mathematical errors, Very quick to spot issues Very clear about student Creative ways of explaining but did not identify correct thought processes. topics from different answer in scenario 5 perspectives Multiple avenues Extreme ease in discussing maths CS10 Multiple errors Took time to spot issues, but Not too focused on student Struggled to come up with generally did thoughts strategies Difficulty in discussing maths 170 Understanding What Teaching Practices are Used, Why They are Used and How They Relate to Student Learning Appendices TECHNICAL REPORT AS A SEPARATE APPENDIX A great deal of data and information was generated for this study and has been documented in detail in a separate technical report. This was done in order to keep the main report at a manageable length and to not bog down the reader in procedural details. The technical report can be considered a large appendix to this main report. The technical report will be of interest for those who are interested in the following: • Full literature review of international research related to the components of the study • In-Depth description of design, methods and analytical approach • Full results of all analysis • Detailed presentation of results including step-by-step • Full description of new conceptual model for the dynamic beliefs-knowledge-practices relationship • Very little policy discussion • Full appendices of instruments, case study profiles, results used for pattern analysis Main Report 171 ÷ + 5 1