Teacher Practices in Mindanao: Results of the Teach Classroom Observation Study Results of the Mindanao Teach Observation Study | 1 Results of the Mindanao Teach Observation Study | 2 Teacher Practices in Mindanao Results of the Teach Classroom Observation Study Authors: Ezequiel Molina, Syeda Farwa Fatima, Iva Trako, and Tracy Wilichowski Date: October 2018 Results of the Mindanao Teach Observation Study | 3 Table of Contents ‘At a Glance’ Summary ..................................................................................................................................................7 1 Introduction..........................................................................................................................................................10 2 Theoretical Framework: Capturing Teacher Practices ........................................................................................11 3 Teach Results: Insights into Teacher Practices ..................................................................................................14 3.1 Area 1: Classroom Culture Results ............................................................................................................16 Classroom Culture: Heterogeneity Analysis ........................................................................................................20 3.2 Area 2: Instruction Results .........................................................................................................................22 Instruction: Heterogeneity Analysis .....................................................................................................................28 3.3 Area 3: Socioemotional Skills .....................................................................................................................30 Socioemotional Skills: Heterogeneity Analysis ....................................................................................................35 3.4 Explaining the Teach Results .....................................................................................................................37 References...................................................................................................................................................................40 Appendix ......................................................................................................................................................................42 A. Teach Element Score Distributions .................................................................................................................42 B. Teach Element Score Distributions by Teacher’s Educational Level ..............................................................43 C. School selection for Teach survey ..................................................................................................................45 List of Figures Figure 0.1: Sample Districts in Mindaneo ......................................................................................................................7 Figure 2.1: Teach Areas……………………………………………………………………………………………………..….11 Figure 3.1: Average Teach Scores by Area .................................................................................................................14 Figure 3.2: Average Teach Scores by Element............................................................................................................15 Figure 3.3: Supportive Learning Environment ..............................................................................................................16 Figure 3.4: Positive Behavioral Expectations ...............................................................................................................18 Figure 3.5: Opportunities to Learn ...............................................................................................................................19 Figure 3.6: Classroom Culture – Heterogeneity Analysis.............................................................................................21 Figure 3.7: Lesson Facilitation .....................................................................................................................................22 Figure 3.8: Checks for Understanding .........................................................................................................................24 Figure 3.9: Feedback ...................................................................................................................................................25 Figure 3.10: Critical Thinking .......................................................................................................................................26 Figure 3.11: Instruction – Heterogeneity Analysis ........................................................................................................29 Figure 3.12: Autonomy .................................................................................................................................................30 Figure 3.13: Perseverance ...........................................................................................................................................31 Figure 3.14: Social and Collaborative Skills .................................................................................................................33 Figure 3.15: Socioemotional Skills – Heterogeneity Analysis ......................................................................................36 Results of the Mindanao Teach Observation Study | 4 Figure 3.16: Teachers Self-Evaluation vs Teach scores ..............................................................................................38 Figure A. 1: Teach Elements – Score Distributions......................................................................................................42 Figure B. 1: Teach Elements – Score Distributions by Teacher’s Educational Level ...................................................43 Figure C. 1: Student learning in Mindanao (% of students)..........................................................................................46 List of Tables Table 0.1: Overview of Study .........................................................................................................................................7 Table 0.2: ‘At a Glance’ Table ........................................................................................................................................8 Table 3.1: Teacher Profile ............................................................................................................................................39 Table C.1: School access and completion (by region, 2016) .......................................................................................46 Table C.2: Public elementary schools classified into two selected parameters (Region X) .........................................47 Table C.3: Summary statistics of the Teach sample schools .......................................................................................47 List of Acronyms MoE Ministry of Education DepEd Philippines Department of Education CLASS Classroom Assessment Scoring System FFT Framework for Teaching PLATO Protocol for Language arts Teaching Observations OPERA Observation of Teaching Practices in Relation to Pupil Learning SCOPE Standards-based Classroom Observation Protocol for Egypt SDI Service Delivery Indicators TIPPS Teacher Instructional Practices and Processes System Results of the Mindanao Teach Observation Study | 5 Acknowledgements This report was prepared by a team led by Ezequiel Molina. The core team included Syeda Farwa Fatima, Iva Trako, and Tracy Wilichowski. Additionally, Adelle Pushparatnam provided advice during the preparation of this report and Marie Evane Tamagnan provided support in compiling the various parts of the report. The team is thankful to the Philippines World Bank Education Task Team Leader(s), Franco Russo and Binh Thanh Vu, for initiating the study and leveraging support from the government, and to Andrew Ragatz, for continuing to utilize the Teach findings in government dialogue. The team also thanks Regina Calzado, Takiko Igarashi, Javier Luque, Jacqueline Pacampara, and Sharon Piza from the Country Office Team as well as Mara Warwick (Country Director) and Gabriel Demombynes (Program Leader) from the Country Management Unit for their support and guidance on the ground. Data collection was undertaken by RIMCU Consulting Firm, consisting of Dong and Lita Sealza (Managing Partners) and Isaias Sealza (Project Manager and Team Supervisor). Data was collected by 24 enumerators who were chosen by RIMCU and passed the Teach certification exam, a prerequisite to partake in the study. The team is also grateful to Dr. Lorna Dig-Dino, Undersecretary of Curriculum and Instruction and the staff of the Department of Education, for her wholehearted support and guidance. Moreover, the team would like to thank the Department of Education Region 10, as well as divisions and schools who provided useful advice and helped coordinate the fieldwork. Moreover, the team is grateful to the security team led by Agapito Carmelo Jr Senson Nagrampa for guaranteeing our safety during the training in Cagayan de Oro. Finally, the team would like to extend its sincerest gratitude to the Australia Department of Foreign Affairs and Trade, who made this project possible, and to all the teachers’ who allowed us to us to record their classrooms for the Teach training. Results of the Mindanao Teach Observation Study | 6 ‘At a Glance’ Summary Table 0.1: Overview of Study Location Mindanao Teach measures the quality of classroom practices over the course of Number of Schools 45 a teacher’s lesson. The tool is organized into three broad areas: % of Urban Schools 27% Classroom Culture, Instruction, and Socioemotional Skills. These % of Rural Schools 73% areas have ten corresponding elements that point to twenty-seven Number of Teachers 140 behaviors. The behaviors are characterized as low, medium, or high, % of Male Teachers 5% based on the evidence observed in this classroom. These preliminary % of Female Teachers 95% scores are translated into a five-point scale, which quantifies the # of Teach Observations 316 teacher’s practices as captured in two, 15-minute observations. Medium Class Size 22 Number of Students 3,258 This study documents the results of a pilot study designed to measure % of Male Students 52% the quality of teacher practices in a selection of 45 public primary % of Female Students 48% schools (73% rural and 27% urban) in Mindanao, a province of the 31% - 1st grade Philippines. 24 observers, who were trained and certified to conduct 29% - 2nd grade % of Grade Level Observed classroom observations, captured the practices of 140 teachers (95% 33% - 3rd grade female) from 41 districts in Mindanao (Table 0.1; Figure 0.1). 6% - Multi-grade Classroom observations were conducted in grades 1, 2, and 3 and 35% - Math 30% - English across several subjects including Mathematics, English, and Science. 16% - Reading The sample was chosen based on student learning outcomes and Subject Distribution 9% - Filipino school size. Due to the small sample size, the findings from this report 6% - Other may not apply to other regions of the Philippines (see Appendix 3 for 4% - EsP a detailed description of the survey sample). Results Table 0.2 Figure 0.1: Sample Districts in Mindanao Source: Teach Philippines 2018 Results of the Mindanao Teach Observation Study | 7 ‘At a Glance’ Table 0.2 Element Description & Distribution Behaviors Low Description Low Medium Description Med High Description High N/A 1.1: Treats all students 1 0% Does not treat all respectfully 1% Treats all somewhat respectfully 26% Consistently treats all respectively 72% respectfully SUPPORTIVE LEARNING ENVIRONMENT: The teacher 1.2: Uses positive language 2 2% Does not use positive language 15% Uses some positive language 55% Consistently uses positive language 30% creates a classroom environment where students can feel with students emotionally safe & supported. Moreover, all students feel 1.3: Responds to students' 3 21% Does not respond to needs 7% Responds but does not address the problem 10% Responds & addresses the problem 14% 69% welcome, as the teacher treats all students respectfully. needs 4 53% 1.4: Treats students of all Exhibits explicit gender bias 1% Exhibits some implicit gender bias 9% Treats all genders with equal regard 90% 5 24% genders with equal regard Classroom Culture 1 2% 2.1: Sets clear behavioral POSITIVE BEHAVIORAL EXPECTATIONS: The teacher Does not set clear expectations 12% Sets unclear or superficial expectations 53% Sets clear expectations 35% 2 29% expectations promotes positive behavior by acknowledging students’ 2.2: Acknowledges positive behavior that meets or exceeds expectations. Moreover, the 3 33% Does not acknowledge positive student behavior 36% Acknowledges some behavior, but not explicitly 54% Acknowledges positive behavior 10% behavior teacher sets clear behavioral expectations for different parts 4 34% 2.3: Effectively redirects Effectively redirects or students are well- of the lesson. Ineffectively redirects 13% Effectively redirects or somewhat effective 55% 32% 5 5% misbehaviors behaved OPPORTUNITIES TO LEARN: The teacher maximizes opportunities to learn by ensuring most students are provided with a learning activity most of the time. 40% or more time spent on non-learning activities 1 3% 30-40% of time spent on non-learning activities 2 2% 20-30% of time spent on non-learning activities 3 11% 10-20% of time spent on non-learning activities 4 22% Less than 10% of time spent on non-learning activities 5 62% Results of the Mindanao Teach Observation Study | 8 Element Description & Distribution Behaviors Low Description Low Medium Description Med High Description High N/A 4.1 Articulates lesson States specific objective that's aligned to 1 1% Does not state objective or cannot be inferred 11% States broad objective or can be inferred 60% 30% Objectives activities LESSON FACILITATION: The teacher facilitates the lesson to promote comprehension by explicitly articulating the 4.2 Provides clear 2 29% Confusing or no explanation 6% Somewhat clear explanation 44% Clear & straightforward explanation 50% objectives, providing clear explanations of concepts, & explanations connecting the lesson with other content knowledge or 4.3 Meaningfully connects 3 36% Does not connect 57% Superficially or unclearly connects 26% Meaningfully connects 17% students’ experiences. the lesson 4 32% 4.4 Models by enacting or Completely models by enacting & thinking Does not model 22% Partially models 54% 24% 5 24% thinking aloud aloud 1 4% 5.1 Asks questions & prompts to determine Either does not ask or asks one-worded questions 13% Sometimes asks ineffective questions 63% Consistently asks effective questions 25% CHECKS FOR UNDERSTANDING: The teacher checks for 2 30% understanding understating to ensure most students comprehend the lesson Instruction content. Moreover, the teacher adjusts the pace of the lesson 3 40% 5.2 Monitors during Does not monitor students 12% Monitors some students 26% Systematically monitors most students 19% 42% to provide students with additional learning opportunities. independent / group work 4 24% 5.3 Adjusts teaching for Does not adjust 33% Briefly & superficially adjusts for some students 57% Substantially adjusts for most students 10% 5 3% students 1 12% FEEDBACK: The teacher provides specific comments or 2 0% Consistently provides specific & substantive comments to help identify misunderstandings, understand 3 72% 6.1 Provides feedback Does not provide comments or comments are simple 12% Provides general or superficial comments 73% 15% comments successes, & guide thought processes to promote learning. 4 1% 5 15% 1 8% 7.1 Asks thinking or open- Asks 3+ thinking questions that build upon Does not ask thinking questions 30% Asks 2+ thinking questions 53% 17% 2 46% ended questions student responses CRITICAL THINKING: The teacher encourages students to think critically by helping them identify & synthesize relevant 3 28% 7.2 Provides thinking tasks Does not provide thinking tasks 20% Provides superficial thinking tasks 61% Provides substantial thinking tasks 19% information, analyze problems, & evaluate solutions. 4 18% 7.3 Students ask open- ended questions & perform Students neither ask nor perform 52% Students do not ask, but perform 45% Students ask &/or perform 3% 5 1% thinking tasks Element Description & Distribution Behaviors Low Description Low Medium Description Med High Description High N/A 8.1 Provides students with AUTONOMY: The teacher provides students with 1 4% Does not provide choices 67% Provides some superficial choices 25% Provides substantive, learning choices 8% choices opportunities to make choices & take on meaningful roles in 2 43% 8.2 Provides opportunities Provides opportunities to take meaningful the classroom. Students make use of these opportunities by Does not provide opportunities 32% Provides opportunities to take limited roles 50% 19% 3 33% to take on roles roles volunteering to take on roles & expressing their ideas & 4 18% 8.3 Students volunteer to opinions throughout the lesson. Students don't volunteer 10% Few students volunteer 56% Most students volunteer 35% 5 1% participate PERSEVERANCE: The teacher promotes students’ efforts 1 5% 9.1 Acknowledges students' Socioemotional Skills Does not acknowledge effort 41% Sometimes acknowledges efforts 53% Frequently acknowledges & identifies efforts 7% toward the goal of mastering new skills or concepts, instead 2 78% efforts of focusing solely on results, intelligence, or natural abilities. 9.2 Positive attitude toward In addition, the teacher has a positive attitude toward 3 13% Has a negative attitude 9% Has a neutral attitude 80% Has a positive attitude 11% student challenges challenges, framing failure & frustrations as useful parts of the learning process. The teacher also encourages students 4 4% Encourages short or long-term goal-setting, or 9.3 Encourages goal-setting Does not encourage goal-setting 92% 7% Encourages short & long-term goal-setting 1% to set short- &/or long-term goals. 5 0% discusses their importance SOCIAL & COLLABORATIVE SKILLS: The students 1 25% 10.1 Students collaborate Students don’t collaborate or display negative Students collaborate some & rarely display negative Students consistently collaborate & display cooperate with one another in the classroom, creating an 46% 35% 19% with one another behavior behavior no negative behavior environment free from physical or emotional hostility. The 2 35% teacher complements this by promoting students’ 3 24% interpersonal skills, so that they can take the perspective of 4 12% 10.2 Promotes students Does not promote interpersonal skills 44% Briefly or superficially promotes interpersonal skills 49% Explicitly promotes interpersonal skills 7% others, empathize, regulate their emotions, & socially problem- interpersonal skills 5 4% solve. Results of the Mindanao Teach Observation Study | 9 1 Introduction School enrollment has increased substantially over the last 25 years in low and middle-income countries. Schooling, basic reading, writing, and arithmetic skills (World Development Report, 2018) — a state of affairs UNESCO (2013) dubbed the “global learning crisis.” In both in the Philippines and around the world, the learning crisis is, at its core, a teaching crisis (Bold et al., 2017). Previous reports have indicated Filipino teachers’ lack basic content knowledge in most subjects (Al-Samarrai et al., 2015). However, no report has measured what actually happens in the classroom. This report details the nature of teaching practices across Mindanao as captured by the high-inference classroom observation tool, Teach. WHY MEASURE TEACHER PRACTICES? Identifying effective teaching is not easy. Research indicates teacher characteristics such as formal education, years of experience (beyond the first two), cognitive skills, and entry exam performance scores only explain a small fraction of the variation in teacher effectiveness (Staiger & Rockoff, 2010; Araujo et al., 2016; Bau & Das, 2017; Cruz-Aguayo et al., 2017). Variation in student learning is better explained by teachers’ practices in the classroom . For example, a seminal study in Ecuador found a one SD increase in teacher quality, as measured by teachers’ scores on the CLASS observation tool, is associated with a 0.18 SD increase in learning outcomes (Araujo et al., 2016). Moreover, teachers’ scores on classroom observation tools in the United States are positively associated with student achievement gains (Kane & Staiger, 2012; Kane et al., 2011; Hamre et al., 2014; Holtzapple, 2003; Milanowski, 2004). However, it’s not simply teacher practices that exhibit positive effects as the improvement of their practices also has positive effects on student outcomes. For instance, students of Chilean teachers who were given access to classroom observation feedback and coaching performed .05-.09 SD higher on state tests and .04-.06 SD higher on national tests than those whose teachers did not receive such feedback (Bruns et al., 2016). Moreover, a study of over 60 coaching programs found those designed to advance teacher practices (0.58 SD) also resulted in increased student learning (0.15 SD) (Kraft et al., 2018). THIS REPORT This report is organized as follows. Section 2 describes Teach’s theoretical framework, content, and development process. Section 3 discusses the main results and highlights differences across Teach elements and behaviors by i) urban/rural school, ii) grade level, and iii) teacher’s education level. Finally, using data from teacher and principal surveys, the report sheds additional insights on Teach results. Results of the Mindanao Teach Observation Study | 10 2 Theoretical Framework: Capturing Teacher Practices WHAT DOES TEACH MEASURE? Teach measures the quality of classroom practices over the course of a teacher’s lesson. The tool is organized into three broad areas: Classroom Culture, Instruction, and Socioemotional Skills.1 Figure 2.1: Teach Areas These 3 areas have 10 corresponding elements that point to 27 behaviors (Figure 2.1). The behaviors are characterized as low, medium, or high, based on the quality of teacher practices observed. These behavior scores are then translated into a 5-point scale that quantifies teacher practices into a numerical score. These observations are captured in a series of two 15-minute lesson observations. 1. Classroom Culture: The teacher creates a culture that is conducive to learning. The focus here is not on the teacher correcting students’ negative behaviors but rather the extent to which the teacher creates: (i) a supporting learning environment by treating all students respectfully, consistently using positive language, responding to students’ needs, and not exhibiting gender bias in the classroom; and (ii) positive behavioral expectations by setting clear behavioral expectations, acknowledging positive student behavior, and effectively redirecting misbehavior; (iii) offers opportunities to learn by ensuring the majority of class time is spent on learning activities. 2. Instruction: The teacher instructs in a manner that intellectually challenges and engages students. The focus here not on content-specific methods of instruction, but rather the extent to which the teacher (i) facilitates the lesson so that the objectives are articulated with the classroom activities, the content is presented in a clear 1 It should be noted that it is impossible to draw a clear line between teacher practices linked to academic versus socioemotional learning. Many teacher practices included in common professional teaching frameworks do impact student’s socioemotional develo pment, though are usually thought of in terms of academic rather than socioemotional learning. Explicitly linking teacher practices with socioemotional outcomes in measures used for assessment will serve to increase the salience of student’s socioemotional skills to teachers, as well as to other stakeholders and policymakers, thus ensuring a focus on both academic and socioemotional learning in the classroom. Results of the Mindanao Teach Observation Study | 11 manner, further enhancing student understanding by modeling and connecting the lesson to past content and experiences; (ii) does not simply move from one topic to the next but stops to check for understanding by posing questions to students, monitors what they are doing during independent/group work, and differentiates instruction to meet the needs of each individual student; (iii) provides feedback to deepen understanding by giving specific comments or guiding hints/questions to help students clarify misunderstandings and understand successes; (iv) encourages students to think critically by helping them identify and synthesize relevant information, analyze problems, and evaluate potential solutions. 3. Socioemotional Skills: The teacher fosters socioemotional skills that encourage students to succeed both inside and outside the classroom. To develop student’s social and emotional skills, the teacher: (i) instills autonomy by providing students with opportunities to make choices and take on meaningful roles in the classroom. Students’ exhibit their autonomy by volunteering to participate in classroom activities; (ii) promotes perseverance by acknowledging student’s efforts, rather than focusing solely on their intelligence or natural abilities, by having a positive attitude toward students’ challenges by framing failure and frustrations as part of the learning process, and by encouraging students to set short- and long-term goals; and (iii) fosters social and collaborative skills by promoting interpersonal skills, such as perspective taking, empathizing, emotion regulation, and social problem solving. Students exhibit social and collaborative skills by collaborating with one another through peer interaction. HOW WAS TEACH DEVELOPED? The Teach development team rigorously researched, revised, and piloted different iterations of the tool over a 2-year timeframe: First, the development team — which comprised 1 education measurement expert, 1 instructional expert, 1 psychologist and 1 teacher — assessed 5 classroom observation tools widely used in the United States to create an inventory of teacher practices that are commonly evaluated.2 The team then built upon this list to include behaviors from international classroom observation tools used in developing countries.3 Based on this preliminary analysis, the team created an inventory of 3 areas and 43 elements.4 Secondly, the development team hosted a working group of 22 education experts and practitioners to help further reduce and prioritize elements for the Teach framework. Participants were asked to indicate whether any elements were missing from the inventory, to rank the elements and areas by relevance, and to identify elements they characterized as unobservable. This process reduced the framework to 25 elements. Then, the development team reviewed the theoretical and empirical evidence from developing countries to further eliminate elements from the framework. This process resulted in a downsized framework of 14 elements. These 14 elements comprised the first working version of the tool, which aimed to capture both quality and frequency of teaching practices as measured by each element.5 This preliminary tool was piloted in person in Pakistan and Uruguay and using classroom video footage in Afghanistan, China, Pakistan, the Philippines, Tanzania, Uruguay, and Vietnam. From these pilots, it became apparent that observers struggled to code reliably when they had to simultaneously capture the frequency and quality of teaching practices for each element. In response, the development team revised the structure of the tool to address this challenge as well as other errors and logical inconsistencies. This 2 The Teach framework built upon the inventory created by Gill and others (2016), who conducted a content analysis of the differences in dimensions of instructional practice of 5 commonly used classroom observation tools comparing the behaviors they measure with the extent to which they predict student learning. The tools included CLASS, FFT, PLATO, Mathematical Quality of Instruction, and UTeach Observational Protocol. The content, predictive power, and potential bias of these instruments were also analyzed as part of this preliminary framework. 3 These included OPERA, SCOPE, SDI, Stallings, and TIPPS. 4 Elements refer to groups of multiple, similar behaviors that aim to capture teaching practices related to positive learning outcomes 5 For example, the tool aimed to capture not just the quality with which a teacher checked for understanding (adjusting the lesson, prompting students to determine their level of understanding, etc.), but the frequency with which the teacher checked for understanding in each lesson. Results of the Mindanao Teach Observation Study | 12 process resulted in a tool that comprised 10 elements. Finally, the development team convened a technical advisory panel, including Lindsay Brown, Pam Grossman, Heather Hill, Andrew Ragatz, Sara Rimm-Kaufman, Erica Woolway, and Nick Yoder, to provide written feedback on the tool. These comments were compiled and addressed as part of a 1-day technical workshop. During the workshop, the experts advised the team on which issues to prioritize and how to incorporate the comments to further improve the tool. The updated version of the tool was pre-piloted using videos and used that version to apply in Philippines where observers were given a certification exam that ensured they could reliably code using Teach.6 6 Aside from Philippines, this version of the tool was applied in 3 settings, where observers were given a certification exam that ensured they could reliably code using Teach. In Mozambique, 76% of the observers passed the reliability exam; in Pakistan, like in Philippines, 96% passed; and in Uruguay, 100% passed. The observers also provided feedback on the tool and training that was considered during the revision process. After this, the development team worked closely with Andrew Ho (Harvard University) to analyze the psychometric properties of the data from the tool’s field applications. Based on this analysis and feedback from the trainers and observers, the development team revised each element’s structure and complementary examples to improve the consistency and clarity of the tool’s elements (See footnote 3). Results of the Mindanao Teach Observation Study | 13 3 Teach Results: Insights into Teacher Practices TEACH RESULTS Data collected from Teach indicates 7 out of every 10 teachers struggle (score less than 2.5) in at least one area. Results indicate Mindanao teachers have strong ability in Classroom Culture; however, they exhibit weaker ability in Instruction and Socioemotional Skills (Figure 3.1). Figure 3.1: Distribution of Average Teach Scores by Area Classroom Culture 28 68 3 Instruction 27 59 13 1 Socioemotional Skills 69 27 5 0% 20% 40% 60% 80% 100% Proportion of Teachers Low Medium Low Medium High High Note: Low is defined as less than 2.5, medium low 2.5-3.5, medium high 3.5-4.5, and high 4.5-5. Teachers that have strong ability in Classroom Culture and weak ability in Instruction and Socioemotional Skills is consistent with Teach findings in other countries (Molina et al., 2018). Teachers in Mindanao are relatively skilled at creating a supportive learning environment and maximizing opportunities to learn, though they are less effective at setting positive behavioral expectations. Moreover, these teachers score around the medium range in facilitating the lesson, checking for understanding, and providing feedback; however, they are less likely to encourage students to think critically. Lastly, Mindanao teachers are poor at promoting student autonomy, fostering perseverance, and at promoting social and collaborative skills. Overall, teachers are much more likely to create opportunities to learn and less likely to foster perseverance and social and collaborative skills (Figure 3.2). The next section characterizes teacher practices in Mindanao for each area. Results of the Mindanao Teach Observation Study | 14 Figure 3.2: Distribution of Average Teach Scores by Element Classroom Culture 1.Supportive Learning Environment 4.0 2.Positive Behavioral Expectations 3.1 3.Opportunities to Learn 4.4 4.Lesson Facilitation 3.1 Instruction 5.Checks for Understanding 2.9 6.Feedback 3.0 7.Critical Thinking 2.6 8.Autonomy 2.7 Socioemotional Skills 9.Perseverance 2.2 10.Social and Collaborative Skills 2.3 1 2 3 4 5 Source: Teach Philippines 2018 Results of the Mindanao Teach Observation Study | 15 3.1 Area 1: Classroom Culture Results Classroom Culture measures the extent to which the teacher creates a culture that is conductive to learning. The focus here is not on the teacher correcting students’ negative behaviors, but rather the extent to which the teacher (i) creates a supportive learning environment, in which students are treated respectfully. The teacher does this by consistently using positive language, attending to students’ needs, and not implicitly or explicitly favoring one gender over the other; (ii) focuses on positive behavior, rather than wasting time on negative behavior, by setting clear expectations, reinforcing student’s behavior when expectations are met, and redirecting misbehavior without interrupting the lesson; (iii) offers opportunities to learn by ensuring the majority of class time is spent on learning activities. Overall, teachers performed well on Classroom Culture and they perform consistently well on this element. On average, they score 4 points out of the 5 points possible in this element. They were most effective at creating a supportive learning environment, somewhat effective at setting positive behavioral expectations, and highly effective offering ample opportunities to learn (See Table 0.2). Supportive Learning Environment. Most teachers create a supportive learning environment. On average, they score 4 points out of the 5 points possible in this element. Figure 3.3 shows the distribution of scores for supportive learning environment and its respective behaviors. Mindanao teachers consistently treat all students respectfully (72%) and treat all genders with equal regard (90%), though they tend to use some positive language with students (30%). Student needs rarely come up in the classroom (69%), though when they do, teachers often do not respond to them in a way that addresses the problem (14%). Figure 3.3: Supportive Learning Environment 1.1 The teacher treats all students 1.2 The teacher uses positive language with respectfully students 72% 55% 30% 27% 15% 1% L: Does not treat all respectfully L: Does not use positive language M: Treats all somewhat respectfully M: Uses some positive language H: Consistently treats all respectfully H: Consistently uses positive language Results of the Mindanao Teach Observation Study | 16 1.3 Responds to students needs 1.4 Treats students of all genders with equal regard 69% 90% 10% 14% 7% 9% 1% L: Does not respond to needs L: Exhibits explicit gender bias M: Responds but does not address the problem M: Exhibits some implicit gender bias H: Responds and addresses the problem H: Treats all genders with equal regard N/A 100 % Proportion of Teachers 75 53% 50 21% 24% 25 0% 2% 0 1 2 3 4 5 Distribution of Scores Source: Teach Philippines 2018 This is exhibited when teachers (72%) use words like ‘please’ and ‘thank you’ when responding to and addressing students: "Please, could you fix your things?", “Okay, children, now we will read a story together” or “Can you get that pencil, please? Additionally, teachers use positive language to encourage students, such as “Now, I know that you are very good at dividing.” or “Let’s clap our hands for Jaye -a classmate-!”. Moreover, almost all teachers (90%) give students of all genders equal opportunity to participate in and lead class activities. For instance, both boys and girls are asked to read difficult words like "Barangay Makugihon" and "Manong Ambo” aloud; they also are equally represented as group leaders during a group activity. Results of the Mindanao Teach Observation Study | 17 Positive Behavioral Expectations. On average, teachers score 3 points out of the 5-points possible in this element. Figure 3.4 shows the distribution of scores for the positive behavioral expectations element and its respective behaviors. Mindanao teachers generally set unclear or superficial expectations (53%), acknowledges some positive student behavior, but do not do so explicitly (54%), and either effectively redirects misbehavior but focuses on misbehaviors, or somewhat effectively directs misbehavior, and focuses on the expected behavior (55%). Figure 3.4: Positive Behavioral Expectations 2.1 Behavioral expectations 2.2 Acknowledges positive behavior 53% 54% 35% 36% 12% 10% L: Does not set clear expectations L: Does not acknowledge positive student behavior M: Sets unclear or superficial expectations M: Acknowledges some behavior, but not explicitly H: Acknowledges positive behavior 2.3 The teacher redirects misbehaviors 55% 32% 13% L: Ineffectively redirects M: Effectively redirects or somewhat effective H: Effectively redirects or students are well behaved 100 % Proportion of Teachers 75 50 33% 34% 29% 25 2% 3% 0 1 2 3 4 5 Distribution of Scores Source: Teach Philippines 2018 For example, while one-third of teachers (35%) set clear behavioral expectations, over half of teachers (53%) set unclear or superficial behavioral expectations, and the rest (12%) fail to set any behavioral expectations for classroom activities. This is exhibited when teachers introduce a group activity, teachers tell students to “not leave your group,” “go to your group,” and “do the work properly.” While some of these directions are clear (“Don't leave your group”), others, such as: “Do the work properly”, “do not clearly articulate what “properly” entails. Teachers (36%) also struggle to Results of the Mindanao Teach Observation Study | 18 acknowledge positive student behavior that meets or exceeds expectations. While teachers praise and acknowledge students who provide correct answers, they rarely acknowledge students who raise their hands, sit quietly, patiently wait to participate, and quietly transition through class activities. For instance, while the teacher may say, “you are working great together in your small groups,” she does not explicitly articulate what is “great” about the students’ behavior. Finally, more than one third of teachers are ineffective at redirecting misbehavior and focus on what students are doing wrong rather than what’s expected of them (13%). This is indicated by a classroom in which students are generally well-behaved; however, there are multiple instances where the teacher tries to correct misbehavior but is forced to repeat herself multiple times to students who are only half-listening. Opportunities to Learn. Most teachers (84%) provide most students with a learning activity most of the time; of these, 62% spend less than 10% of class on non-learning activities, such as classroom disruptions or transitions. On average, teachers score 4.4 points out of the 5 points possible in this element, which makes opportunities to learn the highest scoring element. Figure 3.5 shows the distribution of scores for the opportunities to learn element. Figure 3.5: Opportunities to Learn 100 75 % Proportion of Teachers 62% 50 25 22% 11% 3% 2% 0 1 2 3 4 5 Distribution of Scores Score 1: More than 40% time spent on non-learning activities Score 2: 30-40% of time spent on non-learning activities Score 3: 20-30% of time spent on non-learning activities Score 4: 10-20% of time spent on non-learning activities Score 5: Less than 10% of time spent on non-learning activities Note: This element does not have any behaviors. Source: Teach Philippines 2018 These behaviors are exhibited during classroom observations. In addition to providing students with a learning activity most of the time with minimal wait time, materials tend to be prepared and teachers do not spend much time on administrative tasks. In the event teachers spend time on administrative tasks, this generally takes less than 1 minute when students transition from one activity to another. Results of the Mindanao Teach Observation Study | 19 Classroom Culture: Heterogeneity Analysis Figure 3.6 shows the average scores across the three elements that constitute Classroom Culture. This information is broken down into three subcategories: urban/rural school, grade level, and teachers’ educational level. Rural teachers do not differ significantly from urban teachers at creating a supportive learning environment, but they are better at providing positive behavioral expectations and creating learning opportunities than their urban counterparts. Moreover, most rural (80%) and urban (73%) create a supportive learning environment. However, rural teachers tend to focus significantly more on positive behavioral expectations and offer significantly more opportunities to learn compared to urban teachers. In particular, 41% of rural teachers promote positive behavioral expectations compared to just 24% of urban teachers. In addition, almost three-quarters of rural teachers (67%) maximize opportunities to learn compared to just half (47%) of urban teachers. There is no statistically significant difference among teachers responsible for different grade levels in the Classroom Culture area. Teachers in higher grade levels (e.g. Grade 3 and Multi-grade) tend to be slightly better at creating a Classroom Culture that is conductive to learning compared to those in lower grade levels, but this difference is not statistically significant. Teachers with different educational levels do not differ significantly in creating a supportive learning environment. However, teachers with a higher degree (i.e. a master’s degree) tend to be better at promoting positive behavioral expectations and maximizing learning opportunities compared to those with a bachelor’s degree or lower. Notably, 50% of teachers with a master’s degree or more are more likely to promote positive behavioral expectations, compared to only 25% of those with bachelor’s degree or lower. Similarly, almost all teachers with a master’s degree maximize opportunities to learn compared to just three-quarters of those with bachelor’s degree or lower (See Figure B. 1). Results of the Mindanao Teach Observation Study | 20 Figure 3.6: Classroom Culture – Heterogeneity Analysis Classroom Culture by Urban/Rural School 5 4.4 4.0 4.0 4.2 4 3.2 2.9 3 2 1 Supportive Learning Positive behavioral Opportunities to learn Environment expectations Urban Rural Classroom Culture by Grade Level 5 4.6 4.5 4.0 3.9 4.1 4.0 4.2 4.4 4 3.3 3.0 3.0 2.9 3 2 1 Supportive Learning Positive behavioral Opportunities to learn Environment expectations Grade 1 Grade 2 Grade 3 Multigrade Classroom Culture by Teacher's Education Level 5 4.5 4.3 4.0 4.0 4.1 4.0 4 3.5 3.0 2.9 3 2 1 Supportive Learning Positive Behavioral Opportunities to Learn Environment Expectations < Bachelor Bachelor Master Source: Teach Philippines 2018 Results of the Mindanao Teach Observation Study | 21 3.2 Area 2: Instruction Results Instruction measures whether the teacher instructs in a manner that intellectually challenges and engages students. The focus here is not on content-specific methods of instruction, but rather the extent to which the teacher (i) facilitates the lesson so that the objectives are articulated with the classroom activities, the content is presented in a clear manner, further enhancing student understanding by modeling and connecting the lesson to past content and experiences; (ii) does not simply move from one topic to the next but stops to check for understanding by posing questions to students, monitors what they are doing during independent/group work, and differentiates instruction to meet the needs of each individual student; (iii) provides feedback to deepen understanding by giving specific comments or guiding hints/questions to help students clarify misunderstandings and understands successes; (iv) encourages students to think critically by helping them identify and synthesize relevant information, analyze problems, and evaluate potential solutions. Few teachers perform well in Instruction. Almost 1 in 3 do not perform above the 2.5 threshold. In addition to the 30% that score low on Instruction, another 60% score on the medium-low range (Table 0.2). What does this mean in terms of what teachers do in the classroom? Lesson Facilitation. On average, teachers score 3 points out of the 5 points possible in this element. Figure 3.7 shows the distribution of teacher’s scores for the lesson facilitation element and the respective behaviors. While teachers can broadly articulate the class activities (60%) and clearly explain the content (50%), many of them partially model7 the learning activity (54%) and do not connect the lesson to other content knowledge or students’ daily lives (57%). Figure 3.7: Lesson Facilitation 4.1 Articulates lesson objective 4.2 Clear explanation of content 50% 60% 44% 30% 11% 6% L: Does not state objective or cannot be inferred L: Confusing or no explanation M: States broad objective or can be inferred M: Somewhat clear explanation H: States specific objective that's aligned to activities H: Clear and straightforward explanation 7To model for students, the teacher needs to perform the task or parts of the task s/he is asking the students to do. The teacher may also demonstrate her/his thinking process as part of the modeling. Results of the Mindanao Teach Observation Study | 22 4.3 The teacher makes connections in the 4.4 Models by enacting or thinking alouds lesson 54% 57% 22% 24% 26% 17% L: Does not connect L: Does not model M: Superficially or unclearly connects H: Meaningfully connects M: Partially models 100 % Proportion of Teachers 75 50 36% 29% 32% 25 1% 3% 0 1 2 3 4 5 Distribution of Scores Source: Teach Philippines 2018 Teachers fail to explicitly articulate the objectives of the lesson and relate classroom activities to those objectives. Most teachers, (60%) explicitly state a broad lesson objective like “Today, we will read a story together,” without further explanation. Others do not state it, but it can be inferred from the lesson. For instance, after giving an example of personification, the teacher clearly explains the activity when she says, “Each of you construct a sentence using personifying technique.” From this, it can be inferred they are working on personification; however, the teacher does not make an explicit lesson objective statement.  Approximately half of teachers are highly effective at clearly explaining content to students. This is indicated by their use of pictures or drawings to accompany difficult vocabulary terms and explain their meaning. Moreover, more than half of teachers (57%) do not connect what’s being taught to other content knowledge or students’ daily lives. Oftentimes, teachers will use examples with objects such as dogs and flowers that students likely encounter; however, they rarely connect these objects to the lesson content. Finally, almost a quarter of teachers (22%) do not model by enacting procedures or thinking aloud. Although they ask students to read a text, answer specific questions, or complete activities, they rarely walk them through the process of how to solve for a task. For instance, teachers ask students to solve for single addition problems by solving for “2+5” on the board, however, they do not explain the thinking behind how the two parts combined equal 7. Check for Understanding. On average, teachers score 2.9 points out of the 5 points possible in this element. Figure 3.8 shows the distribution of teacher’s scores for the checks for understanding element and its respective behaviors. Results of the Mindanao Teach Observation Study | 23 Teachers sometimes ask questions or prompt students intermittently throughout the lesson, although these questions are ineffective at determining their level of understanding (63%). Students rarely engage in group or independent work (42%), though when they do, teachers tend to monitor some students (26%). Teachers may slightly adjust teaching for some students, but this is brief and focused on a subset of the classroom, not most students (57%). Figure 3.8: Checks for Understanding 5.1 Asks thinking questions and prompts to 5.2 Monitors during independent/group determine understanding work 63% 42% 26% 25% 19% 13% 12% L: Either does not ask or askes on-worded questions L: Does not monitor students M: Sometimes asks ineffective questions M: Monitors some students H: Consistently asks effective questions H: Systematically monitors most students N/A 5.3 The teacher adjusts teaching 57% 33% 10% L: Does not adjust M: Briefly and superficially adjusts for some students H: Substantially adjusts for most students 100 % Proportion of Teachers 75 50 40% 30% 24% 25 4% 3% 0 1 2 3 4 5 Distribution of Scores Source: Teach Philippines 2018 Results of the Mindanao Teach Observation Study | 24 Approximately 63% of teachers ask a few questions or prompt students intermittently throughout the lesson; however, these questions are ineffective at determining students’ level of understanding. For instance, after reading a text, a teacher asks, "Who understood the story?", without clarifying what the students did or did not understand. Additionally, when the teacher asks, "Is this correct or not?” the students may respond in synchrony. Many teachers do not challenge students to work independently or in groups to apply what they learn. Even when they do, only one-fourth of teachers tend to monitor some students as they work and slightly less tend to do so systematically. Finally, most teachers either do not adjust the lesson to the level of the student or do so briefly. For example, when the teacher helps a group of students’ fold pieces of paper into rectangles, one of them incorrectly folds the paper twice (into fourths) rather than into half. Instead of further explaining the source of the error, the teacher simply asks, "Look, is this half, how many times was it divided?" Feedback. On average, teachers score 3 points out of the 5 points possible in this element. Figure 3.9 shows the distribution of teacher’s scores for the feedback element. Figure 3.9: Feedback 100 % Proportion of Teachers 72% 75 50 25 12% 15% 0% 1% 0 1 2 3 4 5 Distribution of Scores L: Does not provide comments or comments are simple M: Provides general or superficial comments H: Consistently provides specific and substantive comments. Note: This element has only one behavior that measures whether the teacher provides specific comments or guiding hints/questions to help students clarify misunderstandings and understand successes. Source: Teach Philippines 2018 More than 70% of teachers provide students with comments about their work; however, these comments are either general or superficial, and tend to be simple, evaluative statements. For example, after completing a problem set, the teacher says, "very good", without specifying what went well. Alternatively, the teacher provides feedback in the form of hints or asks questions to guide student thinking or procedures. For example, when asking students why the cat was scared, the teacher immediately follows up with, "isn’t it because the dog startled him from his nap?" Moreover, less than 2 out of every 10 teachers provide substantive information about what students did well on and/or help clarify misunderstandings. For example, when students write stories the teacher says, “By starting the opening paragraph with ‘no one knew what would happen’ you do a really great job of getting the readers’ attention.” Moreover, when the teacher highlights an exemplary example to the class, s/he says, “Look at the work Darna’s example, see how s/he Results of the Mindanao Teach Observation Study | 25 used the number line to solve this subtraction problem? A lot of you are drawing the number line, but not applying it to a particular problem.” Critical Thinking. Within the area of Instruction, teachers’ score lowest on critical thinking. On average, they score 2.8 points out of the 5 points possible in this element. Figure 3.10 shows the distribution of teacher’s scores for the critical thinking element and its respective behaviors. While teachers sometimes ask two or more thinking or open-ended questions (53%), they do no not build upon student responses or ask three or more questions as often (17%). Teachers provide superficial thinking tasks (61%), like comparing and contrasting content, rather than tasks that require students to analyze content at a higher level (19%). This trend is reflected in students as well, as over half of students neither ask thinking questions nor perform thinking tasks (52%). Figure 3.10: Critical Thinking 7.1 The teacher asks thinking or open-ended 7.2 The teacher provides thinking tasks questions 53% 61% 30% 17% 20% 19% L: Does not provide thinking tasks L: Does not ask thinking questions M: Provides superficial thinking tasks M: Asks a 2+ thinking questions H: Provides substantial thinking tasks H: Asks 3+ thinking questions that build upon student responses 7.3 Students ask open-ended questions, perform thinking tasks 52% 45% 3% L: Students neither ask nor perform M: Students do not ask, but perform H: Students ask and/or perform Results of the Mindanao Teach Observation Study | 26 100 % Proportion of Teachers 75 50 46% 28% 25 18% 8% 1% 0 1 2 3 4 5 Distribution of Scores Source: Teach Philippines 2018 Most teachers do not encourage students to identify and synthesize relevant information, analyze problems, or evaluate solutions. Moreover, most teachers (83%) either do not ask students’ thinking or open -ended questions or ask questions that do not necessarily build upon students’ responses. This is evidenced by a preponderance of close- ended questions throughout the lesson. For example, when students fold shapes as part of a math lesson, the teacher asks, "If the shape is folded into one-half, it means you divide the shape into how many parts?" After receiving a one- word response from the students, the teacher proceeds without further explaining. Although most teachers (61%) provide students with some thinking tasks,8 these tasks are superficial. For example, the teacher asks students to come to the board to fill in sentences with the correct word or verb, or to compare a set of facts and opinions and decipher which fit best into each category. This issue doesn’t simply lie with teachers as 97% of students do not ask thinking questions and only half of them perform a few thinking tasks like folding paper shapes into halves to learn fractions or comparing and contrasting content. 8Thinking tasks are activities or classroom tasks that require students to actively analyze content and connect it to other information, ideas, and experiences, discover meaning, draw conclusions, interpret information, make generalizations, formulate explanations (or arguments), identify patterns, consider other perspectives, make connections, or classify information. Results of the Mindanao Teach Observation Study | 27 Instruction: Heterogeneity Analysis Figure 3.11 shows the average scores across the three elements that constitute Instruction. This information is broken down into three subcategories: urban/rural school, grade level, and teachers’ educational level. Rural teachers do not differ significantly from urban teachers in Instruction, however, rural teachers are slightly better at checking for students understanding. 30% of rural teachers compared to just 18% of urban teachers ensure most students understand the lesson. This practice may be easier to implement in rural school since the student-teacher ratio is in general lower compared to urban schools (World Bank, 2013).9 There are no statistically significant differences in Instruction among teachers of different grade levels. Teachers in higher grade levels (i.e. Grade 3 and multi-grade) tend to be slightly better at instructing in a way that is conductive to learning compared to those who teach lower grade levels; however, this difference is also not statistically significant. Teachers with a master’s degree are more likely to score higher in all the elements of Instruction compared to those with less education (i.e. bachelor’s degree or lower). In particular, 42% of teachers with a master’s degree check for understanding, compared to only 22% of those with bachelor’s degree or lower. Similarly, only 10% of teachers with a bachelor’s degree provide students’ feedback compared to 25% of teachers with a master’s degree. Lastly, one -third of teachers with master’s degree challenge students to critically think compared to just half of teachers with a bachelor’s degree or less (See Figure B. 1). 9 A 2013 World Bank report on basic education and school management in the Philippines states that the comparison of city and non-city divisions indicates that, in all three indicators of student-classroom ratio, student-teacher ratio and teacher salary, non-city division schools are better off than city division counterparts. For instance, in 2010 the student-teacher ratio in non-city areas was 34:1, while in city areas it was 41:1. Results of the Mindanao Teach Observation Study | 28 Figure 3.11: Instruction – Heterogeneity Analysis Instruction by Urban/Rural 5 4 3.1 3.1 2.8 3.0 2.9 3.1 3 2.5 2.6 2 1 Lesson Facilitation Checks for Feedback Critical Thinking understanding Urban Rural Instruction by Grade Level 5 4 3.3 3.2 3.1 2.9 3.1 3.0 2.9 2.9 3.0 2.9 3.2 3.1 3 2.7 2.7 2.5 2.5 2 1 Lesson Facilitation Checks for Feedback Critical Thinking understanding Grade 1 Grade 2 Grade 3 Multigrade Instruction by Teacher's Education Level 5 4 3.2 3.4 3.1 3.0 3.2 2.9 2.8 3.0 2.9 2.9 3 2.4 2.5 2 1 Lesson Facilitation Checks for Feedback Critical Thinking Understanding < Bachelor Bachelor Master Source: Teach Philippines 2018 Results of the Mindanao Teach Observation Study | 29 3.3 Area 3: Socioemotional Skills Socioemotional Skills measures whether the teacher fosters the social and emotional skills that give students the skills to succeed both inside and outside the classroom. To develop these skills, the teacher (i) instills autonomy and provides opportunities to lead by constructing lessons where students make choices, providing them with the chance to take on roles and actively participate in a variety of forms; (ii) fosters perseverance by acknowledging not just their natural abilities, but the improvements students make over time, having a positive attitude to challenges, framing failure and frustrations as useful parts of the learning process, and encouraging students to set short- and/or long- term goals; (iii) fosters social and collaborative skills amongst students and promotes interpersonal skills, so students are able to take the perspective of others, emphasize, regulate their emotions, and solve problems by working together. Of all the areas, teachers perform poorest in Socioemotional Skills. Almost 70% of teachers perform below the 2.5 threshold. Meaning, they provide few or no opportunities for students to make choices and take on meaningful roles, rarely promote student efforts or encourage goal-setting, have either a negative or neutral attitude toward student challenges, and do not foster a collaborative classroom environment (See Table 0.2). Autonomy. On average, teachers score 2.7 points out of the 5-points possible in this element. Figure 3.12 shows the distribution of teacher’s scores for the autonomy element and its respective behaviors. Most teachers do not allow students to make choices (67%) or provide opportunities to take on meaningful roles in the classroom (32%). Relatedly, a little more than half of students (56%) volunteer to take on meaningful roles in the classroom. Figure 3.12: Autonomy 8.1 Provides students with choices 8.2 Provides opportunities to take on roles 67% 50% 32% 25% 19% 8% L: Does not provide choices L: Does not provide opportunities M: Provides some superficial choices M: Provides opportunities to take limited roles H: Provides substantive learning choice H: Provides opportunities to take meaningful roles Results of the Mindanao Teach Observation Study | 30 8.3 Students volunteer to participate 56% 35% 10% L: Students don't volunteer M: Few students volunteer H: Most students volunteer 100 % Proportion of Teachers 75 50 43% 33% 25 18% 4% 1% 0 1 2 3 4 5 Distribution of Scores Source: Teach Philippines 2018 Most teachers (67%) do not provide students with any choices on how to complete a learning activity or approach a task. In addition, most teachers (82%) either do not provide students with any opportunities to take on roles (32%) or to only take on limited ones (50%). Students may take attendance, assign tasks, pass out materials, and/or write on the board; however, they are not responsible for leading a learning activity. In contrast, most students volunteer to participate in a classroom activity in only one-third (35%) of classrooms. For instance, when the teacher asks questions such as “Who wants to come at the board?”, students are generally eager to participate in activities by raising their hands and some asking, "Me next, teacher?". Perseverance. Fostering perseverance is the weakest teaching practice among Mindanao teachers. On average, they score 2.1 points out of the 5-points possible in this element. Figure 3.13 shows the distribution of teacher’s scores for the perseverance element and its respective behaviors. Most teachers do not acknowledge student efforts (41%), most have a neutral attitude toward student challenges (80%), and almost all teachers do not encourage goal-setting (92%). Results of the Mindanao Teach Observation Study | 31 Figure 3.13: Perseverance 9.1 Acknowledges students efforts 9.2 Positive attitude toward student challenges 53% 41% 80% 7% 9% 11% L: Does not acknowledge effort L: Has a negative attitude M: Sometimes acknowledges efforts M: Has a neutral attitude H: Frequently acknowledges and identifies efforts H: Has a positive attitude 9.3 Encourages goal-setting 92% 7% 1% L: Does not encourage goal-setting M: Encourages short or long-term goal-setting, or discusses thei rimportance H: Encourages short and long-term goal-setting 100 78% % Proportion of Teachers 75 50 25 13% 5% 4% 0% 0 1 2 3 4 5 Distribution of Scores Source: Teach Philippines 2018 Results of the Mindanao Teach Observation Study | 32 41% of teachers do not acknowledge student efforts at all and 53% sometimes do, though most praises are focused on outcomes or student intelligence, rather than the effort it took to succeed. For instance, the teacher says, “very good” but does not acknowledge what was good about their efforts. Most teachers (80%) have a neutral attitude towards student challenges. Although students are not penalized for incorrect answers during a lesson on fractions, the teacher does not encourage students to persevere by seeking additional resources, discussing amongst themselves, or helping them to think through how they could tackle the challenge. Finally, there is no evidence Mindanao teachers encourage students to set short- or long-term learning goals, as over 90% of teachers do not encourage students to set goals of any sort. Social & Collaborative Skills. There is some evidence that students’ exhibit social and collaborative skills, but these behaviors are isolated and minor, and are not a core characteristic of the classroom. Similarly, there is some evidence that teachers promote students’ interpersonal skills, but this promotion is brief or superficial. On average, they score 2.3 points out of the 5 points possible in this element. Figure 3. shows the distribution of teacher’s scores for the social and collaborative skills element and the respective behaviors. A little less than half of students collaborate with one another (46%) and nearly half of teachers either do not promote interpersonal skills (44%) or do so superficially (49%). Figure 3.14: Social & Collaborative Skills 10.1 Students collaborate with one another 10.2 Promotes students interpersonal skills 46% 35% 49% 44% 19% 7% L: Students don't collaborate or display negative L: Does not promote interpersonal skills behavior M: Superficially promotes interpersonal skills M: Some students collaborate & rarely display H: Explicitly promotes interpersonal skills negative behavior H: Students consistently collaborate & display no negative beahvior 100 % Proportion of Teachers 75 50 35% 25% 24% 25 12% 4% 0 1 2 3 4 5 Distribution of Scores Results of the Mindanao Teach Observation Study | 33 Source: Teach Philippines 2018 Teachers encourage social and collaborative skills among students, but these actions are isolated, minor, and are not a core component of the lesson. In most classrooms, students either do not collaborate with one other (46%) or they collaborate in a brief and superficial way (35%). For example, students may share materials among themselves in a group, but they complete the learning activity independently and do not work to solve problem sets. Notwithstanding, a substantial portion of teachers either do not promote interpersonal skills (44%) or do so superficially (49%).10 For instance, the teacher may tell students to “help each other” during a group exercise, ask a child to say “I am sorry” to a classmate, or encourage children to take turns during an activity. The teacher may also use examples or instructional materials to encourage the idea of helping or sharing. However, the thinking or reasoning behind these actions are not acknowledged. 10This is understood as the extent to which the teacher promotes perspective-taking, empathizing, emotion regulation, and social problem- solving. Results of the Mindanao Teach Observation Study | 34 Socioemotional Skills: Heterogeneity Analysis Figure 3.15 shows the average scores across the three elements that constitute Socioemotional Skills. This information is broken down into three subcategories: urban/rural school, grade level, and teachers’ educational level. Even though rural teachers scored very similar to urban teachers in Socioemotional Skills, they tend to be slightly better at promoting autonomy and perseverance. However, they do not differ in promoting social and collaborative skills. Notably, 21% of rural teachers instill autonomy to their students, while just 15% of urban teachers perform this practice at the same level. There are no statistically significant differences in Socioemotional Skills among teachers of different grade levels. Teachers in higher grade levels (i.e. Grade 3 and multi-grade) tend to be slightly better at promoting socioemotional skills that are conductive to learning compared to those in lower grade levels; however, this difference is not statistically significant. Teachers with a master’s degree are likely to be better at promoting Socioemotional skills compared to those with lower educational level (e.g. bachelor’s degree or lower) in all the elements of Socioemotional Support (e.g. autonomy, perseverance and social and collaborative skills). More specifically, around 40% of teachers with master’s degree are provide students with opportunities to make choices and take on meaningful roles in the classroom compared to only 17% of those with bachelor’s degree or less. In addition, 25% of teachers with a master’s degree foster perseverance, while only 10% of teachers with a lower educational level do so. Lastly, 22% of teachers with a master’s degree promote students’ social and collaborative skills compared to just 15% of those with bachelor’s degree or less (See Figure B. 1). Results of the Mindanao Teach Observation Study | 35 Figure 3.15: Socioemotional Skills – Heterogeneity Analysis Socioemotional Suport by Urban/Rural 5 4 3 2.6 2.8 2.2 2.3 2.4 2.1 2 1 Autonomy Perseverance Social and collaborative skills Urban Rural Socioemotional Suport by Grade Level 5 4 3 2.7 2.7 2.7 2.7 2.5 2.6 2.1 2.1 2.2 2.1 2.3 2.2 2 1 Autonomy Perseverance Social and collaborative skills Grade 1 Grade 2 Grade 3 Multigrade Socioemotional Suport by Teacher's Education Level 5 4 3.0 3 2.5 2.6 2.6 2.1 2.3 2.3 1.9 2 1.6 1 Autonomy Perseverance Social and Collaborative Skills < Bachelor Bachelor Master Source: Teach Philippines 2018 Results of the Mindanao Teach Observation Study | 36 3.4 Explaining the Teach Results The provision of education services in many settings – including Mindanao – is characterized by a combination of centralized, but typically poor, state control. The services are provisioned by local officials with weak capacity that work within weakly governed institutions. This has resulted in a vicious cycle in which teachers and principals have gone through an education system that does not adequately prepare them, through a training system with low entry requirements that does not compensate for the flaws in the education system, or through no training at all. Without the necessary skills needed to succeed in the classroom, they are sent into a school where they struggle to teach the next generation of students. Notwithstanding, the institutional incentives for high teacher and principal performance are largely absent, with both career progression and financial rewards delinked from performance. Moreover, teacher and principal salaries and promotions are largely determined by seniority and education qualifications, unrelated to effort or performance. Finally, state and local authorities provide limited technical support or supervision to poorly performing teachers. While teachers have the autonomy to choose what and how they instruct, they do not receive strong professional development from their school principals or coaches, and often do not have complementary teaching materials. Considering this evidence, it’s no surprise teachers in Mindanao exhibit poor teaching practices. Most teachers (70%) and principals (50%) believe teachers’ main responsibility is to arrive to school on time. As such, improving student learning is not seen as a main responsibility. In fact, only 20% of teachers and 14% of principals consider improving student learning as their main responsibility so efforts are not focused on this. This section, while it does not present causal evidence, provides descriptive information to help understand the Teach results in Mindanao. This evidence is based on a teacher and principal survey that was administered as part of the study. To understand why teachers are falling short, it’s important to consider both how they scored on each behavior and how they think they scored. For 23 of the 27 behaviors, teachers exhibited wide gaps between their perception of their skill in a particular behavior and how they actually scored on Teach. Notably, this gap is wider for Instruction and Socioemotional Skills (see Figure 3.16). For example, while 65% of teachers believe they ask students questions throughout the lesson to ensure understanding, only 25% of them do so in practice. Moreover, 66% of teachers believe they help students understand that failure and frustration are normal parts of the learning process; however, only 11% actually do this in practice. This gap implies that without active policies, it will be difficult to change what teachers do in the classroom as they’re convinced they’re already doing it. Potential improvements to: (i) pre-service training policies, (ii) monitoring and accountability policies, and (iii) in-service teacher training policies may be considered to address this current deficiency. Results of the Mindanao Teach Observation Study | 37 Figure 3.16: Teachers Self-Evaluation vs Teach Scores 1.1 Treats all students respectfully 75 72 1.Supportive Environment 72 Learning 1.2 Uses positive language with students 30 1.3 Responds to student needs 57 14 1.4 Treats students of all genders with equal regard 78 90 2.1 Clear behavioral expectations 39 Expectations 35 Behavioral 2.Positive 2.2 Acknowledges positive student behavior 77 10 2.3 Redirects misbehaviors 38 32 3.1 Provides learning activity 22 62 6 Understanding 4.Lesson Facilitation 3 4.1 Explicitly articulates the objectives of the lesson 50 30 4.2 Explanation of content is clear 53 50 4.3 Makes connections 69 17 4.4 The teacher models 52 24 5.1 Poses questions and prompts students 65 5.Checks for 25 5.2 Monitors most students during independent/group work 71 19 5.3 Adjusts teaching to the level of the students 72 10 6.1 Provides comments or guiding hints for clarification 42 15 7.1 Asks thinking questions 35 17 7.Critical Thinking 7.2 Provides thinking tasks 35 19 7.3 Students ask open ended questions and/or perform… 28 3 8.1 Provides students with choices 43 ive Skills 9.Perseverance 8.Autonomy 8 8.2 Opportunities to take on roles in the classroom 62 19 8.3 Students volunteer to participate 54 35 9.1 Teacher acknowledges student's efforts 56 7 9.2 Teacher has a positive attitude toward student… 66 11 9.3 Teacher encourages goal-setting 40 1 Collaborat 27 10.Social 10.1 Students collaborate with one another 19 & 10.2 Promotes student's interpersonal skills 51 7 0 10 20 30 40 50 60 70 80 90 100 Proportion of Teachers (%) Self-evaluated him/herself as effective Scored H (High) in Teach Source: Teach Philippines 2018 Results of the Mindanao Teach Observation Study | 38 3.4.1 Pre-service Training Policies According to the sample, teachers, on average, are 42 years-old, almost all of whom have completed a bachelor’s degree and the necessary training to become a teacher. Finally, almost all teachers have a permanent contract and an average of 14 years of teaching experience (Figure B. 1). Table 3.1: Teacher Profile Mean Urban Rural Age 42 41 43 Less than bachelor’s degree 3.5 2.8 3.7 Bachelor’s degree 66.0 74.2 63.2 Master's degree (or above) 30.5 22.8 33.0 Completed teacher training 98.6 100 98.1 Permanent teachers 99.3 100 99.1 Contract teachers 0.7 0 0.9 Years of teaching experience 14.3 13 15 Source: Teach Philippines 2018 While this study did not assess teachers content knowledge or survey the institutions that provide in-service training, it did include a survey that asked teachers and principals specific questions about their induction program. From teachers’ responses, it can be implied almost 15% of teachers have not taught classes under the supervision of an experienced teacher during their induction. This evidence, combined with the findings from the previous chapter, indicate an area induction training can improve upon to better prepare teachers for the classroom. 3.4.2 Monitoring & Accountability Policies Evidence from the surveys suggest most principals conduct orientations for new hires, regularly observe classrooms, and meet with teachers to discuss their performance. Despite these best practices, principals often receive a short training on management (1 to 5 days) that does not include the necessary knowledge to conduct observations and effectively develop teachers’ practices. This is particularly problematic, as even educators find it difficult to identify the nuances of effective teaching. For example, one study found less than 50% of teachers and principals could accurately identify ‘effective’ teachers using video recordings of teacher practice. This is indistinguishable from what one would expect if they simply flipped a coin (Strong et al., 2011). Although principals can generally identify the most and least effective teachers, they are unable to distinguish the vast majority of teachers who fall in the middle of the distribution (Jacob & Lefgren, 2008). Similarly, Harris and Sass (2009) find positive but low correlations between teacher’s value added and principal judgement. 3.4.3 In-Service Training Policies Evidence from the survey suggest that 14% of teachers did not attend any teacher training in the last school year, while the number rises to 27% among teachers in the urban area. Among those who attended trainings, only 13% reported the main topic was on specific teacher practices. The lack of training in specific teacher practices is also indicative of the systemic lack of emphasis on classroom teacher practices. Results of the Mindanao Teach Observation Study | 39 References Al-Samarrai, Samer (2015). “Assessing Basic Education Service Delivery in the Philippines: The Philippines Public Education Expenditure Tracking and Quantitative Study.” World Bank. Washington, D.C. Araujo, Caridad, Pedro Carneiro, Yyannu Cruz-Aguayo, and Norbert Schady (2016). "Teacher-Quality and Learning Outcomes in Kindergarten." Quarterly Journal of Economics: 1415-53. Bau, Natalie, and Jishnu Das (2017). “The Misallocation of Pay and Productivity in the Public Sector: Evidence from the Labor Market for Teachers.” Policy Research Working Paper 8050. World Bank. Washington, D.C. Bold, Tessa, Deon Filmer, Gayle Martin, Ezequiel Molina, Christophe Rockmore, Brian Stacy, Jakob Svensson, and Waly Wane (2017). “What Do Teachers Know and Do? Does It Matter?: Evidence from Primary Schools in Africa.” Policy Research Working Paper 7956. Washington, D.C. World Bank. Washington, D.C. Cruz-Aguayo, Ibarraran, Pablo, Schady, Norbert (2017). Do tests applied to teachers predict their effectiveness? Economic Letters (159), 108-111. Gill, Brian, Megan Shoji, Thomas Coen, and Kate Place. 2016. “The Content, Predictive Power, and Potential Bias in Five Widely Used Teacher Observation Instruments.” National Center for Education Evaluation and Regional Assistance. Washington, DC. Hamre, Bridget, Robert Pianta, Bridget Hatfield, and Faiza Jamil (2004). Evidence for general and domain-specific elements of teacher-child interactions: associations with preschool children’s development. Child Development. (85), 1257-1274. Harris, Douglas and Tim Sass (2009). What makes for a good teacher and who can tell? National Center for Analysis of Longitudinal Data in Education Research. Washington, D.C. Holtzapple, Elizabeth (2004). Criterion-related validity evidence for a standards-based teacher evaluation system. Journal of Personnel Evaluation in Education. 17(3), 207-219. Jacob, Brian and Lars Lefgren (2008). Can principals identify effective teachers? Evidence on subjective performance evaluation in education. Journal of Labor Economic 26(1), 101-136. Kane, Thomas & Staiger, Douglas. (2009). Estimating Teacher Impacts on Student Achievement: An Experimental Evaluation. NBER Working Paper. Washington, D.C. Kraft, Matthew A., David Blazar, and Dylan Hogan (2018). The effect of teaching coaching on instruction and achievement: A meta-analysis of the causal evidence. Review of Educational Research. Milanowski, Anthony (2004). “The Relationship between teacher performance evaluation scores and student achievement: evidence from Cincinnati.” Peabody Journal of Education. 79 (4), 33-53. Rockoff, Jonah E. (2010). “The Impact of Individual Teachers on Student Achievement: Evidence from Panel Data.” The American Economic Review. 94 (2), 247-252. Snilstveit, Birte, Jennifer Stevenson, Radhika Menon, Daniel Phillips, Emma Gallagher, Maisie Geleen, Hannah Results of the Mindanao Teach Observation Study | 40 Jobse, et al. (2016). “The Impact of Education Programmes on Learning and School Participation in Low and Middle-Income Countries: A Systematic Review Summary Report.” Systematic Review Summary 7, International Initiative for Impact Evaluation, London, U.K. Staiger, Douglas O., Rockoff, Jonah E. (2010). Searching for effective teachers with imperfect information. Journal of Economic Perspectives. 24 (3), 97-118. UNICEF (2015). Child Poverty in the Philippines. United National Children’s Fund. Makati City, Philippines. https://www.unicef.org/philippines/ChildPovertyinthePhilippines_web.pdf. World Bank (2013). Basic Education Public Expenditure Review Phase II: School Based Management in the Philippines, An Empirical Investigation. World Bank. Washington, D.C. World Bank (2018). World Development Report 2018: Learning to Realize Education’s Promise. World Bank. Washington, D.C. Results of the Mindanao Teach Observation Study | 41 Appendix A. Teach Element Score Distributions Figure A. 1: Teach Elements – Score Distributions Classroom Culture 1.Supportive Learning Environment 2 21 53 24 2.Positive Behavioral Expectations 2 29 33 34 3 3.Opportunities to Learn 3 2 11 22 62 4.Lesson Facilitation 1 29 36 32 3 5.Checks for Understanding Instruction 4 30 40 24 3 6.Feedback 12 0 72 1 15 7.Critical Thinking 8 46 28 18 1 Socioemotional Skills 8.Autonomy 4 43 33 18 1 9.Perseverance 5 78 13 40 10.Social and Collaborative Skills 25 35 24 12 4 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Proportion of Teachers 1 2 3 4 5 Score L (1): Teacher practice is rated as very ineffective. Score M (2): Teacher practice is rated as ineffective. Score M (3): Teacher practice is rated as somewhat effective. Score H (4): Teacher practice is rated as effective. Score H (5): Teacher practice is rated as highly effective. Results of the Mindanao Teach Observation Study | 42 B. Teach Element Score Distributions by Teacher’s Educational Level Figure B. 1: Teach Elements – Score Distributions by Teacher’s Educational Level 1. CLASSROOM CULTURE Supportive Learning Environment Positive Behavioral Expectations Opportunities to Learn Master 2 22 79 33 Master 17 44 61 12 Master 13 28 93 Bachelor 9 70 180 80 Bachelor 117 108 107 3 Bachelor 10 43 77 200 < Bachelor 4 8 4 < Bachelor 5 6 5 < Bachelor 1 1 3 9 0% 20% 40% 60% 80% 100% 0% 20% 40% 60% 80% 100% 0% 20% 40% 60% 80% 100% 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 2. INSTRUCTION Lesson Facilitation Checks for Understanding Feedback Master 26 56 44 8 Master 27 49 50 7 Master 90 36 Bachelor 115 114 106 4 Bachelor 119 132 71 7 Bachelor 1 250 334 < Bachelor 4 4 7 < Bachelor 2 13 1 < Bachelor 16 0% 20% 40% 60% 80% 100% 0% 20% 40% 60% 80% 100% 0% 20% 40% 60% 80% 100% 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 Results of the Mindanao Teach Observation Study | 43 Critical Thinking Master 37 45 41 2 Bachelor 176 90 47 3 < Bachelor 7 3 3 0% 20% 40% 60% 80% 100% 1 2 3 4 5 3. SOCIOEMOTIONAL SKILLS Autonomy Perseverance Social and Collabortive Skills Master 3 48 39 40 6 Master 5 93 29 9 Master 27 39 36 23 11 Bachelor 13 150 127 51 1 Bachelor 19 273 38 10 1 Bachelor 13 150 127 51 1 < Bachelor 1 9 3 3 < Bachelor 1 15 < Bachelor 8 7 1 0% 20% 40% 60% 80% 100% 0% 20% 40% 60% 80% 100% 0% 20% 40% 60% 80% 100% 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 Score L (1): Teacher practice is rated as very ineffective. Score M (2): Teacher practice is rated as ineffective. Score M (3): Teacher practice is rated as somewhat effective. Score H (4): Teacher practice is rated as effective. Score H (5): Teacher practice is rated as highly effective. Note: The numbers inside the bars represent the number of teachers. The category “Doctoral Level” has been excluded from the analysis due the low number of observations Results of the Mindanao Teach Observation Study | 44 C. School selection for Teach survey11 To select schools that best illustrate the local context and challenges in Mindanao,12 the Northern Mindanao Region (Region X) was selected out of six Mindanao regions to represent key features of the target area. Figure C.1 illustrates the overall distribution of student learning in Mindanao based on the 2013 Grade 6 National Achievement Test (NAT) data. The Autonomous Region of Muslim Mindanao (ARRM) and CARAGA did not show the typical distribution and were excluded from the sample. Specifically, ARMM’s student learning was skewed toward lower mastery to average while that for CARAGA was concentrated to higher levels of mastery. While the rest showed comparable distribution of student learning, Zamboanga Peninsula (Region IX) and Soccsksargen (Region XII) were additionally excluded from the survey due to the on-going security concern for the fieldwork. Finally, the team decided to focus on just one region instead of choosing both Region X and Region XI. Allocating sufficient number of sample schools to different contexts within either of these regions was more efficient than splitting the sample into two regions. In choosing the sample schools as the second stage, the two parameters, (a) student learning and (b) school size, were used. These parameters were identified by the WB team as these factors showed most significant association with student learning. Other factors such as rural/urban schools, as well as mono-grade/multi-grade schools, were also considered as additional parameters, but they were mostly overlapped to the school size. To maintain the sufficient sample size, only two parameters were used. For both parameters, all public schools are classified into three levels: (i) low or small, (ii) medium, and (ii) high or big across the country. A total of 1,949 public elementary schools in Region X are mapped into 9 groups (Table C.1). The survey sample of 45 schools were randomly drawn in the proportion of schools identified and classified by these parameters and their levels as in Table C.1. The total sample size was initially fixed based on the project budget and survey’s technical features. A reserve sample was also prepared in case the survey team was not able to locate the selected school; however, none from the reserve list was used during the fieldwork. Table C.3 shows the summary statistics of the sample schools as defined by two parameters. 11 This section was written by Takiko K. Igarashi, who was also responsible for drawing the sample. 12 Compared to other regions in the world, Mindanao’s elementary enrollment and completion rates level have remained low, and certain indicators, such as net enrollment and dropout rates for high schools, have actually worsened in the same period (UNICEF 2015; See Table C. 1) Results of the Mindanao Teach Observation Study | 45 Table C. 1: School access and completion (by region, 2016) Net enrollment Completion rate Region rate (elementary) (elementary) 2015 2015 Non-Mindanao 93% 91% Region I - Ilocos Region 95% 95% Region II - Cagayan Valley 96% 91% Region III - Central Luzon 94% 95% Region IV-A - CALABARZON 90% 95% Region IV-B - MIMAROPA 92% 89% Region V - Bicol Region 91% 87% Region VI - Western Visayas 95% 90% Region VII - Central Visayas 96% 89% Region VIII - Eastern Visayas 90% 89% CAR - Cordillera Administrative Region 92% 91% NCR - National Capital Region 88% 84% Mindanao 88% 77% Region IX - Zamboanga Peninsula 90% 81% Region X - Northern Mindanao 90% 85% Region XI - Davao Region 97% 89% Region XII - Soccsksargen 88% 84% CARAGA - CARAGA 94% 86% ARMM - Autonomous Region in Muslim Mindanao 70% 40% NATIONAL Total: 91% 86% Source: Department of Education Figure C.1: Student learning in Mindanao (% of students) Source: Department of Education Results of the Mindanao Teach Observation Study | 46 Table C. 2: Public elementary schools classified into two selected parameters (Region X) Student learning Low Medium High Total Small 196 203 277 676 School Medium 170 213 242 625 size Big 217 239 192 648 Total 583 655 711 1949 Source: Department of Education Table C.3: Summary statistics of the Teach sample schools Student learning (NAT mean percentage score, Grade 6) mean sd max min High 76.09 4.93 88.77 70.95 Low 42.16 7.53 51.74 26.78 Medium 60.10 5.53 68.55 53.25 School size (Number of Grade 6) mean sd max min Large 96.7 71.4 331 50 Medium 33.5 7.5 44 24 Small 13.3 4.8 21 6 Source: Department of Education Results of the Mindanao Teach Observation Study | 47