IMPLEMENTATION BRIEF What is Teach? Teach’s value proposition Teach is a free classroom observation tool that provides a window Teach holistically measures what happens in the classroom. into one of the less explored and more important aspects of a It does so by considering not just time spent on learning but, student’s education: what goes on in the classroom. The tool is intended to be used in primary classrooms (grades 1-6) and was more importantly, the quality of teaching practices. designed to help low- and middle-income countries track and improve teaching quality. • Teach captures practices that nurture children’s cognitive and socioemotional skills. • Teach was developed with low- and middle-income countries in mind and can be contextualized for different settings. For Why is it important to measure instance, additional elements can be added at the request of the government and local video footage is used to train observers teaching practices? on the tool. The learning crisis is, at its core, a teaching crisis. A growing body • Teach includes a free complementary toolkit that helps teams of research indicates teaching is the most important school-based determinant of student learning. The difference between the impact conduct the training with a detailed script and training guide, of a weak and great teacher on student test scores is equivalent collect data using a data collection app available in several to one to two years of schooling. Moreover, evidence suggests languages, and clean and analyze data with automatized several consecutive years of effective teaching can offset learning programs — including assessing the validity of Teach scores. shortfalls and help students reach their full potential. Although A template report to communicate the results is also available. better teaching practices are needed to tackle the learning crisis, most education systems in low- and middle-income countries do • Teach has been already applied in a dozen low- and middle- not regularly monitor them. income countries and shown to be reliable. Emerging evidence indicates it is also a good predictor of student learning. How can Teach be used? Teach framework Teach can be used as a tool as a tool for system diagnostic and for professional development. As a system diagnostic, Teach  allows governments to monitor the effectiveness of their policies to improve teacher practices. As a professional development  tool, Teach is used to identify individual teachers’ strengths and weaknesses. The World Bank is currently developing Coach, which will help principals and coaches use the information from Teach to provide targeted feedback on how teachers can improve their classroom practices. 2 3 What does Teach measure? Development and validation Teach differs from other classroom observation tools in that it captures Before the launch of the tool, Teach underwent a rigorous development (i) the time teachers spend on learning and the extent to which and validation process over a 2-year timeframe. A Technical Advisory students are on task, and (ii) the quality of teaching practices that help Panel provided extensive feedback and inputs on the tool’s design. develop students’ socioemotional and cognitive skills. Teach was also piloted in over 1,000 classrooms across Mozambique, Pakistan, the Philippines, and Uruguay, and tested with global video As part of the Time on Task component, 3 snapshots of 1–10 seconds footage from 11 low- and middle-income countries. Analyses of the are used to record both the teacher’s actions and the number of training data indicate that after only 4 days, almost 90% of participants students who are on task throughout the observation. The Quality of passed the Teach Reliability Exam, which involves coding 3 videos Teaching Practices component, on the other hand, is organized into reliably. Meaning, for each video, the participants scored within 1-point 3 primary areas as shown below: Classroom Culture, Instruction, and of the master codes, at least 80% of the time (Table 1). This feat was Socioemotional Skills. These areas have 9 corresponding elements achieved by local observers who have a comparable level of education that point to 28 behaviors. The behaviors are characterized as to the average citizen in their country and who had no previous low, medium, or high, based on the evidence collected during the experience conducting classroom observations. observation. These behavior scores are translated into a 5-point scale that quantifies teaching practices as captured in a series of two, Table 1: Teach Reliability Exam pass rate 15-minute lesson observations. COUNTRY N OF OBSERVERS PASS RATE Mozambique 46 74% Pakistan 53 96% Philippines 25 96% Uruguay 21 100% CLASSROOM INSTRUCTION: SOCIOEMOTIONAL Raters also have high levels of inter-rater reliability. Results from CULTURE: SKILLS: field trials from 845 schools in Pakistan show that paired raters have The teacher instructs The teacher creates a in a way that deepens The teacher fosters 97% agreement within 1 point and 87% agreement within a half-point culture that is conducive student understanding and socioemotional skills that (Molina et al., 2018). The Intraclass Correlation Coefficient (ICC) is to learning. The focus encourages critical thought encourage students to .75 for the overall Teach score, indicating strong inter-rater reliability. here is not on the teacher and analysis. The focus here succeed both inside and Finally, teachers who display effective practices, as measured by Teach, correcting students’ is not on content-specific outside the classroom. To negative behaviors but methods of instruction, but develop students’ social are associated with students who achieve higher learning outcomes. rather the extent to which rather, the extent to which and emotional skills, the the teacher creates a the teacher facilitates teacher instills autonomy, supportive learning the lesson, checks for promotes perseverance, environment and sets understanding, provides and fosters social and positive behavioral feedback, and encourages collaborative skills. expectations. students to think critically. 4 5 5. Observers Implementing Teach: collect data From conception to completion EXPECTED TIME: 21–28 DAYS After the observers are trained, they proceed This implementation roadmap provides a detailed timeline, cost to conduct classroom estimate, and list of complementary resources needed to apply Teach observations using Teach. Data collection is either in a new setting. Though the timeline and cost may vary slightly based 3. Implementors done by the government or on the size of the survey and context-specific realities, the roadmap use collected an external entity, depending captures the implementation process from start to finish and outlines videos to create on the context. In either scenario, observers collect the recommended steps teams should follow to ensure to timely and master codes data using PAPI or CAPI. The accurate data collection. Teach team provides a do-file EXPECTED TIME: 14–21 DAYS (Stata) that automatizes In addition to the 3 the data cleaning process videos prepared by the 2. Teach trainer trains 5-8 Teach trainer, all 5-8 and generates a report that 6 highlights potential mistakes. local implementors on tool implementors prepare master codes for 7 EXPECTED TIME: 5 DAYS (15-minute) local videos Using 3 of the 15 collected videos (2 for practice, 5 for the 5 (2 for practice, 1 for the certification certification exam) to be 0. Consultations exam), the Teach trainer prepares local used during the observer master codes for the implementor training. These materials on Teach training. The remaining 7 videos are prepared under the needed for the training are sourced guidance and supervision 6. Teach resources EXPECTED TIME: VARIES Before Teach becomes from the Teach international library. This training is designed to (i) ensure of the Teach trainer. automatize data an active activity in any cleaning, analysis, 4 implementors can reliably code project, an in-depth using Teach (3 days) and (ii) provide & validation 3 consultation with implementors with training strategies government officials, they can apply during the observers’ EXPECTED TIME: 1 DAY development partners, training (2 days). Implementors will After the data have been or researchers is be given a host of complementary cleaned, another do-file held to review the training materials, including a full- (Stata) automatically tool’s structure and length training script, to assist them analyzes the data, which outline steps for with the observer training. At the end generate descriptive implementation. of the implementor training, the Teach 4. Implementors statistics and confirm the trainer will choose 1-2 implementors to train observers validity of the Teach scores 1 conduct the observer training. (Excel/Stata graphs and 0 EXPECTED TIME: 5 DAYS tables). These outputs The training is designed are then inputted into pre- to i) ensure observers can formatted MS Word file or reliably code using Teach (3 a tex file (LaTeX). 2 days), ii) conduct a practice observation at local schools 1. Source local (1 day), and iii) ensure observers can accurately videos for training and consistently code classroom observations in EXPECTED TIME: 3-5 DAYS accordance with the Teach 15 full-length lessons (60-minute) rubric as demonstrated by are recorded by government, country passing the reliability exam office, or local staff. When possible, (1 day). For best results, we these videos should reflect the larger recommend 1 trainer for sample. Of these, ten (15-minute) every 20 observers. video segments are selected for the implementor and observer trainings, based on video quality and variation of practices observed. 6 7 Teach complementary toolkit RESOURCE DESCRIPTION Implementor This individual is a necessary hire. S/he is responsible for passing the Teach reliability exam (TOR) and developing a set of master code justifications under the direction and oversight of the Teach trainer. 1-2 of the implementors will be chosen by the Teach trainer to facilitate the observers’ training, based on performance. 0. Consultations on Teach Quality This individual is an optional hire and is particularly useful if the training is being conducted Brief & These documents can be used by World Bank staff to present Teach to different audiences, Assurance for a large number of observers or in a low-capacity setting. S/he is responsible for helping Slide Deck including, but not limited to: governments, donors, and partners. They provide a high-level Assistant (TOR) the Teach trainer prepare the materials for the training of implementors and helps manage overview of why teaching practices matter for student learning, current shortfalls in how many and oversee the development of master coders for the observer training. countries measure teaching practices, what Teach is, how it addresses these shortfalls, how it differs from other tools, and background on the development and validity of the tool. 5. Observers collect data Teach Expert This individual is an optional hire. S/he is hired to help draft an additional element at the request of the government. This consultant either drafts the element in full or advises government Guidelines for This document provides a step by step explanation on how to create a web and mobile (TOR) version of the questionnaire to be collected on computer/tablets, how to use the web version officials as they draft the element, depending on the government’s needs and preferences. Using Electronic Application and the app version, and how to export the data collected. 1. Source local videos for training SurveyCTO This is a computer-assisted personal interviewing style wherein observers input scores (CAPI) digitally using SurveyCTO. This form makes the survey available on phone/tablets/computer Guide for This document provides a step by step explanation on how to select and record classroom in English, French, Spanish, and Portuguese. The team will have to create a SurveyCTO server Selecting & video footage for Teach. to collect data using this form. Data can be collected using a computer or the SurveyCTO Recording Videos app on tablets/phones. Once the data is collected, it can be exported from SurveyCTO as an Excel file. This file is perfectly aligned with the pre-created do-files that will automatically label, clean, and analyze the data. Recording The document is a legal form teachers must sign before their classroom is recorded. By Authorization signing the form, teachers not only consent to be recorded, but for their recordings to be used as part of the Teach training. It also provides the option of permitting their videos to be made Data Entry This is a pen-and-paper interviewing style wherein observers input scores on the Teach Form (PAPI) observation form and manually transfer them to an Excel database. In the case where PAPI publicly available online. data collection is done, this Excel file is a template for data entry and is perfectly aligned with the pre-created do-files that will automatically label, clean, and analyze the data. If the team Video Editor This individual is an optional hire. S/he is responsible for editing the local classroom lesson wishes to program the data entry to another software such as CSPro, a file is available that (TOR) videos into 15-minute segments and embedding subtitles into the training videos (where details the logic of the questionnaire. applicable). Translator (TOR) This individual is an optional hire. S/he is responsible for translating official Teach documents Program for After data has been collected, either through CAPI or through PAPI, a pre-written do-file (Stata) labeling the data automatically labels the data and saves it as a first Stata dataset. from English to the local language. The manual is available in Spanish, Portuguese, and French; if a project team is using a language other than these, it needs to be translated. using Stata All other documents in the complementary toolkit are currently only available in English. Survey Firm This document provides the terms for a survey firm to oversee the collection of data and (TOR) hiring of field supervisors and observers to implement Teach. 2., 3., & 4. Teach trainer trains implementors; implementors create local master codes & train observers 6. Teach resources automatize data cleaning, analysis, & validation Guidelines for This document provides a step by step explanation on how to write master codes. A master Program for After data has been collected and labeled, a pre-written do-file (Stata) automatically checks Writing Master code is a detailed justification for why a behavior or element, as observed in classroom cleaning the for strange patterns in the data and exports the data to be checked in an Excel file. This Codes footage, warrants a particular Teach score, which is established by a consensus of several data using Stata Excel file can be shared with the team or firm responsible for data collection. In this do-file, master coders. corrections can also be applied, following the feedback from the data collection team. The do-file saves the cleaned data as a second Stata dataset. Training Manual This document includes all resources necessary to conduct the 5-day reliability training. This includes a detailed script, 5-day agenda, example bank, in-class activities, common challenges Program for After data has been cleaned, a pre-written do-file (Stata) automatically analyzes the data and by element, and overall tips for trainers to execute a successful training. Reliability using confirms the validity of the Teach scores (Stata graphs and Excel tables). Stata Training This presentation is to be used during the 5-day reliability training. It is aligned to the training PowerPoint manual and includes video clips and detailed notes for participants to practice and identify Program for After data has been cleaned, a pre-written do-file (Stata) automatically analyzes the data, which the tool’s various behaviors, elements, and areas. generates descriptive statistics in Excel and analytical graphs (both in Excel and Stata format). Analysis using Project This document provides information on the roles, responsibilities, and timeline needed to Stata Management conduct the master coding process. MS Word This document is a sample report. This template has automatically generated sections with Plan areas (text and graphs) to be filled in with country-specific results. The graphs and tables Template produced by the do-file can be manually included in the report. Teach This Excel file automatically calculates the participants’ reliability score and whether they Reliability Exam passed the Teach reliability exam. Tex File (LaTeX) This document is a sample report. This template has automatically generated sections with texts to be filled in with country-specific results. The graphs and tables produced by the do-file Teach Trainer This individual is a necessary hire. S/he is likely a member of the Teach team, who is are automatically integrated. (TOR) responsible developing master codes for the implementors’ training and conducting the training. S/he then oversees the implementors as they develop the master codes for the Stata/Analysis This individual is an optional hire and is particularly applicable if the project calls for a highly observers’ training. The Teach trainer will choose 1-2 implementors to lead the observers’ (TOR) technical analysis of the Teach findings. S/he cleans the data and conducts an in-depth training, based on performance. analysis for use in an extended report or presentation. 8 9 What Task Team Leaders How much does Teach cost? are saying about Teach The cost to implement Teach varies by context; however, the implementor training, training package, local master-coded videos used Applicability in Policy dialogue Teach training for the implementor training, support for observer training, software local context to collect data, automatized results, and travel costs are estimated at ≈$8,000 USD. These costs may vary based on the consultancy rate of the “For our new project, “Teach provided the “Preparing for the Teach Trainer, which ranges from $300–$400 USD per day. This estimate we had to produce a evidence needed Teach training involves baseline of teaching to have a real collecting videos and does not include local implementors’ time, facilities for the training, or practices in a very conversation regarding creating master codes data collection costs, which will vary depending on sample size and tight timeframe — our the challenges using the tool. The location. It also does not include additional translation costs. counterparts put a teachers’ face in process of master large emphasis on classrooms. With coding local videos both the quality of the this information we is a crucial step in The manual is currently available in Arabic, Bulgarian, English, French, instrument and its were able to engage the implementation Portuguese, Spanish, and Swahili, if teams need the manual translated adaptation to the local the government on of Teach. When context. Teach allowed specific ways the Bank implementing Teach, to a language other than these it costs $600–$1,000 USD. Moreover, us to meet both of can support these we had a lot of the master coded videos and descriptions for the Teach training are these requirements and teachers to strengthen questions from our currently only available in English; additional translation and subtitling our deadline in a very their competencies and client on whether cost-effective way.” pedagogical skills.” we know what fees may apply for use in contexts where master coders do not the behaviors and Francisco Haimovich Franco Russo, understand English (≈$2,500 USD). These translation costs may be Paz, Uruguay Philippines elements mean, and whether our observers partially subsidized by the Teach team, where applicable. could understand “At first, I was skeptical “Teach gave us the them. After master to use a high inference chance to discuss coding the videos, we tool in Mozambique. several issues with the trained and certified However, the simplicity government, including observers, which of Teach and the fact it the importance of took a few attempts uses local videos made observing teachers, to get right. Despite it possible implement which was relevant all this, we reached with a high degree of to introduce a culture an extremely high reliability.” of monitoring and reliability rate of 96%. feedback. It helped us In the field, the strong Marina Bassi, to identify gaps in the training really paid Mozambique teacher professional off, as we reached development extremely high IRR. component of our Through this process, project and plan we were much more innovations for the knowledgeable of the future.” tool and were able to Helena Rovner, convince our partners Uruguay and observers that this was done in the right way.” Koen Martijn Geven, Pakistan 11 What experts are saying about Teach “Teach is a magnificent example of research taken to the practical level with the possibility of providing enormous social value. This initiative turns research and evaluation into a clear improvement mechanism.” Eric Hanushek Paul and Jean Hanna Senior Fellow, Hoover Institution, Stanford University “Before Teach, the lack of open source, flexible, easy-to-learn observational measures that can be used systematically in classrooms has stood as a major stumbling block in international efforts to improve education.” Sara Rimm-Kaufman Professor of Education, Center for Advanced Study of Teaching and Learning, Curry School of Education, University of Virginia “Teach represents a major innovation in our efforts to improve education for all. It will be catalytic for enhancing learning all around the world.” Oon-Seng Tan Director, Centre for Research in Child Development, National Institute of Education, Singapore “Teach is an essential resource for countries seeking to improve education results. It combines the best features of prior instruments into an all-in-one package.” Barbara Bruns Center for Global Development and Walsh School of Foreign Service, Georgetown University “Teach provides excellent guidance for observing and rating global classroom instruction. It is impressive not only for its comprehensiveness, but also its specificity, naming key classroom practices, and describing concrete examples of how those practices occur at different levels of quality.” Heather Hill Jerome T. Murphy Professor in Education, Harvard Graduate School of Education; Creator of the Mathematical Quality of Instruction (MQI) instrument “Teach provides a practical tool for educators around the world who are serious about improving the quality of classroom practice.” Pam Grossman Dean and George and Diane Weiss Professor, Graduate School of Education, University of Pennsylvania; Creator of the Protocol for English Language Arts Teaching Observation (PLATO) instrument “Teach has clearly been designed with the realities of the Global South in mind. The clear explanations, well-crafted examples, and FAQs ease interpretation and ensure commonality of understanding between observers.” Sara Ruto Director, People’s Action for Learning (PAL) Network Contact us at teach@worldbank.org and visit us at www.worldbank.org/education/teach