from EVIDENCE to POLICY Learning what works for better programs and policies January 2018 BRAZIL: Can Providing Teachers with Feedback and Coaching Improve Learning? Improving the classroom effectiveness of teachers is a key In Brazil, policymakers from the Ceará state government part of improving student learning. How to get this right is a worked with the World Bank and the Brazilian non- core challenge for education systems. Teacher quality varies governmental Lemann Foundation to design a program to widely and teachers may not know the best teaching prac- improve secondary school teacher effectiveness. The program tices and how to keep students on task and engaged with the provided feedback to teachers on their classroom practices material. Especially in and gave them access to expert educational coaching through low-income countries pedagogical coordinators who were trained via one-on-one or regions, teachers may sessions delivered via Skype. With support from the World EDUCATION not have the resources, Bank’s Strategic Impact Evaluation Fund, the World Bank team knowledge or motiva- incorporated an impact evaluation into the program to test the tion to teach effectively. effectiveness of the approach. The evaluation found that over Governments and de- the course of the year, teachers’ classroom practices improved, velopment partners in- teaching time increased, students were more engaged, and vest heavily in teacher students’ standardized test scores improved. Based on the training. Evidence on which approaches raise student learn- results, the Ceará government made the program’s curriculum ing is crucial for education progress. and technology available to municipal schools in 2017. Context The amount of time teachers spend actually teaching while of teacher working hours for in-school teacher collabo- in the classroom can vary widely. A 2015 World Bank re- ration and development programs. In 2014, the Ceará port* on teaching in six countries in Latin America and education secretariat, located in the country’s northeast the Caribbean found that on average, teachers lose the and one of Brazil’s most populous and poorest states, equivalent of one day of school per week because they fail worked with SIEF researchers to develop and test a train- to make full use of class time for instruction. They spend a ing program for secondary schools. The program was im- quarter or more of class time on administrative duties, such plemented with support from the Lemann Foundation, as cleaning the blackboard or taking attendance, and are a non-profit focused on education. Coaches focused on absent about 10 percent of the time. Even when they are strengthening the capacity of the schools’ pedagogical co- teaching, teachers rely on the “chalk and talk” method— ordinators, who are responsible for providing individual lecturing to students while writing the information on the feedback to teachers and promoting teacher collabora- blackboard—and often fail to keep students engaged. tion and best teaching practices within schools. However, To address these challenges, a federal government pol- surveys before the program revealed that pedagogical coor- icy in 2013 mandated that schools free up about a third dinators rarely observed teachers in the classroom. *Bruns, Barbara; Luque, Javier. 2015. Great Teachers: How to Raise Student Learning in Latin America and the Caribbean. Washington, DC: World Bank. The program had two main elements. First, schools Second, the pedagogical coordinators participated in received benchmarked performance feedback based on three face-to-face one-day training sessions with the coach- an initial round of classroom observations at the end ing team on key strategies for increasing instruction time “ of the 2014 school year. and keeping students engaged. The data was gathered us- In developing countries, teacher quality Over the year, each pedagogi- ing the Stallings “classroom can matter even more than in wealthier countries. cal coordinator also had at least snapshot” method. The re- two private sessions with an But most education systems do not attract ” sults were shared through assigned coach via Skype. Co- school-specific bulletins that applicants with strong backgrounds. ordinators also had access to a provided easy-to-compare From the World Development Report 2018, LEARNING to Realize private website with support Education’s Promise data on things like the share materials for lesson planning of class time teachers used for instruction, pedagogical and for sharing videos of themselves providing feedback to techniques, classroom materials used most frequently, their teachers, so the coaches could make suggestions. Co- and the share of class time students were engaged. Teach- ordinators and all teachers in the treatment schools also re- ers weren’t identified by name, only by the hour, grade ceived Aula Nota 10, the Portuguese translation of Teach Like and subject taught, and these results were benchmarked a Champion, a book on effective teaching by Doug Lemov, an against the top-performing school in their district, and educator from the United States. Finally, teachers and their against the averages for Ceará state, for Brazil and Stall- coordinators had access to online videos filmed in Brazilian EDUCATION ings good practice benchmarks. classrooms that illustrated examples from the book. Evaluation The program was evaluated through a randomized control School system supervisors and pedagogical coordina- trial. Of Ceará’s 573 secondary schools, 292 were randomly tors from the treatment schools were trained to use Stall- selected to participate in the program, with 156 treatment ings to collect the classroom data. Supervisors and coordi- and 136 control schools. Baseline classroom observations nators didn’t observe classrooms in their own districts.To were conducted over five weeks in November and Decem- prevent contamination, they were sent to other schools to ber 2014, which corresponded to the end of the school year collect the data, and coordinators from the control schools in Brazil. Endline classroom observations were conducted were not involved in the observations. Student learning re- in November 2015, at the end of the 2015 school year. sults were measured on two different tests—the Ceará state All classroom observation visits, which used the Stallings achievement test and the national high school exit exam. methods to collect data, were not announced in advance. An advantage of the Stallings method is that it requires Brazil’s education at a glance: relatively little training to use and generates comparable data across different subjects, languages, types of school Brazilian 15-year-olds lag almost three years behind their OECD and country contexts. Using an electronic tablet, observers peers in math and more than four years behind peers in top- code what the teacher is doing, what materials she or he is scoring countries like Singapore, China and Japan. using, and what students are doing at 10 equally-spaced From “Through the Looking Glass: Can Classsroom Observation and Coaching Improve Teacher Performance in Brazil ” intervals in each class. This policy note is based on “Through the Looking Glass: Can Classsroom Observation and Coaching Improve Teacher Performance in Brazil,” Barbara Bruns, Leandro Costa, Nina Cunha, Education Global Practice Group, July 2017, World Bank Policy Research Working Paper 8156. Results By the end of the school year, teachers in the just 20 percent of class time, meaning that there are al- program schools spent almost 10 percent more most always one or more students “tuned out” in Ceará of their time in the classroom teaching; they classrooms. achieved this by reducing time spent on classroom management and time out of the classroom. Teacher instruction time in schools that received the pro- gram increased to 76 percent of class time, as compared with 70 percent in control schools. This increase trans- lated into an additional 59 more hours per year, or an additional two weeks of schooling per year. The reason for the change was that teachers spent much less time on classroom management and off-task. For example, the amount of time on classroom management, such as tak- ing attendance, grading papers and disciplining students, dropped to 18 percent of class time in schools that re- ceived the program, compared with 21 percent of class time in schools that didn’t receive the program. Teachers’ time off-task, such as being out of the classroom or chat- ting with visitors, dropped to 5.8 percent of classroom time compared with 8.4 percent in schools that didn’t receive the program. There also was a decline in the share of class time that teachers were out of the room. In treat- ment schools, this fell to three percent, compared with five percent in the control schools. All the results were statistically significant. In schools where teachers received the mentoring Teachers in the program were more likely to and feedback, they were better able to keep use interactive teaching methods, but they still students engaged in classroom activities. spent the bulk of their time lecturing from the blackboard. Progress in this area was modest, but teachers that re- ceived the feedback and coaching support were able to Both the coaching program and the book that teachers re- reduce the share of time a large group of six or more stu- ceived, Teach Like a Champion, (Aula Nota 10 in Portuguese) dents were visibly off-task—talking, texting, day dream- stressed the importance of using questions as a method for ing or otherwise not paying attention—to 16 percent, stimulating class discussion and as a way to measure students’ compared with 19 percent in control schools, a statisti- understanding of the material. Use of these techniques by cally significant change. But teachers in program schools teachers in the program accounted for 10.5 percent of class made little headway in keeping the entire class engaged. time by the end of the school year, compared with 8.4 percent Indeed, schools on average had the entire class engaged in control schools. However, lecturing in front of a blackboard remained the dominant teaching mode—used 38 percent of viations of 0.04 and 0.06, respectively. Program impacts were the time on average in treatment schools, as compared to 34 strongest in the classrooms that were weakest at the start of percent in control schools. In all the cases, the differences the program. In the bottom 25 percent of classrooms—de- between the treatment and control groups after the program fined as classrooms in which teachers spent the largest time on were statistically significant. classroom management rather than on instruction—student test scores on the Ceará state exams were eight points higher The program led to an increase in math and in math and five points higher in Portuguese (0.17 and 0.12 Portuguese test results, with the strongest standard deviations, respectively)—and on the national exam impacts seen in the classrooms where teachers they were 14 points higher in both math and Portuguese, initially had the lowest times on instruction. (0.14 and 0.15 standard deviations, respectively) when com- pared with the control schools. A key goal of the program was Students in schools that received the program scored four to reduce the large variation in teacher performance within points higher in math and two points higher in Portuguese schools by helping weaker teachers to improve. It was hypoth- on the Ceará state assessment, SPAECE, which translates esized that exposing gaps in classroom practice between the into 0.08 and 0.05 standard deviations, respectively. On the best and worst teachers in the school would create motivation national high school graduation test, ENEM, students in to improve, and the pedagogical coordinators and coaches schools in the program scored four points higher in math and would provide support. These results indicated that the pro- five points higher in Portuguese, translating into standard de- gram succeeded at this. EDUCATION Conclusion Improving the effectiveness of teachers in service is one single school year. It also showed that there are cost effec- of the biggest challenges school systems face. Developing tive ways to approach teacher training and improve stu- countries spend huge amounts on teacher training each dent learning. The teacher feedback and coaching program year and while very few programs are rigorously evaluated, cost $2.40 per student, less than one percent of Ceará’s there is little evidence of impact on teachers’ skills. Brazil— annual spending per secondary student. The research also as one of the lowest-scoring countries on the 2015 PISA shows the value of using standardized classroom observa- exam—has large challenges in education, but this impact tions to measure how changes in teacher practice produce evaluation showed that real progress is possible, even in a improvements in student learning. The Strategic Impact Evaluation Fund, part of the World Bank Group, supports and disseminates research evaluating the impact of development projects to help alleviate poverty. The goal is to collect and build empirical evidence that can help governments and development organizations design and implement the most appropriate and effective policies for better educational, health, and job opportunities for people in developing countries. For more information about who we are and what we do, go to: http://www.worldbank.org/sief. The Evidence to Policy note series is produced by SIEF with generous support from the British government’s Department for International Development and the London-based Children’s Investment Fund Foundation (CIFF). THE WORLD BANK, STRATEGIC IMPACT EVALUATION FUND 1818 H STREET, NW WASHINGTON, DC 20433 Series editor: Aliza Marcus; Writer: Daphna Berman