Policy Research Working Paper 8970 Using Student and Teacher Assessments to Design More Pertinent In-Service Teacher Training The Case of Ecuador Diego F. Angel-Urdinola Sebastian Burgos-Dávila Education Global Practice August 2019 Policy Research Working Paper 8970 Abstract The development of pertinent and effective in-service teacher student and teacher assessments. Although countries in the training remains a policy challenge for many countries in region have made important efforts to carry out periodic Latin America and the Caribbean. Ecuador stands out as a evaluations of student and teacher performance, the data country in the region that has made significant investments resulting from these evaluations are rarely used to guide in teacher training in the past decade. However, most in-ser- teacher development programs. The analysis presented in vice training provision has been designed without enough this paper suggests that doing so has the potential to raise elements to properly address teachers’ skills gaps. This paper program pertinence while allowing the prioritization of proposes a roadmap for improving the design of in-ser- investments in teachers and students with the greatest needs. vice teacher training in Ecuador using available data from This paper is a product of the Education Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at dangelurdinola@worldbank.org. The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Produced by the Research Support Team Using Student and Teacher Assessments to Design More Pertinent In- Service Teacher Training: The Case of Ecuador Diego F. Angel-Urdinola and Sebastian Burgos-Dávila The World Bank1 JEL Classification: I20, I24 Keywords: Education quality, teacher training, professional development, Ecuador.                                                              1 This paper was prepared to guide the technical assistance to support the implementation of Ecuador’s Comprehensive System for the Development of Education Professionals (SIPROFE) program within the context of the World Bank’s Supporting Education Reform in Targeted Circuits Project. The authors acknowledge the technical support and strategic guidance of Rafael de Hoyos, World Bank; Juan Ponce, Facultad Latinoamericana de Ciencias Sociales (FLACSO, Ecuador); and Jorge Yepez, Hortencia Bustos, and Lucia Pinto; Ministry of Education of Ecuador. 1    I. Introduction In-service teacher training is an essential component of Ecuador’s teachers’ professional development strategy. However, available training interventions offered by the state do not necessarily respond to the teachers’ skills gaps. This occurs because available training programs do not systematically rely on diagnostics of the teachers’ needs. This paper proposes a roadmap for improving the design of in-service teacher training in Ecuador using available data from student and teacher assessments. Current teacher assessments in Ecuador have an active cognitive component capable of assessing the level of knowledge of teachers in the field they teach. They provide valuable information for determining the specific areas of knowledge that teachers have problems with. Coupling such information with available data on student academic achievement, the state could develop not only more pertinent in-service training programs, but also target interventions to teachers that serve the student populations with the greatest needs. Many countries in Latin America and the Caribbean (LAC) face shortages of good teachers, which undermines the capacity of their systems to improve student achievement in the short, medium, and long term (Rivkin, Hanushek, and Kain 2005; Hanushek and Woessmann 2006; Figlio and Kenny 2007; Chetty, Friedman, and Rockoff 2014). Individuals who become teachers in LAC, most of whom are female, tend to be socioeconomically vulnerable and to underperform in university entrance exams compared to the average university student (Bruns and Luque 2014). Many structural factors explain this phenomenon, such as the lack of incentives to choose teaching as a profession and high turnover of teachers (Ingersoll 2004; Woessmann 2011; Muralidharan and Sundararaman 2011). Moreover, salaries for teachers in the LAC region tend to be substantially lower than in the average profession (Moura-Castro and Ioschpe 2007), which limits the capacity of the sector to attract the best professionals. Despite remarkable progress in teachers’ educational attainment (at present, most teachers in the LAC region have completed a tertiary degree), graduates from the teaching profession enter classrooms with significant cognitive and pedagogical gaps (Bruns and Luque 2014).2 These deficiencies weaken education systems in the region, as teachers’ cognitive skills influence the achievement of their students (Hanushek, Piopiunik, and Wiederhold forthcoming).                                                              2 According to data from Ecuador’s Ministry of Education (MINEDUC), the public system of primary education in Ecuador employs 163,999 teachers. Of this total, 81 percent have a bachelor’s degree in education, 14 percent have a postgraduate degree (mainly in education), and the remaining 5 percent are not college graduates. 2    In response to the cognitive gaps of teachers, countries in LAC have relied on in-service teacher training as part of their teachers’ professional development strategies (Levy and Schady 2013; Avalos 2007). In-service teacher training is a significant element of non-salary spending in most LAC education systems (Bruns and Luque 2014). In some countries in the region, teachers participate in training between 21 and 35 days per year (OECD 2009). A primary weakness of in-service teacher training programs is that they seldom respond to the actual skills gaps of teachers (Bruns and Luque 2014; Popova et al. 2018). Ávalos (2007) conducted a review of several in-service teacher training programs in LAC. The study concludes that in-service teacher training in the region should focus more on strengthening teachers’ cognitive deficiencies, given the fragility of their initial training. Although teacher effectiveness involves several factors (such as adequate mastery of the curricula, high motivation, and proper use of pedagogy), the author highlights the importance of prioritizing training efforts in the region as a corrective measure to help teachers reach an adequate mastery of the curricula. Ecuador has made substantial efforts to improve teacher quality in recent years. In the past decade, the Government of Ecuador has embraced important reforms in teacher development practices including (a) periodic assessment of teachers through standardized tests; (b) setting standards for the admission of teachers into the elite group of nationally recognized teachers, known as the Magisterio Nacional; (c) better teacher remuneration, partially linked to performance evaluations; (d) the development of a structured teaching career path; and (e) creation of the National Education University (Universidad Nacional de Educación) for new teachers (Angel- Urdinola and Vera-Jibaja 2018).3 Moreover, Ecuador has invested significant resources in teacher training. Nevertheless, in-service training programs in Ecuador have not necessarily responded to teachers’ development needs, and their effectiveness remains unassessed. Teachers who underperform in their knowledge assessments do not always participate in available training programs, take-up rates for some available programs are low, and there is no hard evidence that training programs have contributed to improving teaching effectiveness or student learning.                                                              3 Between 2006 and 2015, teacher salaries increased on average from US$396 to US$817 per month (surpassing inflation). Moreover, 17,000 new teaching positions were created from 2011 to 2013, contributing to an 11 percent expansion in the number of teachers employed by the MINEDUC (Harvey 2016). In 2016, 90.6 percent of the teachers were regularized with open-ended appointments, and 29.4 percent (or about 479,000 teachers) benefited from a salary increase due to the Recategorización Program. 3    This paper proposes a roadmap for improving the design of in-service teacher training in Ecuador using available data from student and teacher assessments. Even though Ecuador has made substantial efforts to periodically evaluate student and teacher performance, the data resulting in these evaluations have not been used systematically to guide teacher development programs. The results of this study indicate that systematically using the data has the potential to help design more relevant programs that respond to observed skills gaps and allow prioritizing investments in teachers and students with the most substantial needs. The study also intends to raise awareness about the potential gains of capitalizing on available information on student and teacher assessments. Many top education systems invest heavily in student and teacher evaluations. Finland, Japan, the Republic of Korea, China, the United States, and Singapore, among others, all have robust systems that rely on comprehensive student and teacher assessments that serve as inputs to develop more pertinent in-service teacher training programs. However, to date, the use of data resulting from teacher evaluations in Latin America for formative purposes is much more limited (Bruns and Luque 2014). Chile and Mexico are the education systems in Latin America with the most consolidated systems of teacher evaluation. Among other information, both systems collect information about teachers’ cognitive skills. Data resulting from these assessments are often used to make decisions about a teacher's career progression, but not necessarily to develop tailor-made in-service training programs. Thus, the framework presented by the study applies to other countries that, like Ecuador, dispose of detailed information about cognitive skills gaps of teachers, but largely underuse it. The paper is structured as follows. Section II briefly describes the design and implementation of, and investments made in, in-service teacher training in Ecuador by the Ministry of Education (MINEDUC) under its main national teacher development program, the Comprehensive System for the Development of Education Professionals (SIPROFE) program. Section III provides a brief overview of the information collected by the national teacher’s assessment (Ser Maestro) and discusses how the information included in this assessment could guide the design of in-service training courses. Section IV presents different policy scenarios that use data from student and teacher evaluations to better design in-service teacher training. A brief conclusion follows. 4    II. In-service teacher training in Ecuador In 2011, Ecuador enacted a new education law (Ley Orgánica de Educación Intercultural) that provided the education sector with a renewed vision of the country’s educational processes and actors. The law recognizes and gives high value to the work of teachers and introduces important elements to promote and develop the teaching career. For instance, Chapter 4 of the law specifies that teachers in Ecuador shall have free access to professional development opportunities, such as in-service training, to continuously update their knowledge and pedagogical practices. Chapter 10 establishes the need for the creation of a National University of Education, whose objective would be to support and develop the teaching profession and to better train new teachers. The law also establishes a more transparent teaching career path, whereby promotions are based on obtaining satisfactory results in teaching evaluations and on active participation in continuing education programs. In 2008, the MINEDUC began making important efforts to promote in-service teacher training through its Comprehensive System for the Development of Education Professionals (SIPROFE), overseen by the MINEDUC’s Undersecretariat of Professional Development. SIPROFE’s main objective is to improve the country’s education quality through the provision of training to teachers and school administrators. To deliver in-service teacher and administrator training, the SIPROFE offers a variety of courses, including remedial and developmental courses, orientation to new teachers, and school management. The duration of courses varies between 8 and 60 hours, although most courses have a duration of either 20 or 40 hours (MINEDUC 2016). Some courses are sequential (that is, are a prerequisite for others) to promote and incentivize the sustained participation of teachers. Training programs under the SIPROFE have traditionally been delivered through agreements with universities and pedagogical institutes (Institutos Superiores Pedagógicos, ISPED) that satisfy predetermined quality and service delivery requirements.4 Data from the MINEDUC suggest that during 2010–18, the SIPROFE program delivered in-service teacher training to about 113,000 teachers (representing about 75 percent of the total teacher population). According to the MINEDUC, between 2015 and 2018, the program benefited                                                              4  ISPEDs do not operate anymore. Also, universities that can participate as providers of training under the SIPROFE program have to satisfy the maximum level of accreditation (Level A). This requirement limits the territorial supply of training courses, as not many regional universities satisfy this requirement. 5    an average of over 10,000 teachers a year, with an average yearly investment of US$2.7 million.5 The cost of in-service teacher training is between US$28.5 and US$38.5 per hour of training (MINEDUC 2016). In-service training under the SIPROFE program is mandatory for teachers who underperform (that is, teachers with a score lower than 700 points in the test) on the teacher assessment Ser Maestro, a test implemented periodically by the National Institute of Evaluation (INEVAL). However, this requirement has not been systematically enforced. Participation in training is optional for all other teachers and school administrators in the public education system. Teachers’ participation in these training programs is among one of many criteria considered for career promotion. In terms of design features, most training programs take place on school premises and outside the teachers’ working time (that is, holidays, evenings, and weekends). However, in recent years, training providers have introduced some flexibility in the curricula by providing some content (and thus training hours) online. Also, teachers can take training courses during working hours if they obtain permission from school district authorities. To complete a course, participating teachers need to attend all the required course hours and pass a course evaluation. Also, to promote attendance and completion, the MINEDUC requires that teachers who sign up for a training program and fail the course (due to lack of attendance or if they fail the course exam) pay for the training cost out-of-pocket. Training opportunities are normally communicated to teachers by e- mail when new courses open. The content and curricula of the courses are generally designed between MINEDUC administrators and training providers based on the ministry’s identified priorities and the provider’s capacity to deliver courses (that is, courses are largely supply-driven). Neither the MINEDUC nor training providers conduct a systematic diagnostic to identify teacher learning needs. To date, the effectiveness of the SIPROFE program remains largely unassessed. The MINEDUC has not conducted a systematic evaluation of the program and its effects on teacher and student outcomes. There have been some attempts to collect qualitative information to assess the program’s effectiveness through interviews and focus groups with teachers and school                                                              5 Moreover, between 2014 and 2017, funds from the SIPROFE were used to support 3,924 teachers in obtaining a master’s degree in education. 6    principals. Nevertheless, most of the information available relies on small samples and cannot be assumed to be representative for statistical inference. Results from qualitative interviews conducted by Portaluppi Castro (2012) and Fabara- Garzon (2013) show, for instance, that teachers identify several problems with the training courses offered by the SIPROFE program. Focus group participants indicated that the training programs offered under SIPROFE lacked relevance and that courses were theoretical and provided little space for dialogue, analysis, and contextualization. Participants also indicated that there was little follow-up (for example, mentoring, coaching, feedback mechanisms) after training courses were completed. Some participants complained about the quality and pedagogical approaches of the trainers. On the positive side, participants agreed that the efforts made through the SIPROFE program are a clear signal of the interest of the state in teachers’ professional development. Participants also indicated that some teachers participated in the program to be promoted. Finally, participants suggested that the program could be improved by (a) having a better diagnostic of teachers’ training needs and offering progress to address them; (b) complementing training courses with more follow-up support, such as coaching and mentoring; (c) making it easier to obtain permission to attend training during working hours; and (d) making courses more interactive, flexible, and relevant to local contexts. III. Teacher assessments in Ecuador The Ser Maestro assessment, the government’s main instrument to quantify the level of knowledge of Ecuadoran teachers, is implemented by Ecuador’s National Institute of Evaluation and is an important pillar of the country’s teacher evaluation model. The assessment measures the level of knowledge of teachers in the field they teach, including specific topics they must master to promote an adequate learning process in their subjects. The assessment is applied to all teachers in active service (that is, it is census-based). The last assessment was conducted in 2016. The assessment includes 19 knowledge areas (Saberes) that include subjects spanning Ecuador’s three main educational cycles: Early Education (pre-K), Basic General Education (BGE) (grades 1 to 10), and Unified General Baccalaureate (UGB), equivalent to high-school (grades 1 to 3). Each subject includes several “curricular areas,” which assess the level of knowledge of the teacher in specific aspects of the national curriculum within the subject matter. 7    For each operational definition, in turn, the assessment includes a series of questions that make it possible to determine exactly which areas of knowledge teachers have problems with. For example, the information provided by the assessment identifies whether a teacher of mathematics (subject area) in BGE (education cycle) has knowledge gaps on set theory or geometry (curricular areas). The Ser Maestro assessment is applied to teachers according to the subject and grade they teach. Questions in the assessment relate to the main subject teachers teach in the classroom in the academic year of the assessment. Teachers are only assessed on one subject. Generalist teachers in primary schools (grades 2 to 7) are assessed on basic general education. The Ser Maestro database includes a variable that provides teachers’ total scores in the assessment (scores oscillate between zero and 1,000 points). Teachers with an overall score below 700 points are considered underperformers (that is, they fail the exam). For these teachers, in-service training is mandatory, although, as mentioned, this policy is not systematically enforced. As illustrated in figure 1, a significant share of teachers in BGE and UGB score below the 700 threshold in the assessment. Figure 1. Distribution of Teacher Performance in the Ser Maestro Assessment Source: Author using Ser Maestro 2016 data. Based on their scores, teachers can be further classified into four different groups: (a) low- performance teachers, which are those with scores lower than one standard deviation below the mean; (b) low-standard-performance teachers, which are those with scores between the 8    distribution mean and one standard deviation; (c) high-standard-performance teachers, which are those with scores between the mean and one standard deviation above the mean; and (d) high- performance teachers, which are those with scores above one standard deviation above the mean. Based on these definitions, 6.28 percent of all teachers in Ecuador are classified as being in the low-performance group, 44.45 percent are classified as being in the low-standard-performance group, 38.26 percent are classified as being in the high-standard-performance group, and 11.01 percent are classified as being in the high-performance group. Assessment results by knowledge topic indicate that a large majority of teachers (50 to 70 percent) underperform in mathematics, Basic General Education, and natural sciences (BGE level), and in mathematics, physics, and biology (UGB level). Overall teacher performance is generally better in subjects that pertain to social sciences and literature (table 1). Table 1. Knowledge Areas Included in the Ser Maestro Assessment, by Education Cycle Performance % Teachers Standard Standard Low High Total Evaluated Low High (%) (%) Sample in the (%) (%) Subject General Subjects (all levels) Management 6.67 39.28 33.78 20.27 2.235 2.38 Computers 4.84 38.99 42.44 13.73 3.655 3.89 Philosophical thinking 5.74 44.93 29.39 19.93 296 0.32 Artistic education 6.02 38.44 38.35 17.20 1.163 1.24 Special education 6.8 37.62 45.78 9.82 723 0.77 Physical education 6.48 33.72 44.25 15.55 4.102 4.37 Early education 4.44 45.61 41.66 8.30 14.281 15.20 Citizenship education 6.17 42.08 34.95 16.81 827 0.88 Basic General Education (BGE) Basic general education (grades 2º–7º) 4.71 47.33 39.02 8.94 41.993 44.69 Natural sciences 8.64 47.94 30.38 13.03 5.056 5.38 Social studies 6.80 34.62 42.69 15.88 4.659 4.96 Language and literature (grades 8º–10º) 5.15 27.38 41.89 25.58 2.991 3.18 Mathematics (grades 8º–10º) 13.97 55.52 23.88 6.64 3.208 3.41 Unified General Baccalaureate (UGB) Physics 21.67 51.71 23.38 3.23 526 0.56 History and social sciences 18.10 40.10 31.36 10.43 1.591 1.69 Language and literature 9.01 36.50 37.66 16.83 2.775 2.95 Biology 6.66 47.10 39.33 6.91 811 0.86 Mathematics 16.91 56.49 22.28 4.32 2.271 2.42 Chemistry 7.81 33.25 38.79 20.15 794 0.85 Total 6.28 44.45 38.26 11.01 93.957 100.00 Source: Author using Ser Maestro 2016 data. 9    A most interesting feature of the Ser Maestro assessment, as mentioned, is that teachers’ knowledge is tested against a series of “curricular areas” within each subject. Such a level of granularity of information enables the identification of very specific knowledge areas where teachers display cognitive gaps. Data from the Ser Maestro identify the share of correct and incorrect teacher responses in the assessments. The specificity of this information enables personalized teacher development programs to be designed based on actual needs and knowledge gaps (table 2). 10    Table 2. Curricular Areas (by Subject) with Low Levels of “Correct” Response Rates % of correct Curricular areas where teachers display the lowest performance (by SUBJECT AREA answers teaching subject) 1. Identification of the main elements of labor contracts Management 30.0 2. Knowledge of the main elements required for strategic planning Computers 25.0 1. Characteristics of “data mining” 1. Identification of aesthetic postulates of emblematic exponents Philosophical thinking 30.0 2. Identification of main characteristics of the Andean cosmovision 1. Identification of characteristics of technical elements of the arts Artistic education 35.0 2. Identification of characteristics of elements of the arts 3. Identification of genres or plastic expressions 1. Identification of characteristics of the components of a curriculum Special education 35.0 2. Description of weakened psychological processes 1. Recognition of adequate postures Physical education 40.0 2. Determination of parameters to assess physical capabilities 1. Identification of phonological awareness features Early education 30.0 2. Identification of rights and responsibilities of children Citizenship education 1. Identification of practices that promote a culture of peace 25.0 2. Recognition of the characteristics of citizenship 1. Set theory Basic general education 25.0 2. Identification of ordered pairs 3. Identification of metabolic processes 1. Identification of characteristics of autotrophic organisms Natural sciences BGE 25.0 2. Identification of properties of chemical compounds 1. Relationship between humanism and cultural changes Social studies BGE 20.0 2. Recognition of laws or actors that regulate markets Language and literature BGE 40.0 1. Recognition of authors’ shortcomings after reading a text 1. Identification of the main functions of nucleic acids Biology UGB 25.0 2. Recognition of hormone functions 3. Identification of different types of aquatic biomes 1. Geometry Mathematics BGE 25.0 2. Set theory 1. Notions of thermodynamics Physics UGB 15.0 2. Calculation of physical magnitudes 3. Characterizations of energy 1. Recognition of authors of contemporary philosophy History and social sciences UGB 20.0 2. Identification of social characteristics from reading texts Language and literature UGB 30.0 1. Recognition of logical connectors from the reading of a text 1. Set theory Mathematics UGB 20.0 2. Geometry 3. Notions of cardinality Chemistry UGB 20.0 1. Identification of types of photovoltaic cells Source: Author using Ser Maestro 2016 data. Note: BGE = Basic General Education; UGB = Unified General Baccalaureate. 11    IV. Use of assessments to design in-service training programs Identification of More Pertinent Training Content Based on available funding and preestablished yearly training targets, administrators of the SIPROFE program reach out to universities (or other training providers) to have them propose training content. As mentioned, the training curricula for these courses are not based on a systematic assessment of needs, but rather on the university’s expertise, the supply of courses, the availability of trainers, and general delivery capacity (that is, courses are supply-driven). According to data provided by the National Directorate of Continuous Education (a division under the MINEDUC’s Undersecretariat of Professional Development), since its introduction in 2008, the SIPROFE program has offered, on average, 15 in-service training courses per year. These training programs are categorized into two main groups: (a) complementary programs, which are largely focused on building the capacity of teachers to revise and update the curricula; and (b) remedial programs, which aim to address teachers’ knowledge gaps. Table 3 lists the remedial courses most recurrently offered by the SIPROFE program. Results indicate that courses offered are highly focused on pedagogy and general education, with a limited focus on improving cognitive skills. Table 3. Remedial Courses Offered under the SIPROFE Program, 2012–16 Course Name Training Hours Pedagogy for Natural Sciences (BGE) 40 Pedagogy for Social Sciences (BGE) 40 Pedagogy for Mathematics (BGE) 40 Pedagogy for promoting critical thinking in the classroom 40 Pre-K education 50 Strategies to evaluate student learning 60 Inclusive education 10 Introduction of information technology and communications 50 Critical reading 60 General pedagogy 60 Prevention of student risks (sexual abuse, violence, etc.) 30 Source: MINEDUC 2016. Program administrators and participating training providers could use the information of the Ser Maestro assessments to make courses more demand-driven and responsive to teachers’ observed cognitive skills gaps. As mentioned, the assessment identifies both the subjects and curricular areas where teachers display cognitive deficits. Administrators, for instance, could develop reports using the Ser Maestro results that show the observed performance of teachers by 12    subject and curricular areas at the district level. For example, figure 2 displays (for a selected school district in Ecuador) the average performance of teachers in the district in curricular areas related to the teaching subject literature and language for UGB. A similar exercise could be replicated in other subject areas, such as mathematics for UGB, physics for UGB, and so forth. The figure plots the average percentage of correct answers in a given curricular area for all teachers in the district who currently teach this subject. The red, yellow, and green bars display the subjects on which teachers, on average, correctly answered less than 30 percent of the subject-related questions, between 30 and 35 percent of the subject-related questions, and between 40 to 60 percent of the subject-related questions, respectively. This information could be used by program managers and universities to develop a more pertinent training curriculum that responds to actual knowledge gaps of teachers in a given school district. A similar exercise using the results of the Ser Maestro assessments could be conducted at the school level. The MINEDUC could provide school directors with individualized teacher reports that contain similar information for all schoolteachers. Based on such information, teachers and directors could develop individualized training plans. Since at the district level the supply of services (in partnership with local universities and pedagogical institutes) would also consider the training needs of teachers in the district, teachers and directors could choose from the available courses offered at the district level the best ones that suit the teacher’s individual training needs. Of course, adapting the supply of services to the actual needs of teachers would take time. Also, for the process to work, the MINEDUC probably would need to provide technical assistance to school directors to be able to interpret teacher reports and to develop individualized training plans. The MINEDUC could start by testing this model in some school districts (and learn from the experience) before scaling up the model nationally. 13    Figure 2. Average Percentage of Teacher’s Correct Answers by Curricular Area in a Selected School District in Ecuador for Teachers of Literature and Language, UGB Identify language level or levels. From a text, identify synonyms or antonyms according to the context. From a fragment or list of characteristics, identify non‐literary texts From a text or list of sentences, recognize the correct use of capital letters Recognize language functions. MODERATE From a list or table, recognize words that are formed with affixes. COGNITIVE GAPS Identify the morphology of the verb. From a list of text characteristics, identify the literary subgenres. From a list of words or in tables, recognize the meaning of the affixed From a list of sentences, identify the correct use of punctuation marks From a list of words or sentences, recognize foreign words in the text From a text, recognize implicit ideas. From a text, recognize explicit ideas. From a list of Ecuadoran literary works, relate them to their Latin… From a fragment of a literary work, identify the literary movement / epoch… From a fragment of a literary work, identify the literary gender where it… From a sentence, recognize its variable or invariable parts. From a sentence, identify its syntactic elements. Recognize the definitions of syntactic elements in a sentence. From a list of verses (independent) or a stanza, identify the literary work From a text, table, or list of sentences; recognize the author's vices From a stanza, identify the kind of rhyme or rhythm of the verse. From a text, list, or table; recognizes the characteristics of literary schools From a list, identify the use of homonymous words. From a text, table, or list of sentences; recognize the author's vices LARGE  From a list of literary works, identify its author. COGNITIVE GAPS From a list / table of authors, identify the literary genre to which it belongs From a case, list, or table; identify the main communication elements From a text, list, or table; recognize famous authors or works Recognize the subgenre to which the fragment of a literary work belongs. From a list of verses (independent) or a verse, recognize the metrics. From a text or list, recognize the logical connectors. 0.0 20.0 40.0 60.0 80.0 100.0 % of correct answers Source: Author using Ser Maestro 2016 data. Note: The red, yellow, and green bars display the subjects on which teachers, on average, correctly answered less than 30 percent of the subject-related questions, between 30 and 35 percent of the subject-related questions, and between 40 to 60 percent of the subject-related questions, respectively. Can the results of the Ser Maestro assessments be trusted? Using the Ser Maestro assessments to guide the content of in-service teacher training could be problematic if the knowledge areas measured by the exam are not necessarily related to the curricula imparted in the classroom. If this were the case, investing in training programs to solve the cognitive skills gaps of teachers may not necessarily translate into better student learning. Ecuador uses the Ser Bachiller assessment, a census-based instrument, to assess student learning. The Ser Bachiller assessment is a compulsory exam students take to graduate from the 14    UGB. The assessment is used by most higher education institutions as the entry exam to access higher education.6 Figure 3 plots school-level results of the Ser Bachiller (y-axis) and the Ser Maestro (x- axis) assessments in the areas of language (panel A) and mathematics (panel B). The panels indicate that schools with better-performing teachers in language and mathematics tend to also have better-performing students in language and mathematics. Moreover, estimates from a simple Ordinary Least Squares regression model indicate that schools having teachers with 10 percent higher average scores in language and mathematics in the Ser Maestro assessments are associated with schools with students with 15 percent higher scores in the language and mathematics Ser Bachiller assessment (table 4). While these results do not imply causality, they indicate a positive correlation between teacher and student performance in language and mathematics in both assessments. These results provide prima facie evidence that improving teachers’ cognitive skills in language and mathematics (proxied by the Ser Maestro assessment) is expected to improve the cognitive skills in language and mathematics of the students in the schools they serve (proxied by the Ser Bachiller assessment). Figure 3. Correlation between Ser Maestro and Ser Bachiller Assessments at the School Level – Language and Mathematics (UGB) Panel A. Language Panel B. Mathematics     Source: Author using Ser Maestro 2016 and Ser Bachiller 2017/2018 datasets.                                                              6 The National Institute of Evaluation also uses the Ser Estudiante assessment, which is conducted on a sample of BGU students. Not being census-based is an important limitation of this assessment, as it does not enable school-level student performance to be assessed. 15    Table 4. Correlations between Scores on Ser Bachiller and Ser Maestro Assessments at the School Level – Language and Mathematics (UGB) Language Mathematics Log score Log score Log score Log score Ser Bachiller Ser Bachiller Ser Bachiller Ser Bachiller 0.186*** 0.176*** 0.156*** 0.152*** Log of score on Ser Maestro [0.017] [0.018] [0.014] [0.014] 0.000 -0.000** School’s total students [0.000] [0.000] 0.007*** School’s total mathematics teachers [0.002] 0.003* School’s total language teachers [0.002] 0.824*** 0.877*** 0.995*** 1.007*** Constant [0.113] [0.116] [0.089] [0.089] Observations 1.271 1.271 1.164 1.164 Adjusted R-squared 0.081 0.086 0.097 0.107 Source: Author using Ser Maestro 2016 and Ser Bachiller 2017/2018 datasets. Note: *** = P ≤ 0.001. Standard errors in brackets. Improving Program Targeting In-service training opportunities in Ecuador under the SIPROFE program are made available to teachers on a first-come, first-served basis. Once these training opportunities become available, program administrators contact teachers by e-mail or through local campaigns organized by the MINEDUC in coordination with school district authorities. Teachers who want to participate in the available programs sign up until all slots are filled. The combined information from the Ser Bachiller and the Ser Maestro assessments constitutes a powerful tool to target interventions in schools (and school districts) with the largest needs. Figure 4 plots school-level overall results of the Ser Maestro (y-axis) and Ser Bachiller (x- axis) assessments. The dotted lines in the figure represent test scores that are one standard deviation above and below the average scores. Each dot in the figure represents a school. The color of the dots represents the poverty rates of the parishes (the smallest administrative territorial disaggregation) where schools are located. The blue dots represent the schools located in areas with the highest poverty rates, and the black dots represent the schools located in areas with the lowest poverty rates.7 Figure 4 illustrates that schools with the worst-performing teachers also have the worst- performing students (bottom left panel of the chart). While this relationship does not necessarily imply a causality, it contributes to identifying the worst-performing institutions, presumably those                                                               Poverty is measured by the (census-based) Unsatisfied Basic Needs Index produced in 2010 by Ecuador’s National 7 Office of Census and Statistics. 16    where teachers and students can benefit the most from available (and limited) government interventions, such as in-service training programs. Figure 4. Student Performance compared to Teacher Performance on Standardized Tests, by School Source: Author using Ser Maestro 2016 and Ser Bachiller 2017/2018 datasets. Note: Poverty rates are calculated using the 2010 Unsatisfied Basic Needs Index produced by Ecuador’s National Office of Census and Statistics. Thus, collecting information from the worst-performing schools and school districts in the Ser Bachiller and the Ser Maestro assessments could be used to target the school districts with the greatest needs. As an illustration, table 5 provides a list of the school districts that display the worst performance in both assessments. The MINEDUC could consider targeting interventions under the SIPROFE program to these districts. Other targeting criteria and additional filters could be added in the process (such as proximity to training providers, local poverty rates, and school size). In any case, the main idea behind these results is that Ecuadoran policy makers could use their available assessment data to target in-service teacher training interventions to the student and teacher populations with the greatest needs. 17    Table 5. Education Districts with Average Scores in the Ser Maestro and Ser Bachiller Assessments that are Below One Standard Deviation from the Mean Number District % Ser Bachiller Ser Maestro of Province Region ID Schools Average Score Average Score Schools 21D02 2 2.33 Sucumbíos Amazonía 6.70 484.38 14D05 8 9.30 Morona Santiago Amazonía 6.23 497.28 16D02 4 4.65 Pastaza Amazonía 6.53 497.31 13D01 1 1.16 Manabí Costa 6.75 499.50 21D04 5 5.81 Sucumbíos Amazonía 6.65 500.96 Zamora 1 1.16 6.52 509.60 19D01 Chinchipe Amazonía 16D01 9 10.47 Pastaza Amazonía 6.53 512.91 15D01 7 8.14 Napo Amazonía 6.74 520.99 06D01 1 1.16 Chimborazo Sierra 6.57 521.00 14D01 1 1.16 Morona Santiago Amazonía 6.35 527.70 09D20 1 1.16 Guayas Costa 6.94 528.00 08D02 4 4.65 Esmeraldas Costa 6.60 528.67 08D04 3 3.49 Esmeraldas Costa 6.75 530.48 14D02 4 4.65 Morona Santiago Amazonía 6.51 532.68 08D03 2 2.33 Esmeraldas Costa 6.60 536.65 14D03 4 4.65 Morona Santiago Amazonía 6.60 537.36 08D01 7 8.14 Esmeraldas Costa 6.76 541.35 22D01 2 2.33 Orellana Amazonía 6.80 543.06 09D02 1 1.16 Guayas Costa 6.83 545.67 22D02 7 8.14 Orellana Amazonía 6.76 547.01 21D03 1 1.16 Sucumbíos Amazonía 6.56 552.50 13D07 1 1.16 Manabí Costa 6.60 552.67 08D06 2 2.33 Esmeraldas Costa 6.76 555.42 05D04 1 1.16 Cotopaxi Sierra 6.80 555.60 08D05 1 1.16 Esmeraldas Costa 6.93 556.25 09D11 1 1.16 Guayas Costa 6.61 561.60 12D06 1 1.16 Los Ríos Costa 6.76 562.17 21D01 1 1.16 Sucumbíos Amazonía 6.45 564.67 12D01 1 1.16 Los Ríos Costa 6.95 568.00 Zamora 1 1.16 6.84 572.80 19D02 Chinchipe Amazonía 09D06 1 1.16 Guayas Costa 6.79 575.94 Total 86 99.97 Source: Author using Ser Maestro 2016 and Ser Bachiller 2017/2018 datasets. 18    V. Conclusion Teacher performance constitutes a critical determinant of student learning. As such, continuous contact with poor-performing teachers over several years could result in significant learning gaps. In Ecuador, evidence indicates that children in poor localities face significant risks of being taught by teachers who do not master the curricula of the class they teach. Results from teacher assessments indicate that a large majority of teachers underperform in mathematics, Basic General Education, and the natural sciences (BGE level), and in mathematics, physics, and biology (UGB level). Given the fact that many teachers in Ecuador enter the education system with important cognitive gaps, in-service teacher training could be an important tool to strengthen teacher performance. Ecuador in recent years has made important investments in improving teacher development and quality. Since 2008, the MINEDUC has been promoting in-service teacher training through its Comprehensive System for the Development of Education Professionals (SIPROFE) program. Since 2010, the SIPROFE program has delivered in-service teacher training to about 75 percent of Ecuador’s total teacher population. Nevertheless, training programs delivered under the SIPROFE program have been supply-driven, focused on improving pedagogy, and not necessarily designed to respond to the observed teacher’s cognitive skills gaps. Moreover, their effectiveness has been largely unassessed, and program targeting has not prioritized school districts where teacher development programs are needed the most. Ecuador currently uses teacher and student assessments that can identify in detail the cognitive deficiencies of students and teachers. These data, however, have not been used to guide the design of in-service teacher training programs. Ecuador’s teacher assessment (Ser Maestro) evaluates teachers on the subjects they teach in the classroom and identifies curricular areas within a teaching subject where teachers display cognitive gaps. The specificity of this information enables personalized development programs to be designed based on teachers’ actual needs. Program administrators and participating training providers could use this information to develop a more pertinent training curriculum that addresses the knowledge gaps of teachers across different school districts. Furthermore, combining available information on student and teacher assessments would allow policy makers in Ecuador to identify the worst-performing schools. Given that resources to provide in-service training opportunities are limited, the government could use this information to 19    target interventions to schools with the greatest needs. SIPROFE administrators could use these assessment results plus other information; such as proximity to training providers, local poverty rates, and school size; in their yearly programming, which targets selected territories nationwide. The results of the Ser Maestro assessments could also serve to develop diagnostics for schools and teachers. The MINEDUC could provide school directors, teachers, and parents with individualized reports that contain information about observed cognitive gaps. Based on such information, teachers and school principals could develop individualized training plans for schools and teachers. Using available information from student and teacher assessments, thus, could become a key input for policy makers in countries that use such information, such as Ecuador, to develop more pertinent and effective teacher development programs. Since more countries are periodically conducting student and teacher assessments, the recommendations presented here for Ecuador could well be taken into consideration regionally and globally. Of course, the process of adapting the supply of training services to the actual needs of teachers (and to local needs) takes time. However, Ecuador, as well as many other countries in the region, could consider using available evaluations to design and implement in-service training programs of higher relevance and impact while allowing the prioritization of investments in teachers and students with the greatest needs. 20    Bibliography Angel-Urdinola, D., and P. Vera-Jibaja. 2018. Achievements and Challenges of Ecuador’s Education Sector in the 21st Century. World Bank. Washington, DC. https://ssrn.com/abstract=3180271 or http://dx.doi.org/10.2139/ssrn.3180271. Avalos, B. 2007. “Teacher Continuous Professional Development: What the international and Latin American experiences tell us.” Pensamiento Educativo 41 (2): 77–99. Bruns, B., and J. Luque. 2014. Great Teachers: How to Raise Student Learning in Latin America and the Caribbean. Washington, DC: World Bank. Chetty, R., J. N. Friedman, and J. E. Rockoff. 2014. “Measuring the Impacts of Teachers II: Teacher Value-Added and Student Outcomes in Adulthood. American Economic Review 104 (9): 2633–2679. Fabara-Garzon, E. 2013. “Estado del Arte de la Evaluación Docente en Ecuador.” Cuadernos del Contrato Social por la Educación. Cuaderno No. 8. Quito, Ecuador. Figlio, D., and L. Kenny. 2007. “Individual teacher incentives and student performance.” Journal of Public Economics 91 (5–6): 901–914. Fryer, Jr., R. G. 2017. “The Production of Human Capital in Developed Countries: Evidence from 196 Randomized Field Experiments.” In Handbook of Economic Field Experiments, Vol. 2. Amsterdam: North-Holland, pp. 95–322. Gersten, R., M. J. Taylor, T. D. Keys, E. Rolfhus, and R. Newman-Gonchar. 2014. Summary of research on the effectiveness of math professional development approaches. Department of Education and Regional Educational Laboratory, Southeast at Florida State University, Washington, DC, and Tallahassee, Florida. http://files.eric.ed.gov/fulltext/ED544681.pdf. Hanushek, E., and L. Woessmann. 2006. “Does educational tracking affect performance and inequality? Differences-in-differences evidence across countries.” Economic Journal 116 (510): C63–C76. Hanushek, E., M. Piopiunik, and S. Wiederhold. Forthcoming. “The Value of Smarter Teachers: International Evidence on Teacher Cognitive Skills and Student Performance.” Journal of Human Development. Hanushek, E., P. Peterson, and L. Woessmann. 2012. Achievement Growth: International and U.S. State Trends in Student Performance. Harvard’s Program on Education Policy and Governance / Education Next. Harvard Kennedy School, Cambridge, MA. Hanushek, E. A., and S. G. Rivkin. 2010. Using value-added measures of teacher quality. (CALDER Brief 32 No. 9). National Center for Analysis of Longitudinal Data in Education Research, Washington, DC. http://files.eric.ed.gov/fulltext/ED509683.pdf. Harvey, J. 2016. “Understanding Ecuador’s Education Reform: A Case Study of 2006–2015.” Concordia University, Portland, Oregon. Ingersoll, R. 2004. “Why Do High-Poverty Schools Have Difficulty Staffing Their Classrooms with Qualified Teachers.” University of Pennsylvania, Philadelphia. Levy, S., and N. Schady. 2013. “Latin America’s Social Policy Challenge: Education, Social Insurance, Redistribution.” Journal of Economic Perspectives 27 (2): 193–218. 21    MINEDUC (Ministry of Education). 2016. Proyecto Sistema Integral De Desarrollo Profesional Educativo – SIPROFE. Reporte de Gestión 2010–2015. Ministry of Education, Quito. Moura-Castro, C., and G. Ioschpe. 2007. La remuneración de los maestros en América Latina: ¿Es baja? ¿Afecta la calidad de la enseñanza? Programa de Promoción de la Reforma Educativa en América Latina y el Caribe Partnership for Educational Revitalization in the Americas (PREAL No. 37). Canadian Institute for Non-destructive Evaluation (CINDE), Santiago, Chile, and Inter-American Dialogue, Washington, DC. Mulford, B., D. Kendall, J. Ewington, B. Edmunds, L. Kendall, and H. Silins. 2008. “Successful principalship of high‐performance schools in high‐poverty communities.” Journal of Educational Administration 46 (4): 461–480. Muralidharan, K., and V. Sundararaman. 2011. “Teacher performance pay: experimental evidence from India.” Journal of Political Economy 119 (1): 39–77. OECD (Organisation for Economic Co-operation and Development). 2009. “The Professional Development of Teachers.” In Creating Effective Teaching and Learning Environments. Paris: Organisation for Economic Co-operation and Development. ISBN 978-92-64- 05605-3. Popova, A., D. Evans, M. E. Breeding, and V. Arancibia. 2018. “Teacher Professional Development around the World: The Gap between Evidence and Practice.” Policy Research Working Paper No. 8572, World Bank, Washington, DC. Portaluppi Castro, C. 2012. El Sistema Integral de Desarrollo Profesional Educativo (SíPROFE): Una mirada desde los involucrados en el sistema. Trabajo Final para la obtención del Título de Magíster en Educación Superior, con mención en Temas Contemporáneos en Docencia Superior. Universidad Casa Grande, Guayaquil, Ecuador. Rivkin, S. G., E. Hanushek, and J. F. Kain. 2005. “Teachers, Schools, and Academic Achievement.” Econometrica 73 (2): 417–458. Rockoff, J. E. 2004. “The Impact of Individual Teachers on Student Achievement: Evidence from Panel Data.” American Economic Review 94 (2): 247–52. Walter, C., and J. Briggs. 2012. What professional development makes the most difference to teachers. Oxford: University of Oxford Department of Education. http://clie.org.uk/wpcontent/uploads/2011/10/Walter_Briggs_2012.pdf. Woessmann, L. 2011. “Cross-country evidence on teacher performance pay.” Economics of Education Review 30 (3): 404–418. 22