Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste: The New Curriculum and the Professional Learning and Mentoring Program THE WORLD BANK OFFICE DILI Avenida dos Direitos Humanos Dili, Timor-Leste Tel +670 3312367 October 2019 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste: The New Curriculum and the Professional Learning and Mentoring Program is a product of staff of the World Bank. The findings, interpretation and conclusion expressed herein do not necessarily reflect the views of the Board of Executive Directors of the World Bank or the government they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denomination and other information shown on any map in this work do not imply any judgment on the part of the World Bank concerning the legal status of any territory or the endorsement of acceptance of such boundaries. Photo credit: World Bank. Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste The New Curriculum and the Professional Learning and Mentoring Program Acknowledgements This document is based on the report Lessons Learned from an Early Assessment (2017) of Two Innovations in Basic Education in Timor-Leste1, produced by a World Bank Team in collaboration with the Timor-Leste’s Ministry of Education. This activity was funded by the Australian Government/DFAT through a trust fund executed by the World Bank. The core team of the study included Tazeen Fasih (Original Task Team Leader, the World Bank), Stephen L. Walter, Karla J. Smith and Pedro Ximenes (Consultants, World Bank), Adelaide Camões (World Bank) with valuable feedback from Hussein Abdul-Hamid and Mark F. LaPrairie (Peer-reviewers, World Bank); and the enumerator team, which consisted of various national and municipal staff of the Ministry of Education, Timor- Leste, including school inspectors, and civil servants from INFORDEPE (National Teacher Training Institute), the National Directorate of Basic Education, and relevant municipal offices. Cornélio de Carvalho and Rideca Duarte (World Bank Team Assistants) provided administrative support. This document, titled “Using EGRA for an Early Assessment of Two Innovations in Basic Education in Timor- Leste: The New Curriculum and the Professional Learning and Mentoring Program”, presents a streamlined version of the original report for the benefit of a wider range of non-academic readers, particularly government officials and school leaders. This document benefitted from technical inputs from Javier Luque and Susiana Iskandar (World Bank). Finally, the team is very thankful to Her Excellency Dulce de Jesus Soares, in both of her positions as the former Vice Minister of Education and the current Minister of Education, Youth and Sport, as well as senior officials of the Ministry of Education for their participation and guidance, including the General Director of Pre-school and Basic Education, Mr. Cidálio Leite, the General Inspector of the Ministry of Education, Youth and Sport, Mr. Evaristo Maria de Jesus, the National Director for Basic Education, Mrs. Engrácia Cabral, municipal directors and school inspectors from the municipalities of Dili, Manatuto, Liquica and Aileu, as well as technical inputs from Deborah Katzman, advisor of the Ministry of Education, Youth and Sport. Finally the team would like to thank the overall guidance received in the elaboration of the study and final report by the World Bank regional senior management. 1 The analysis ‘Lessons Learned from an Early Assessment (2017) of Two Innovations in Basic Education in Timor-Leste’ written by Stephen L. Walter is available upon request. ii Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste List of Abbreviations CBA Curriculum-based Assessment CBO Classroom-based Observation DFAT Department of Foreign Affairs and Trade (of the Australian government) EGMA Early Grade Mathematics Assessment EGRA Early Grade Reading Assessment EMBLI Edukasaun Multilinge Bazeia ba Lian Inan (Mother Tongue-based Multilingual Education) EMIS Education Management Information System INFORDEPE Instituto Nacional de Formação de Docentes e Profissionais da Educação (Institute for Training of Teachers and Education Professionals) MoEYS Ministry of Education, Youth and Sport (of Timor-Leste) NC Index New Curriculum Index (a rubric evaluating level of implementation) PLMP Professional Leadership and Mentoring Program Wpm Words per minute The New Curriculum and the Professional Learning and Mentoring Program iii Table of Contents Acknowledgements................................................................................................................................................................... ii List of Abbreviations..................................................................................................................................................................iii Executive Summary................................................................................................................................................................. vii 1. Background and Context....................................................................................................................................................1 2. Two Innovations...................................................................................................................................................................3 First Innovation: The New Curriculum........................................................................................................................ 3 Second Innovation: Professional Leadership and Mentoring Program (PLMP)................................................ 4 3. The Research Questions.....................................................................................................................................................5 4. Methodology..........................................................................................................................................................................6 Basic Design....................................................................................................................................................................6 Measurement..................................................................................................................................................................6 Primary Tools..................................................................................................................................................................6 Secondary Tools.............................................................................................................................................................7 Sample..............................................................................................................................................................................7 Procedure.........................................................................................................................................................................9 Certain Caveats.............................................................................................................................................................10 5. Report of Findings..............................................................................................................................................................11 Part 1 - Impact of the New Curriculum on Learning Outcomes................................................................................11 Evidence on Implementation of the New Curriculum............................................................................................11 Section A. Observable Impact of the New Curriculum on EGRA Results..........................................................12 Section C: Summary of Findings for Part 1 on the New Curriculum..................................................................16 Part 2: The Professional Learning and Mentoring Program (PLMP)........................................................................17 Theoretical Framework...............................................................................................................................................18 Section A – Impact of the PLMP on Learning Outcomes....................................................................................18 Section B – Differences between PLMP and Non PLMP School Practices......................................................27 Section C – Summary of Findings on the Implementation of PLMP.................................................................31 6. Discussion of Findings........................................................................................................................................................32 Specific Findings with Respect to the Three Research Questions.....................................................................32 A Synthesis of Major Findings...................................................................................................................................33 7. Recommendations..............................................................................................................................................................34 Final Comments about the Two Innovations Motivating the Evaluation...........................................................35 iv Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste 8. References and Resources................................................................................................................................................36 Part 1 – The New Curriculum............................................................................................................................................36 Part 2 – Professional Leadership and Mentoring Program.......................................................................................36 Annexes......................................................................................................................................................................................41 Annex 1. Instruments Employed in the Assessment...................................................................................................43 The New Curriculum and the Professional Learning and Mentoring Program v List of Tables Table 1. Detailed description of the sample population from PLMP schools (treatment group) which participated in the 2017 evaluation................................................................................................................. 8 Table 2. Detailed description of the sample population of non-PLMP schools (control group) which participated in the 2017 evaluation................................................................................................................. 8 Table 3. General statistics on the comparison of PLMP and non PLMP schools (Class 1 only) in the study.9 Table 4. Indicators on the use of the New Curriculum.............................................................................................12 Table 5. Comparison of EGRA test results for Class 1 children between 2009 and 2017................................13 Table 6. Comparison of results for 6 schools participating in the EGRAs of both 2009 and 2017 (Class 1).... 14 Table 7. Impact of the New Curriculum when implemented or not implemented in Class 1............................15 Table 8. Zero scores on the EGRA tasks in 2009 and 2017....................................................................................16 Table 9. Comparison of PLMP and non PLMP schools on the CBA and EGRAs by Class................................19 Table 10. Internal results by Section from the CBA for Class 1................................................................................20 Table 11. Internal results by Section from the CBA for Class 2................................................................................20 Table 12. Comparison of Class 1 students from PLMP and non PLMP schools on the EGRA measures (data from 2017). All numbers reflect raw scores other than those for Reading Comprehension which are expressed as a percentage......................................................................................................................21 Table 13. Comparison of Class 2 students from PLMP and non PLMP schools on six EGRA measures (data from 2017). All numbers reflect raw scores other than those for Reading Comprehension which are expressed as a percentage......................................................................................................................22 Table 14. Impact of the PLMP innovation in terms of the likelihood that the percent of zero scores is reduced by the innovation. Data from Class 1............................................................................................23 Table 15. Impact of the PLMP innovation in terms of the likelihood that the percent of zero scores is reduced by the innovation. Data from Class 2............................................................................................24 Table 16. Detail on the impact of PLMP on excelling in math in Classes 1 and 2 (CBA).....................................25 Table 17. A comparison of various classroom variables (Class 1 only) between PLMP and non PLMP schools...............................................................................................................................................................26 Table 18. School-level comparisons of performance among PLMP and non PLMP schools on 4 points of assessment in Classes 1 and 2.....................................................................................................................27 Table 19. Comparing classroom behaviors of PLMP and non PLMP teachers (Class 1 only)...........................28 Table 20. Comparing PLMP and non PLMP schools on use of the NC in the classroom...................................29 Table 21. Comparing PLMP and non PLMP schools on a measure of teacher skill and quality........................30 vi Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste Executive Summary Context With the gaining of independence in 2002, Timor-Leste for the first time shouldered the responsibility of self-rule and self-determination. In the education sector this meant building an education system responsive to the needs, values, and vision of an independent populace. The task was massive: building schools, finding and training teachers, making decisions about curricula, setting up supervisory systems, developing technical expertise, etc. Adding further complexity was the fact that at least 20 different languages were spoken among a population of close to one million people. In 2009, the Ministry of Education requested that there be a national student assessment as a first appraisal of the progress made in building a national education system. The World Bank arranged the implementation of this national assessment using the Early Grade Reading Assessment (EGRA). The results of this student assessment were below expectations, indicating the need for further strategies to address existing challenges. Two Innovations This document focuses on two innovations that have been introduced in the education system since then: The New Curriculum and the Professional Learning and Mentoring Program (PLMP). The first and largest of these was the development and implementation of a national primary level curriculum (the New Curriculum) to provide guidance and consistency in early basic education. After several years of development, the New Curriculum’s implementation started in the middle of 2015 for Class2 1 and 2, with rollout to subsequent grades on a yearly basis thereafter. In many respects, the New Curriculum is a massive teacher- support system designed to improve teacher’s performance in the classroom by providing guidance on what is to be taught, detailed lesson plans, classroom activity and pedagogy, daily and weekly schedules, detailed explanations of content to be taught, etc. A central feature of the New Curriculum is a commitment to Tetun3 as the initial language of instruction in primary education. Portuguese is gradually added to the instructional mix beginning with oral Portuguese in Class 1 and moving on to basic literacy in subsequent classes. The second innovation, launched in 2016, is known as the Professional Learning and Mentoring Program (PLMP). The PLMP focuses on institutional development at the level of school administration, teachers, and information flow between schools and central offices. It uses a combination of national and international mentors to provide informal guidance to a school’s instructional and administrative staff on best practices in basic education. The PLMP focused on Class 1 and Class 2 during its first year of implementation, although its influence extends to other classes as well. The innovation also includes facilitation of teacher working groups and the use of modern technology—via tablets and the internet—to support monitoring and reporting requirements between schools and central offices. 2 In Timor-Leste, the term ‘Class’ is used rather than the term ‘Grade’ to refer to the levels of basic education. 3 Tetun (aka Tetun Dili) is a creolized mix of Portuguese and Tetun Terik, one of the local languages of the country. The grammar and syntax of Tetun come from Tetun Terik while much of the lexicon, other than common and traditional vocabulary, is borrowed from Portuguese. Tetun is one of two official lan- guages in Timor-Leste (Portuguese is the other) and is the most widely spoken language in the country, though no reliable data are available on the number who speak the language with some degree of proficiently. The New Curriculum and the Professional Learning and Mentoring Program vii The 2017 Evaluation The Ministry of Education, Youth and Sport (MoEYS)4 was keen to get an understanding of how the two innovations were being implemented and to get any early results that might help hone and refine the interventions. The Australian Department of Foreign Aid and Trade (DFAT), as the main funding entity for the PLMP intervention, was also interested in identifying initial impacts of the program. As a result, it was agreed that a preliminary evaluation of the two innovations would be conducted upon the completion of the first full year of implementation. The World Bank led the evaluation on a technical level, with the financial contributions of a DFAT-funded multi- donor trust fund. This evaluation was carried out between January and May of 2017 and consisted of two major components: (a) a learning assessment, and (b) extended classroom observations of instructional and learning activities. The learning assessment itself included two parts. One of these was the administration of the same EGRA instruments used in the 2009 national assessment. The rationale was that this would provide some indication of ‘before-and-after’ comparison between 2009 and 2017. However, this comparison has three caveats: 1) Timor-Leste, and its education sector, experienced other policy changes between 2009 and 2017 besides the introduction of the New Curriculum, 2) the scope of the assessment in 2017 was limited to four municípios (municipalities), unlike the national coverage in 2009, and 3) concerns that EGRA components may not be comparable over a long period of time5. Despite these limitations, it was agreed that information from the EGRA test would provide the government a tentative understanding of the potential impact of the two innovations. (It was generally agreed that one year of implementation of ANY intervention would not provide enough time to do any kind of fair impact appraisal.) The second part of the learning assessment was a curriculum-based assessment (CBA) designed to test the extent to which children in Classes 1 and 2 had achieved the learning standards specified in the New Curriculum. This assessment included testing the mastery of material in reading, math, Tetun language, and Portuguese language. Since the CBA was Class specific, the content of the assessment was adjusted to the learning outcomes pertaining to each class. The second component of the evaluation was the realization of detailed classroom-based observations (CBO). For this component, a team of observers (all from the Ministry of Education) visited and monitored a classroom for a full instructional day in each selected school. Through a specially designed classroom observation tool, the team created a map of time utilization, recorded all pedagogical activity, tracked patterns of language utilization by the teacher(s) and made observations on the type and nature of children’s participation in classroom activity. The CBO allowed for the construction of the New Curriculum Index (NC Index) and Teacher Quality Index. This evaluation was implemented in a total of 128 schools, with a total of 6,326 students. Schools were selected from four municipalities. Of the total participating schools, 70 schools were PLMP schools, and 58 schools were non-PLMP schools. PLMP schools were not selected randomly. Those schools were selected from the initial 125 schools that had implemented the model first, with the rationale that these would be those demonstrating likelihood of successful implementation of the PLMP intervention since they had been implementing for a longer period, although this did not always turn out to be the case. The non-PLMP schools were used as controls in the assessment of the PLMP intervention, and they were selected by the evaluation team to provide a valid comparison group to the PLMP schools. The limited number of municipalities and the non-random selection of schools limit the generalization of the results of this Early Assessment to the country as a whole but provide 4 The Ministry of Education was restructured in July 2018 under the 8th Constitutional Government, establishing that education in Timor-Leste is under the responsibility of two ministries: Pre-primary to secondary is under the Ministry of Education, Youth and Sport (MoEYS); and higher education is under the Ministry of Higher Education, Science and Culture (MHESC). 5 For example, the use of some words may have changed between 2009 and 2017. viii Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste initial insights on the impact of the interventions, which was the key purpose of the study. Additionally, the evaluation allowed the identification of the extent of implementation of the New Curriculum, and initial changes in pedagogical practice linked to the PLMP. Findings The following bulleted lists summarize what was learned from the assessment. The details are provided in the body of the report. The New Curriculum Although several tests were administered as part of this study, the assessments specifically evaluating the impact of the New Curriculum focused on Class 1, as at the end of the school year 2016 this group of students have been exposed to the New Curriculum throughout their school life. tudent learning results in 2017, as measured by EGRA, improved in comparison to those of 2009 for Class • S 1 in all but one dimension of the EGRA test6. The biggest gain was in reading comprehension. • Despite the improvements, results in all dimensions measured by the EGRA test remained low. • New Curriculum materials were available in most schools, but their use in the classroom was limited in many. Schools with higher levels of implementation of the New Curriculum performed better on the EGRA. • The PLMP This assessment focused on Class 1 and Class 2, as the PLMP implementation had only reached those school years by the end of the school year 2016. LMP schools showed higher scores in Class 1 compared to non-PLMP schools on EGRA. Similar results • P were observed in Class 2, though the difference was smaller. Additionally, PLMP schools showed significant reductions in the number of zero scores on the EGRA; PLMP schools also performed at a higher level than non-PLMP schools on the CBA, and the difference was • larger in Class 2 than in Class 1; PLMP schools showed stronger performance on some measures of classroom practice, including in • particular more time on task teaching, more effective interaction with students, and better use of the New Curriculum; Associated factors in the learning process and the two learning innovations he best predictor of Class 2 performance on all academic measures was the academic achievement in • T Class 1. Classroom time for teacher-student interactions is low, limiting the time for implementation of the New • Curriculum. Across all classrooms included in the evaluation, the average instructional day was only 66% of that specified by national standards. Only 7.5% were reported as being in session for the required 4.5 hours. Besides low classroom time, implementation of the New Curriculum in class was limited. Observers reported • that the New Curriculum was being utilized in 90 percent of classrooms, in the sense that teachers had the books with them. However, they also reported that in reality, the New Curriculum was only being followed about 55 percent of the time, and of that time, not always as intended. 6 There are five dimensions in the EGRA test: : (i) Letter Knowledge, (ii) Familiar Word Reading, (iii) Non-word Reading, (iv) Text Reading, (v) Reading Fluen- cy, and (vi) Reading Comprehension. The New Curriculum and the Professional Learning and Mentoring Program ix Recommendations This report presents an early evaluation of the implementation of the NC and the PLMP in the education system in Timor-Leste, and its results are affected by the limited use of the NC as reported above. By design, the New Curriculum addresses the issue of linguistic diversity. At the same time, the current evaluation makes abundantly clear that the New Curriculum has not and cannot quickly solve the challenges and build on the opportunities that linguistic diversity brings to the education system. Time, patience, and persistent commitment to implementation are going to be needed before major improvement is realized. This is in line with the findings from other countries where similar initiatives have been implemented Two general recommendations emerged from the evaluation, both related to observed factors which affected or limited impact of the two innovations: First, incomplete implementation of the NC is limiting its impact in the classroom. To accelerate the implementation of the NC, a multi-faceted action plan should be launched to improve teacher/school compliance and abilities to properly implement the curriculum. Second, the PLMP, with its focus on teachers and school administrators, has demonstrated that improvement is possible by strengthening the professional leadership of the school directors and the teachers’ abilities to deliver the curriculum. While the observed impact of this innovation is promising, the results for the first year indicated the need for additional effort to complement this innovation – including the availability of learning materials of the New Curriculum – before the national system of basic education is able to function as desired. x Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste 1. Background and Context During the 24 years that Timor-Leste was under Indonesian control, basic education in Timor-Leste functioned under an Indonesian model with an Indonesian curriculum, Indonesian-speaking teachers, and Bahasa Indonesia as the language of instruction. At the time of independence in 2002, there was a general push to remove the Indonesian model, but there was no clarity nor consensus on how basic education should be delivered. Furthermore, large-scale rebuilding was needed on many levels; schools (80-90 percent of school facilities were destroyed), teachers, training institutions, management systems, etc. had to be put in place before serious work could be done on establishing the policies, standards, and support systems needed for a healthy education system. During the first years of this interim period, many local schools had to operate in an ad hoc manner - finding or building facilities, hiring teachers, finding school administrators, and making operational decisions on an ‘as-needed’ basis. With little official guidance, teachers and schools made many of their own decisions about books, learning standards, language(s) of instruction, curriculum content, school hours, etc. Teachers used any books they could find as textbooks or as references for what they should teach. Some were in Indonesian, some in Portuguese, some in English and possibly other languages as well. Teachers taught their classes in a variety of languages including Indonesian, Portuguese, Tetun, and a variety of local languages. A national primary level curriculum was elaborated in 2004 with the support of UNICEF, which detailed a plan of teaching in Portuguese, with support from Tetun. The EGRA testing carried out in 2009 tested children’s ability to learn to read in both Tetun and Portuguese using this curriculum. With the recent international focus on universalizing primary school enrollment (Dakar 1990 and following), early focus was placed on access — making schools available and getting children enrolled in these schools. Teacher certification and administrative systems were stressed as large numbers of new teachers had to be hired to staff all the new schools which were built. An early target was to raise enrollment rates to at least 80 percent in the early classes. This objective was achieved, as according to the statistical database of the MoEYS (the educational management information system: EMIS), the current net enrollment rate for basic primary is about 90 percent with the gross enrollment rate being approximately 120 percent (2015). The country has an optional kindergarten system with a net enrollment rate of 22 percent (2018). The push to universalize basic education was followed in 2013 by a major effort to develop and field a universal curriculum that recognized the linguistic situation of the country and attempted to systematize the teaching of the two official languages – in this case starting with Tetun and gradually moving to Portuguese. In line with this, MoEYS implemented The New Curriculum as an ambitious effort to bring high pedagogical standards, commonality and continuity to basic primary education and to provide a substantial teacher support system. Today, there are between 35,000 - 45,000 children in each class of primary education (cycles one and two of Basic Education) attending approximately 1,300 schools. The teaching workforce consists of between 11,000 and 12,000 teachers. Most of these teachers have university degrees or equivalence qualifications, though some only have incomplete secondary education. Most teach with some form of certification ranging from completion of a full teacher training qualification to as little as an emergency waiver to serve as a teacher. Teacher training and teacher quality remain a central issue in the country and one which the New Curriculum sought to address. The New Curriculum and the Professional Learning and Mentoring Program 1 This report describes the findings from an educational evaluation carried out in Timor-Leste early in the school year of 2017. The primary purpose of the evaluation was to develop preliminary insights into the effectiveness of the New Curriculum and a second supporting educational innovation, the Professional Leadership and Mentoring Program (PLMP). As such the report addresses the findings of the evaluations regarding improved student learning and relevant school and contextual factors. The objective is to gauge if these two interventions indicate the ability to have some kind of positive impact on educational quality, and if so to help identify areas where these interventions can be intensified. It needs to be noted that both innovations are broad and structural in nature with the potential to have a pervasive impact on the quality of basic education in Timor-Leste. 2 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste 2. Two Innovations The two innovations which have motivated the evaluation are briefly described below: First Innovation: The New Curriculum Until independence, curricula for basic education were based on those of Portugal and then Indonesia, depending on the time frame. After independence in 2002, the issue of curricula became central to the future of basic education in Timor-Leste. The Indonesian curriculum was no longer acceptable because it was not chosen as an official language. In 2004 an official Primary Curriculum was written with a Portuguese language focus, that would serve as the first official curriculum of Timor-Leste as an independent nation. However, results from the implementation of this curriculum, as evidenced by the low scores of the Early Grade Reading Assessment in 2009, showed that it faced many challenges, as it had to be implemented almost exclusively in Portuguese language without systematic instructional guides. Also problematic was that Timorese teachers generally had weak proficiency in both basic pedagogical skills and the Portuguese language. It was therefore not surprising that the children had very low language skills in Portuguese, and as there was little or no emphasis on specific literacy strategies or grammar skill acquisition in the Tetum language, it was also not surprising that the children’s Tetum skills were virtually no better than their Portuguese skills. Accordingly, work began in 2013 to develop a national curriculum based on a clear and systematic progression plan from Tetun, one of two7 official languages and the most widely spoken language in the country, to Portuguese. Work on curricula for the earliest grades was completed in 2015 and then began the work of training national education staff to use this New Curriculum. The first year of full implementation of the New Curriculum was 2016. The original plan was to field two new classes or grade levels per year until work was complete for all six classes. The entire curriculum and accompanying lesson plans and books for the 6 years of cycles one and two have been completely prepared, although by 2017 the higher grades (5-6) of basic education had not yet been implemented in their entirety because the corresponding teacher guidebooks and student textbooks had not yet been distributed. Indeed, book distribution for all levels lagged a year or more behind actual implementation dates, curtailing full and effective curriculum implementation. The New Curriculum can be described as a combination of textbooks, lesson plans, suggested activities, workbooks, a pedagogical guide, instructions for the teacher and a school calendar. The New Curriculum is delivered to schools in the form of thick printed volumes of approximately 450-900 pages. In the first cycle, lesson plans for each year are divided into 3 volumes – one for each period trimester/term – and include all subjects. All major subjects are included in the New Curriculum. It is organized by daily lessons and by units. The innovation includes a training program on how to implement the New Curriculum. These trainings were meant to take place each trimester, as a preparation for the next period of teaching, though, due to financial limitations this training plan was not fully realized. The training was carried out using a cascade model, with a core group of trainers training the assistant directors of each Basic School cluster, who then in turn had the responsibility of training the relevant teachers in their cluster. Training emphasizes simulations of lessons and activities that are to be implemented in the coming trimester, although quality of implementation was varied. 7 Portuguese is one of the official languages. Portuguese is the language of many official documents and it is used in higher education. The population speaking Portuguese as a first language in the country is very small. The New Curriculum and the Professional Learning and Mentoring Program 3 Second Innovation: Professional Leadership and Mentoring Program (PLMP) Early assessments of basic education such as the EGRA of 2009 included notes on the lack of readiness of teachers for the work of classroom instruction.8 This, in fact, has been widely recognized by the MoEYS as a major national challenge—the development of an adequately trained cadre of teachers. To address this need, a national teacher training center was established (currently INFORDEPE), teacher training tracks have been added to the National University of Timor-Leste (UNTL), and a range of other training innovations have been introduced by various international agencies. In 2016, the Professional Leadership and Mentoring Program (PLMP) was introduced. PLMP seeks to address the need for building competency, accountability, and educational professionalism at the level of school administrators and teachers. The PLMP innovation includes a number of elements: leadership training and development for school administrative personnel, a mentoring program focused on school directors but which includes the introduction of processes designed to promote teacher-teacher as well as teacher-administrator interactions, a system of regular assessment of learning outcomes, and a high-tech system9 for creating and managing the flow of education information from individual schools to central locations including both regional and national offices. PLMP was not conceptualized as an alternative to the New Curriculum but rather as a means of supporting and strengthening the implementation of the New Curriculum. 8 https://s3.amazonaws.com/inee-assets/resources/EGRA_toolkit_Mar09.pdf RTI International 2009. Early Grade Reading Assessments, 48. World Bank, Office of Human Development. 9 The term ‘high-tech’ in this case refers to a scheme in which tablet computers were introduced as data-entry devices for recording minutes, test results, administrative reports, teacher evaluations and any other relevant data. At regularly scheduled times, this data was all transmitted digitally over the internet or via a mobile phone connection to a data collection hub in the capital city which automatically entered all data into appropriate databases. Summary reports were also automatically generated and transmitted electronically to designated recipients. 4 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste 3. The Research Questions This evaluation examines the effectiveness of both the New Curriculum and the PLMP by asking the following research questions: 1. Are there early indications of measurable improvement in student learning outcomes at the level of basic education after the introduction of the New Curriculum? 2. Has school performance (measured in terms of learning outcomes) benefited from the implementation of PLMP? The success of an innovation in education depends on more than its mere introduction. Therefore, a useful element of the evaluation was the identification of specific factors which appeared to impact both the implementation and the impact of the two educational innovations in focus — the New Curriculum and PLMP. Thus, a third research question is effectively a part of this report.. 3. What factors positively or negatively impacted implementation and effectiveness of the two innovations in basic education? The New Curriculum and the Professional Learning and Mentoring Program 5 4. Methodology Basic Design Evaluation design for the evaluation of the New Curriculum: The research component which focused on the New Curriculum could not make use of an experimental design since the innovation, by statute, has been implemented throughout the country. Nonetheless, this component has leveraged the fact that there is significant variation in the level of implementation of the innovation in the schools of the country. Evaluation design for the evaluation of the PLMP: The evaluation of the PLMP innovation can be described as a quasi-experimental research design. The evaluation took advantage of the ongoing implementation of the intervention, though the schools selected for this initial deployment were not selected randomly, but to maximize the probability of implementation of the model. The evaluation team selected non-PLMP to match the characteristics of the PLMP schools. Measurement The evaluation used three primary measurement tools, as well as a number of secondary ones. Primary Tools Early Grade Reading Assessment (EGRA) – EGRA is designed to measure children’s progress in learning to read (see Annex 1). The EGRA implemented for this evaluation in 2017 used unmodified components of the EGRAs carried out in 2009 and 201110, thus facilitating the ability to discern improvement (or decline) from 2009 to 2017. There is, of course, the possibility of differences due to sampling. The 2009 sample was relatively small but nationally representative. The 2017 sample, by contrast, was much larger but limited to four of the five municípios in which PLMP had been implemented. EGRA is administered one-on-one to each child so is much more labor intensive (though probably more reliable) than whole class-based assessments. The six components of the EGRA used in this evaluation were: (i) Letter Knowledge, (ii) Familiar Word Reading, (iii) Non-word Reading, (iv) Text Reading, (v) Reading Fluency, and (vi) Reading Comprehension. The EGRA was conducted entirely in Tetun. Curriculum Based Assessment (CBA) – The CBA is tied to the content of the New Curriculum and thus measures the extent to which the content of that curriculum has been mastered (see Annex 2). The subjects tested were Basic Reading, Math, Tetun Language (Class 2 only11) and Portuguese Language (Class 2 only12). The CBA was administered to all children in the target classes in all schools in both treatment and experimental schools (total of 6,326). This instrument was rendered entirely in Tetun, the language of instruction in all schools tested. 10 A combination of EGRA and EGMA (Early Grade Math Assessment) was carried out as well in 2011. As the authors of the 2011 report expressed reserva- tions about the validity of the study, it was decided to not make further reference to that study. 11 TheEGRA of this assessment tested reading ability only in Tetun. Instructions and explanations were also given in Tetun. In contrast, the EGRA of 2009 tested children for reading ability in both Portuguese and Tetun with instructions given in the language of assessment. 12 Portuguese reading and writing is not introduced until Class 3. 6 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste Classroom-based Observations (CBO) – This measurement system was designed to document, in a consistent and detailed manner, what happens in the classroom over the course of a full instructional day (see Annex 3). The protocol tracked, in three-minute blocks of time, what both the teacher and the children were doing in the classroom for the entire school day. The result of this protocol is a map of how much time teachers allocate to each instructional activity, which instructional activities are used, which content areas are covered, and what children actually do in the classroom during the instructional day, and the language of use in the classroom. The CBO assessment was carried out in Classes 1 and 2 since those were the classrooms from which the children came (in 2016) who were tested in February and March of 2017. (See the section on Timing of the Evaluation for an explanation of the relationship between the testing and the classroom observations.) An integral feature of the CBO was a 14-point assessment of the extent to which the New Curriculum is being integrated into the pedagogical functioning of the classroom, which allows the construction of the New Curriculum Index. Additionally, during and after the classroom observations, the observers (who worked in pairs) filled out a 16-point assessment of overall teacher skill or quality using a detailed guide as to the meaning of each point on the assessment scale, which allows the construction of the Teacher Quality Index. Secondary Tools Besides the primary tools described above, the following information was captured by the evaluation team: Teacher assessments of language proficiency. After the instructional day was over, teachers were given language assessments in Tetun and Portuguese to get a basic measure of teacher proficiency in these two languages using listening comprehension as a proxy measure of this proficiency. Although results have been collated, it was decided not to include them as there were questions of reliability of the testing instruments. Additional school information. The team also captured information on school remoteness and accessibility. Sample A total of 128 schools and 6,326 students participated in this study. These schools allowed the evaluation of the two innovations under the following evaluation design: Table 1 presents the description of the total sample of participants in this evaluation by município, class and gender of PLMP schools. Table 2 presents participants of non-PLMP schools. Combining PLMP and non-PLMP students, 6,278 participated in the evaluation13. master data file contains 6,326 individuals. The difference is due to various factors such as missing data, presence for one assessment and not the 13 The second, inconsistent numbers reported by enumerators for class totals, etc. The New Curriculum and the Professional Learning and Mentoring Program 7 TABLE 1. DETAILED DESCRIPTION OF THE SAMPLE POPULATION FROM PLMP SCHOOLS (TREATMENT GROUP) WHICH PARTICIPATED IN THE 2017 EVALUATION Class 1 Class2 CBA EGRA CBA EGRA Monicipios Num. of schools M F M F M F M F Totals Aileu 10 90 75 43 32 89 64 40 34 318 Atauro 10 48 50 35 30 74 69 38 32 241 Dili Remote 10 166 156 45 35 188 138 44 36 648 Liquica 20 244 235 101 91 274 298 102 117 1051 Manatuto 20 172 165 97 89 188 174 77 78 699 TOTALS 70 720 681 321 a 277 813 743 301 297 2957 a The totals in the EGRA columns are not included in the grand total in the far-right column because all participants in the EGRA also participated in the CBA assessment and are accounted for in the CBA numbers. Summing the totals for all EGRA participants, we find that 1,196 children from PLMP schools participated in the evaluation and 954 from non-PLMP schools participated, for a combined total of 2,150 children. The comparison population from the 2009 EGRA was 641 children in the same two classes, making the sample in the current evaluation more than three times greater than that of the 2009 study. TABLE 2. DETAILED DESCRIPTION OF THE SAMPLE POPULATION OF NON-PLMP SCHOOLS (CONTROL GROUP) WHICH PARTICIPATED IN THE 2017 EVALUATION Class 1 Class 2 CBA EGRA CBA EGRA Monicipios Num. of schools M F M F M F M F Totals Aileu 10 103 82 45 32 114 98 49 32 397 Atauro -- -- -- -- -- -- -- -- -- -- Dili Remote 10 189 218 41 39 250 206 37 43 863 Liquica 18 281 267 77 75 300 264 96 69 1112 Manatuto 20 242 214 85 75 254 239 74 85 949 TOTALS 58 815 781 248 221 918 807 256 229 3321 Table 3 compares PLMP and non PLMP schools on seven variables describing general features of the classroom. The most noticeable difference between the two groups of schools is that of CLASS SIZE. Class 1 in PLMP schools averaged seven fewer students per classroom than the non PLMP schools—a difference of 27 percent. Normally, smaller class sizes, especially at the lower levels of basic education, are considered advantageous for children, resulting in improved overall learning outcomes for the children in those smaller classes. This assumes, of course, that the quality of instruction is constant across all class sizes. 8 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste TABLE 3. GENERAL STATISTICS ON THE COMPARISON OF PLMP AND NON PLMP SCHOOLS (CLASS 1 ONLY) IN THE STUDY Non PLMP Schools PLMP Schools Variable N Mean SD N Mean SD Class size 58 27.86 13.64 70 20.26 12.96 SES 57 7.26 2.802 70 6.88 2.034 Absenteeism 57 3.69 2.939 70 3.90 3.567 Remoteness 38 5.43 3.036 52 5.77 2.763 Tetun ability of Teacher 55 3.40 1.471 65 3.13 1.551 Portuguese ability of Teacher 55 1.84 1.273 65 1.84 1.135 The sample for this study is large—6,326 children from 128 schools—and fairly broad, being drawn from four (of 13) different municípios. This represents approximately 6-7 percent of the national population of children in Classes 1 and 2, though no data was gathered in Lautem, which is known to be strong and progressive educationally, nor from Oecusse, which may well be the weakest município educationally. The four municípios from which the data came are all near to the capital city of Dili. It is possible, for this reason, that the spread and influence of Tetun is greater in these municípios than in the country as a whole, and the sample does not represent the country as a whole. The sampling design was significantly influenced by the focus on the PLMP innovation and the fact that this innovation was present in only the five municípios identified earlier. There were also constraints of time and budget, which had a bearing on the breadth of the evaluation. Procedure Enumerators and classroom observers were recruited from the MoEYS. Enumerators and observers received training prior to undertaking these tasks. After training, they went into the field in teams of two to carry out their assignments. Whenever possible, enumerators and observers with experience were paired with those with less experience to maximize the quality of the resultant data. All field trips were unannounced to prevent teachers and school administrators from planning “staged events” for the observers. The intent was to observe a teacher and the classroom as they functioned on a “normal” day despite the distractions posed by the presence of the outside observers. The two assessments (EGRA and CBA) were administered to both Classes 1 and 2 in a given school on the same day. The CBA was administered to ALL children present in those classes in the school on that day. The EGRA was administered to a randomly selected subsample of children with the goal of testing eight children in each classroom. In a few cases, the number was less than eight if there were fewer than that number present due to limited enrollment or higher rates of absenteeism. The New Curriculum and the Professional Learning and Mentoring Program 9 Certain Caveats Timing of the Evaluation A feature of the research design which needs to be clearly understood in reading and understanding the report is related to the timing of the evaluation relative to the local school year in Timor-Leste. All data gathering activity had to be carried out between February and May of 2017.14 It was further specified that testing include children who had completed Class 1 and Class 2 — as these two classes were the only ones that would have had a full year of using the New Curriculum, and were also classes that were tested during the 2009 EGRA, so could be used as a potential baseline for future EGRA work. The school year in Timor-Leste begins in mid-January and ends in November. This meant that the testing work would have to be carried out during the early part of the school year rather than at the end of the school year. The solution was to test children currently in Classes 2 and 3. The critical assumption was that children just beginning Class 2 and Class 3 were educationally equivalent to those who had just finished Class 1 and Class 2 respectively. This seems a reasonably safe assumption for those tested in February just 2-3 weeks after the beginning of the new school year. The assumption would be a little weaker for those tested in early March — a month into the school year. It is also possible that the reverse is true i.e. that children in the first month of a new school year are actually educationally behind those at the end of the previous year because of the knowledge and skill lost during the two months between school years. In the report, all references to Class 1 children are to those who recently completed Class 1 and have begun Class 2. The same is true for those referred to as Class 2 children i.e. these are children who recently completed Class 2 and have begun Class 3. The classroom observations, however, were carried out in actual Classes 1 and 2. The assumption, in this case, is that these classes and teachers were those who provided the actual instructional environment from which the children came who participated in the testing. 14 Thiswas a specific request by the ME to assess students from the 2016 school year, due to timing of upcoming elections in June 2017 which would have most likely delayed the implementation of the assessments, whereas ME was looking for early course correction in implementation as required. 10 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste 5. Report of Findings The presentation of findings will come in two parts corresponding to the two innovations under consideration. Part 1 focuses primarily on the New Curriculum while Part 2 focuses on the PLMP innovation. Each part, in turn, will contain three subsections: (a) impact of the innovation observable from the data, (b) investigation of factors which appear to have facilitated or hindered the impact of the innovation, and (c) a summary of what was learned about the innovation from the analysis. It needs to be noted early on that a certain amount of overlap is unavoidable given that a major feature of the PLMP innovation is that of building capacity in teachers and school administrators to effectively implement the New Curriculum. Part 1 - Impact of the New Curriculum on Learning Outcomes Evidence on Implementation of the New Curriculum Before we look at evidence of impact of the New Curriculum, we need to establish the extent to which the New Curriculum has actually been implemented by the schools included in the study. If/when implementation is lacking, any increases in achievement are likely due to factors other than the New Curriculum. In schools where the New Curriculum has been thoroughly implemented (as documented by the classroom observers), we would expect to see evidence of impact on learning outcomes. A major component of the evaluation included detailed classroom observations monitoring what actually happened in the classroom. These observations were carried out primarily by staff from the MoEYS. This included a detailed listing of evidence that the New Curriculum was actually being used in the classroom. In this section, we summarize the results of this survey. The possible responses are given along with the percentage of those classrooms which responded to each option. The data gathered on the distribution and availability of the New Curriculum indicate that the Ministry and the regions have been quite successful at getting copies of the New Curriculum into the schools and the hands of the teachers. More than two-thirds of teachers also report 1-2 weeks of training on how to make use of the New Curriculum (Table 4). The evidence of actual utilization in the classroom is not quite as strong. For example, in response to the query on whether the content of the lesson specified in the New Curriculum for the day was fully covered (No. 10), only 21.8 percent of the time could a confident “Definitely” be given by the observers. On the summary question of overall use of the New Curriculum (No. 11), only 56 percent of classes earned a strong “Yes.” The New Curriculum and the Professional Learning and Mentoring Program 11 TABLE 4. INDICATORS ON THE USE OF THE NEW CURRICULUM No Yes 1 The school has copies of the New Curriculum (NC) 0.4% 99.6% No Usually Yes 2 The teacher has access to copies of the NC 1.8% 8.6% 90.5% No Yes 3 The teacher has a copy of the NC in the classroom 9.8% 90.2% Not so much Yes 4 The school supports the NC 8.9% 90.1% Little <1 week 1-2 weeks >2 weeks 5 Training in use of the NC 1.8% 27.9% 14.4% 55% No Sometimes Often 6 Teacher consults NC during instruction 10.9% 17.3% 71.8% No Not much Frequently Always 7 Evidence of learning recent content 5.5% 36.8% 50.0% 7.7% No Poorly More or less Definitely 8 The content of the lesson from the NC was fully covered 6.4% 14.1% 57.7% 21.8% Generally good Excellent or full Not being used Fair usage 9 General impression of the extend of use of the NC usage utilization 4.6% 39.3% 45.2% 11.0% We move now to looking for evidence that the New Curriculum has had an impact on learning outcomes in the schools of Timor-Leste. Section A. Observable Impact of the New Curriculum on EGRA Results The first research question is that of whether or the extent to which the introduction of the New Curriculum has improved learning outcomes in the classroom. From the available data, four strategies were identified to address this question. The first is a direct comparison of EGRA results before and after introduction of the New Curriculum. The national representative sample of EGRA test results from 2009 is compared with the EGRA results from four municípios in 2017. The second is a direct comparison of EGRA results from schools which participated in EGRA in both 2009 and 2017. A search of the two data sets found six schools which appear, by their names, to be common to the assessments of 2009 and 2017. While such a comparison would appear to be “nearly perfect” for this study, we need to keep in mind that changes in teachers and school administrations may have happened in one or more of these schools which could have affected educational effectiveness. The third strategy is a comparison of EGRA results in schools in which the New Curriculum appears to have been fully implemented with schools in which it appears that there is much less (or very little) commitment to the New Curriculum based on a 14-point rubric on implementation of the curriculum administered during the CBOs. While, in principle, all schools are supposed to be using the New Curriculum, the reality is that there is significant variation in rates of adoption due to budgetary limitations in providing copies, training of teachers and supervisory personnel, the linguistic skill of teachers (some are not fully proficient in Tetun), limitations of class size, etc. The current evaluation found substantial variation in levels of implementation, thus providing ample data for this approach. 12 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste The fourth strategy is a direct comparison of zero scores on components of the EGRA between 2009 and 2017. A notable feature of the EGRA testing from both 2009 and 2017 is the large number of zero scores on several of the subtasks on the EGRA especially those directly involving reading. If we find a significant reduction in zero scores from 2009 to 2017, such a reduction can be interpreted as evidence that some learning is taking place as a result of use of the New Curriculum. Below are presented the results based on the EGRA of students in Class 1 in 2016. First comparison – EGRA 2009 with EGRA 2017 Table 5 provides summary data on the comparison between Class 1 children in 2009 and 2017. The points of comparison are identical with respect to the instrument used and reading skills being assessed. TABLE 5. COMPARISON OF EGRA TEST RESULTS FOR CLASS 1 CHILDREN BETWEEN 2009 AND 2017 2009 EGRA 2017 EGRA Gain in Percent Percent P value mean Mean SD Mean SD scores zero scores zero scores Letters (100) 27.66 23.95 23.23 33.82 24.26 15.9 0.000 22.3% Words (50) 5.05 9.95 67.10 6.39 10.16 56.3 0.039 26.5% Nonwords (50) 4.08 8.66 70.97 4.04 8.18 69.5 0.940 -0.98% Text reading (58) 6.72 14.22 72.26 7.12 14.88 74.2 0.672 6.0% Fluency (wpm) 6.92 14.87 72.26 7.16 15.02 74.2 0.835 3.5% Comprehension (percent) 6.88 19.54 83.87 14.39 28.33 76.3 0.000 109.2% The left-most column lists the specific literacy skills being assessed. The numbers in parentheses indicate the number of items in the assessment. In the cases of FLUENCY and COMPREHENSION, the units or numbers in parentheses specify the meaning of the numbers in the respective rows. For example, “wpm” means ‘words per minute’, a common way of expressing reading speed. The word “percent” in the last row indicates that the number in the Mean column refers to the percent of comprehension questions answered correctly. Otherwise, the numbers in the Mean column indicate the average number of items answered correctly by all children participating in the evaluation. The comparisons of real interest are those between the Mean for 2009 and the Mean for 2017. The column entitled Gain in mean scores shows the extent to which the level of performance from 2017 is above (or below) that of 2009. The larger the percent of gain, the greater the level of improvement between the 2009 and 2017. From Table 5 we observe that there has been improvement on five of the six literacy tasks of EGRA between 2009 and 2017. The most dramatic improvement is in reading comprehension with a gain of 109 percent. It seems likely that this large gain in comprehension is due to the fact that the New Curriculum systematically promotes the use of Tetun as the language of instruction and as the language of reading in the classroom at the level of Class 1, and the slow but steady growth in Tetun-language reading materials being made available to children. The New Curriculum and the Professional Learning and Mentoring Program 13 Second comparison – performance of the same schools between 2009 and 2017 The 2009 EGRA was based on a national random sample of 40 schools while the 2017 assessment was based on a stratified random sample of 128 schools from four municípios—Aileu, Liquiça, Manatuto and some remote schools in Dili. A comparison of the two lists of school identified six schools which appeared to have participated in both assessments. Even though the sample size is small, we work with the assumption (for the sake of this comparison), that the background, context, and student makeup of these schools have not changed significantly in the eight years between the two assessments. Therefore, a comparison of results should shed light on the impact of the New Curriculum in these schools. TABLE 6. COMPARISON OF RESULTS FOR 6 SCHOOLS PARTICIPATING IN THE EGRAS OF BOTH 2009 AND 2017 (CLASS 1) 2009 EGRA 2017 EGRA Gain in mean Skill being assessed P values Mean SD Mean SD scores Letters (100) 27.00 24.02 40.33 22.96 49.4% 0.000 Words (50) 5.00 12.35 9.06 11.39 84.2% 0.005 Nonwords (50) 3.13 9.06 4.64 7.89 48.2% 0.145 Text reading (58) 6.08 16.25 8.81 14.45 44.9% 0.146 Fluency (wpm) 6.40 25.90 8.81 14.45 37.7% 0.334 Comprehension (percent) 8.33 24.50 14.23 26.85 70.8% 0.063 The column gain in mean scores in Table 6 shows the amount of improvement on each component of the EGRA. These rates of improvement range from a low of 37.7 percent (FLUENCY) to a high of 84.2 percent (WORD READING). The average rate of improvement across all skills is 55.9 percent. In sum, the comparison of the six schools in common between the two assessments also shows significant improvement in learning outcomes in 2017 compared to 2009. The fact that this comparison was between the same schools provides slightly stronger evidence that the New Curriculum does appear to have had a positive impact on learning outcomes. The small sample size, however, requires a cautious interpretation of the finding. Third comparison – full versus minimal implementation of the New Curriculum As noted previously, there is variation in the level of implementation of the New Curriculum. If the New Curriculum is an effective innovation, then we would expect to find that children in schools or classrooms which have fully implemented the New Curriculum would perform at a higher level when all confounding variables are controlled. During the evaluation, classroom observers monitored the extent to which the New Curriculum was being used in the classroom by the teachers. Teachers were also directly asked about the use of the New Curriculum. During the evaluation, it turned out that very few teachers were not using the New Curriculum at all. In most cases, the New Curriculum was being used at least to some extent with considerable variation on how this was being done. It also needs to be noted that there were teachers who were using the New Curriculum intensively (actually reading the text of the New Curriculum, including instructions to teachers, directly to the class) but not correctly. At the same time, there were skilled teachers who referred to the New Curriculum for guidance on the content of the day’s lesson(s) but used their past experience and expertise in the design and presentation of the day’s lessons to augment or otherwise modify the stipulated lesson plan activities. Accordingly, evaluation of use of the New Curriculum was ambiguous in some cases. 14 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste For the sake of this analysis, we compare the performance of children in schools reporting minimal or no use of the New Curriculum to those who were described as using the New Curriculum, without trying to distinguish gradations of usage. The results of this comparison are displayed in Table 7. TABLE 7. IMPACT OF THE NEW CURRICULUM WHEN IMPLEMENTED OR NOT IMPLEMENTED IN CLASS 1 Usage (from some to high) for Class Minimal or No Use Gain in mean Skill being assessed 1 only scores Mean SD Mean SD Letters (100) 30.82 24.99 36.36 25.36 18.0% Words (50) 5.63 9.86 7.64 9.86 35.7% Nonwords (50) 4.11 8.80 4.90 8.73 19.2% Text reading (58) 7.18 16.23 8.48 16.14 18.1% Fluency (wpm) 7.23 16.83 8.53 16.31 18.0% Comprehension (percent) 14.68 30.93 16.54 29.54 12.7% All skills in schools with high level of usage of the New Curriculum show a higher score than when it is not being used, even including the scores from those schools who are perhaps not using it correctly. The differences are all positive though modest in size. In this case, the greatest difference is in word recognition (WORDS). Fourth comparison - the evidence of zero scores The report of the 2009 EGRA contains several references to the number of children with zero ability on one of the various tasks in the EGRA. Before children have been taught a given piece of knowledge or skill, it would not be a surprise that most would score zero if tested for knowledge or mastery of this skill. However, after instruction has begun, one expects to see a steady decrease in the number totally lacking in knowledge and skill and a gradual increase in the level of mastery among the learners. The progression is typically from total lack of knowledge (mostly zero scores) to a normal distribution of skill (a few low scores, many average scores, a few high scores) to mastery (very few low scores and mostly average and high scores). The immediate purpose here is not to further understand the problem of zero scores but to illustrate the nature of this phenomenon in basic education in Timor-Leste. In this section, we investigate the extent to which the New Curriculum has helped reduce (or not) the occurrence of zero scores occurring in the EGRA. As presented in Table 8, in Class 1 where children have had a complete year of instruction under the New Curriculum, we see a consistent reduction in the percentage of zero scores with no change on one task—TEXT READING. The New Curriculum and the Professional Learning and Mentoring Program 15 TABLE 8. ZERO SCORES ON THE EGRA TASKS IN 2009 AND 2017 Percentage of Zero Scores on Each Task in the EGRA Assessments EGRA 2009 EGRA 2017 Reduction in Zero Scores Skill being assessed Class 1 Class 1 For Class 1 Letters (100) 23.23 15.48 33.4% Words (50) 67.10 54.70 18.5% Nonwords (50) 70.97 67.90 4.3% Text reading (58) 72.26 72.37 0.0% Fluency (wpm) 72.26 71.11 1.6% Comprehension (percent) 83.87 74.20 11.5% Section C: Summary of Findings for Part 1 on the New Curriculum The following set of statements summarizes what has been learned so far about the impact of the New Curriculum on learning outcomes. 1. The data gathered on the distribution and availability of the New Curriculum indicate that the Ministry has been quite successful in distributing materials and training teachers. The evidence of actual utilization in the classroom is not quite as strong. 2. The data from the 2017 EGRA shows measurable but limited improvement on learning outcomes in comparison to the 2009 EGRA, particularly for Class 1. Additionally, the Class 1 data from 2017 show a decrease in the number of children scoring at the zero level of approximately 10-15 percentage points depending on the measure of assessment. This is a promising start but the rates of children scoring at the zero level on most skills remains high. 3. Additionally, as will be discussed in the next section, effective learning time in the classroom is limited, due to, among other factors, absenteeism. Limited learning time will limit the exposure of students to the New Curriculum, and their capacity to benefit from it. 16 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste Part 2: The Professional Learning and Mentoring Program (PLMP) The second innovation motivating the current evaluation is the Professional Learning and Mentoring Program. A summary description of this innovation is provided in the following excerpt from project documents. “Professional Learning and Mentoring Program (PLMP) is an MoEYS led Initiative that supports the new child-centered curriculum for Years 1-6. It engages primarily with school leaders to support teachers in the classroom. PLMP is a joint investment by the Governments of Timor-Leste and Australia. It began in 2016 and aims to have all schools in all municipalities phased into the program over five years. Just over one year since the introduction of the Ministry’s New Curriculum and the start of PLMP, monitoring data suggests that classroom practices are improving.” The four PLMP components are: 1. Leadership Program – School leadership strengthening includes a series of events designed to build knowledge and skills. A total of 228 school leaders were participating at the time of this evaluation. 2. Mentoring – Mentoring of school leaders to develop skills in classroom observations and feedback and advice. MoEYS contributes seven national mentors and 36 accompaniers. In addition, 10 international mentors are/have been involved at some point in the program. A total of 4,060 classroom observations have been conducted at the time of this evaluation. 3. Peer Learning Groups – Teacher peer learning groups strengthen teaching of the national curriculum. This initiative is working well, with 105 peer learning groups conducted at the time of this evaluation. 4. Tablets and Technology – A customized IT system is valued for guiding effective monitoring, reporting and informed management. IT is used to record the literacy and numeracy test scores of 299 classes of year 1 and 2. MoEYS consultations and training have been conducted regarding the dashboard data and processes.” The initial implementation of the PLMP, and therefore its impact on student learning, was affected by several factors. First, the program implementation started after the school year and initial activities focused on teacher training, limiting the actual time that students were exposed to the intervention. Second, the first phase of implementation concentrated on grades 1 and 2, though, some project staff have emphasized that their mentoring activities during the first year of implementation gave primary focus to Class 1 teachers. The PLMP innovation aims to raise the level of professional interaction and cooperation among teachers and school administrations; and was officially designed to be meant as a teaching support to teachers of all grade levels. The positive consequences of the development of a school’s educational staff are not conceptualized as being limited to just Class 1 (during the first year). Therefore, it is reasonable to look for evidence of impact in Class 2 as well as in Class 1, though the greatest impact is expected in Class 1. At the time that the data were gathered, the PLMP innovation had only completed a first full cycle of implementation. In this case, a ‘cycle of innovation’ is defined as a period during which the innovation had been fully implemented in a specified block of schools and students had completed a school year in a classroom which operated within the framework of the innovation. A second cycle of innovation was in process during the time of the data-gathering. The New Curriculum and the Professional Learning and Mentoring Program 17 Theoretical Framework A vast range of literature exists on the problems of basic education in developing countries. (See for example Evans and Popova’s 201515 review of six meta studies on what works and what does not in improving educational outcomes in developing countries.) The need to improve the quality of instruction is common to most such countries. The analysis presented in this report on the impact of the PLMP innovation will proceed as follows. Section A presents findings on the impact of the PLMP innovation on learning outcomes. Because the PLMP innovation is specifically aimed at schools (teachers and administrators) as the level of implementation, Section A will present findings at two levels—individuals (both Classes 1 and 2) and schools. Section B will investigate relevant contextual variables which have impacted the implementation of the innovation. This section will also present findings on the same two levels specified under Section A. Finally, Section C will provide a summary of what was learned about the PLMP innovation. Section A – Impact of the PLMP on Learning Outcomes An innovation such as the PLMP has the potential to produce numerous systemic improvements in basic education including such changes as improved classroom instruction, better management of resources, improved supervision, better generation and flow of relevant information between individual schools and the educational hierarchy, and improvements in learning outcomes. Ultimately, improvement in learning outcomes is the metric by which the performance of educational systems is typically measured. In the case of the PLMP, two learning assessments were employed as metrics of comparison between treatment schools (PLMP) and comparison schools (non PLMP). The EGRA instrument is the same as that which had been used in previous years to measure progress towards learning to read. The second assessment is referred to as the Curriculum Based Assessment (CBA). The CBA draws content directly from the New Curriculum for Classes 1 and 2.16 Because of the class-level variation in the case of the CBA, we cannot directly compare results on this instrument between Classes 1 and 2 but we can compare results for treatment versus comparison schools. In a school, the learning of children can be understood as being a product both of personal abilities and school- level factors. Analytical models exist which allow one to account for both sources of influence simultaneously. Such models are quite abstract and become difficult to interpret when a large number of variables are introduced at either or both levels. Accordingly, we have chosen not to employ multilevel models in the analysis. Rather, findings will be presented separately for individual and school level outcomes. For each level, we shall examine the issue of learning outcomes from multiple perspectives in an effort to be as thorough as possible in our analysis. These are: (i) comparisons of group means; (ii) comparisons of extremely low performance; and (iii) comparisons of high or very high performance. These three distinct levels of comparison are designed to ensure that there are no hidden effects associated with the PLMP pertinent to drawing proper conclusions about the impact and effectiveness of the innovation. David K. and Popova, Anna. 2015. What Really Works to Improve Learning in Developing Countries? An Analysis of Divergent Findings in Systemat- 15 Evans, ic Reviews. World Bank Group, Africa Region. Policy Research Working Paper 7203. 16 EachCBA contained a brief section on reading skills, a lengthier section math, and two sections which tested basic knowledge of Tetun and Portuguese respectively. 18 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste Individual-level Findings 1. Comparisons of Group Means Summary comparisons between treatment and comparison schools across both classes are presented in Table 9. TABLE 9. COMPARISON OF PLMP AND NON PLMP SCHOOLS ON THE CBA AND EGRAS BY CLASS N Average (%) SD P % Gain Effect Size CBA Class 1 PLMP 1402 48.69 25.22 0.000 18.7 .311 Non PLMP 1595 41.01 24.14 CBA Class 2 PLMP 1542 39.92 21.37 0.000 12.1 .211 Non PLMP 1719 35.61 19.63 EGRA Class 1 PLMP 564 17.76 20,55 0.000 58.6 .345 Non PLMP 469 11.20 17.00 EGRA Class 2 PLMP 598 37.42 28.32 0.002 17.1 .190 Non PLMP 487 31.96 29.35 In all cases and in both Class 1 and Class 2, children from PLMP schools scored significantly higher than those from non PLMP schools. The column % GAIN indicates how much higher the score from the PLMP program was than that of the non PLMP program. The gain was the lowest in CBA Class 2 (12.1 percent) with the greatest being in EGRA Class 1 (58.6 percent). The numbers in the column, EFFECT SIZE, give an indication of the size or scope of the impact of the PLMP program. In the statistical literature, an effect size of .2 is considered “small” while and effect size of .5 is considered “moderate” in size.17 The effect sizes, defined in this way, all fall in the range between “small” and “medium.” However, the consistency and size of the differences suggest a quite pronounced impact of the PLMP innovation as reflected in the two measures of learning outcomes. Tables 10 and 11 provide additional detail on how students performed on the assessments. 17 Cohen, Jacob. 1988. Statistical Power Analysis for the Behavioral Sciences. Hillsdale, NJ. Lawrence Erlbaum. The New Curriculum and the Professional Learning and Mentoring Program 19 TABLE 10. INTERNAL RESULTS BY SECTION FROM THE CBA FOR CLASS 1 N Average SD P % Gain Effect Size Letters and words PLMP 1404 7.94 (13) 3.86 0.000 21.78 .366 Non PLMP 1595 6.52 (13) 4.00 Reading PLMP 1402 3.31 (8) 2.22 0.000 34.55 .392 Non PLMP 1595 2.46 (8) 2.14 Math PLMP 1414 7.66 (18) 5.04 0.000 9.74 .142 Non PLMP 1603 6.98 (18) 4.61 (NOTE: The scores are ‘raw’ meaning the numbers in the table under Average indicate the number items answered correctly with the number in parenthesis being the total number of items in the assessment.) Table 10 reveals that the PLMP innovation had much less of an impact on math learning outcomes than on those related to language skill development with a percentage gain of only 9.74 percent. In the case of the assessments related to language arts, we see gain percentages of 21.78 to 34.55 percent, gains which are considered substantial in the world of innovation and experimentation. Table 11 presents the corresponding data for the Class 2 CBA. TABLE 11. INTERNAL RESULTS BY SECTION FROM THE CBA FOR CLASS 2 N Average SD P % Gain Effect Size Reading PLMP 1555 6.46 (13) 3.15 0.000 8.57 .167 Non PLMP 1725 5.95 (13) 2.98 Math PLMP 1559 5.31 (18) 4.39 0.000 16.96 .187 Non PLMP 1725 4.54 (18) 3.84 Tetun Language PLMP 1542 3.01 (6) 1.64 0.000 8.27 .142 Non PLMP 1719 2.78 (6) 1.61 Portuguese Language PLMP 1542 1.87 (5) 1.74 0.000 13.33 .126 Non PLMP 1719 1.65 (5) 1.76 (See NOTE above under Table 10). The results for Class 2 are suggestive in two respects. First, the differentials between the PLMP and non PLMP schools appear to be less than the differences observed in Class 1. There are several possible explanations for this finding. First, we must note that the PLMP innovation was implemented simultaneously last year in Classes 1 and 2. An obvious implication is that children currently in Class 2 are facing more demanding curricular content without having had the benefit of explicit instruction in the content specified in the New Curriculum for Class 20 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste 1. This is inevitable with simultaneous multi-grade innovations. Another possible explanation is that the PLMP innovation is more effective for children earlier in the education cycle than for those in later cycles. The Class 2 math results are well below that of the Class 1 children for both PLMP and non PLMP schools. One reason for this finding is that more of the math questions in Class 2 required students to read a line of text to know what the task was. Example, What number comes next in this list of numbers? 5 10 15 20 _____. If a child cannot read, answering this question requires making a guess as to the task to be completed. It is also possible that the math part of the New Curriculum moves at too fast a pace for the teaching and learning environment of Timor-Leste. This problem occurs widely in developing countries so may well be a factor in Timor-Leste as well. Additionally, mentors may not feel as comfortable with helping with math, as the math content of the new curriculum introduces many concepts that were not taught in grade 2 before. Also, as previously discussed, there is evidence that mentors focused on grade 1 during the early implementation of the program, potentially limiting the impact on Class 2. Internal results from the EGRA In this section of the report, the focus is primarily on a comparison of PLMP and non PLMP schools. As in the previous section, the results are broken down by class. Table 12 presents the data for Class 1 and Table 13 presents similar data for Class 2. TABLE 12. COMPARISON OF CLASS 1 STUDENTS FROM PLMP AND NON PLMP SCHOOLS ON THE EGRA MEASURES (DATA FROM 2017). ALL NUMBERS REFLECT RAW SCORES OTHER THAN THOSE FOR READING COMPREHENSION WHICH ARE EXPRESSED AS A PERCENTAGE N Average SD P % Gain Effect Size EGRA Overall PLMP 564 17.76 20.55 0.000 58.57 .345 Non PLMP 469 11.20 17.00 Letter Knowledge PLMP 564 37.31 24,55 0.000 25.71 .318 Non PLMP 468 29.68 23.22 Familiar words PLMP 562 7.95 10.95 0.000 75.50 .341 Non PLMP 468 4.53 8.80 Nonwords PLMP 563 5.03 8.95 0.000 76.50 .268 Non PLMP 467 2.85 6.99 Text Reading PLMP 563 8.89 16.00 0.000 77.80 .264 Non PLMP 468 5.00 13.11 Reading Fluency PLMP 563 8.95 16.20 0.000 78.29 .264 Non PLMP 468 5.02 13.17 Reading Comprehension PLMP 564 18.68% 31.01 0.000 102.16 .338 Non PLMP 469 9.24% 23.74 The New Curriculum and the Professional Learning and Mentoring Program 21 The Class 1 results on the EGRA testing suggest that the PLMP innovation had a strong positive impact on performance on this assessment. Gains ranged from a low of 25.71 percent on letter knowledge to a high of 102.16 percent on reading comprehension. As seen in Table 12, Class 1 students in PLMP schools outscored their counterparts in non PLMP schools on all six of the EGRA tasks included in the assessment. In the case of READING COMPREHENSION, the advantage of PLMP students was more than twice that of students from non PLMP schools. All of the differences were statistically significant. The effect sizes were all between .25 and .35. Table 13 presents the same data for Class 2 students on the same EGRA. These data show a more modest differential between PLMP and non PLMP schools. Note also that all of the effect sizes are below .25. TABLE 13. COMPARISON OF CLASS 2 STUDENTS FROM PLMP AND NON PLMP SCHOOLS ON SIX EGRA MEASURES (DATA FROM 2017). ALL NUMBERS REFLECT RAW SCORES OTHER THAN THOSE FOR READING COMPREHENSION WHICH ARE EXPRESSED AS A PERCENTAGE N Average SD P % Gain Effect Size EGRA Overall PLMP 598 37.42 28.32 0.002 17.08 .190 Non PLMP 487 31.96 29.35 Letter Knowledge PLMP 598 54.48 26.00 0.017 7.88 .148 Non PLMP 485 50.50 28.59 Familiar words PLMP 598 18.31 15.96 0.002 19.36 .185 Non PLMP 487 15.34 16.09 Nonwords PLMP 598 12.38 13.38 0.194 9.65 .080 Non PLMP 486 11.29 11.29 Text Reading PLMP 598 23.81 23.06 0.000 27.53 .224 Non PLMP 485 18.67 22.84 Reading Fluency PLMP 598 25.34 26.30 0.000 28.50 .222 Non PLMP 485 19.72 25.45 Reading Comprehension PLMP 595 42.55 39.41 0.012 16.83 .154 Non PLMP 486 36.42 40.28 All of the differences are statistically significant at the .05 level with the exception of NON-WORDS (p = 0.194). Not surprisingly, all of the performance metrics for Class 2 students for both PLMP and non PLMP are higher than those for Class 1. 22 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste The fact that PLMP schools show greater performance gains in Class 1 than in Class 2 may, in fact, be very significant in terms of better understanding the PLMP innovation and the basis for its observable impact especially in Class 1. Later on, we will investigate the extent to which this advantage is due to the innovation itself versus being due in whole or in part to various contextual factors. 2. Zero scores Previous EGRAs in Timor-Leste have noted that a significant percentage of participants scored a zero on various subtasks of EGRA (or on all of them). An examination of the CBA and EGRA data from this assessment indicates that this “phenomenon” also happened in the 2017 assessment. If few or no participants had scored well, we could simply conclude that the assessments were too difficult. Table 14 reports the impact of the PLMP innovation on the likelihood of a reduction in the number of zero scores because of participation in the PLMP model. Results are reported for six separate points of comparison including both the overall CBA and EGRA as well as several subtests within the two assessments. TABLE 14. IMPACT OF THE PLMP INNOVATION IN TERMS OF THE LIKELIHOOD THAT THE PERCENT OF ZERO SCORES IS REDUCED BY THE INNOVATION. DATA FROM CLASS 1 Program Point of Assessment N Participants Percent 0 Scores Percent Reduction due to PLMP P value Non PLMP Math 1616 8.42 3.0 0.810 PLMP Math 1419 8.17 Non PLMP CBA 1595 3.64 0.00 0.998 PLMP CBA 1402 3.64 Non PLMP Words 468 64.32 23.09 0.000 PLMP Words 562 49.47 Non PLMP Text Reading 468 81.84 17.35 0.000 PLMP Text Reading 563 67.64 Non PLMP Reading Comprehension 469 84.43 15.54 0.000 PLMP Reading Comprehension 563 64.45 Non PLMP EGRA overall 469 18.34 27.48 0.026 PLMP EGRA overall 564 13.30 Interestingly, we see a very large contrast between the paper-based (CBA) and oral (EGRA) assessments. The EGRA subtasks show a greater advantage for the PLMP schools than do the CBA tasks. The percent of zero scores on the written assessment is also dramatically below that of EGRA in both programs. The major conclusion to be drawn from Table 15 is that the children and schools in the PLMP program show a sizeable and statistically significant reduction in the likelihood of scoring at the zero level on the EGRA. Offsetting this observation, we note that the absolute percentage of children scoring at the zero level is high for both the PLMP and non PLMP schools. The New Curriculum and the Professional Learning and Mentoring Program 23 Why is there such a difference between the CBA and EGRA? Several factors help explain this result. First, at least half of the test items in the CBA use a test format (multiple choice, matching, fill in the blanks) which permits guessing. Second, as a paper-based test, the CBA is/was administered at the classroom level so there was much more potential for individual children to benefit from the work of their peers or neighbors. Third, children in PLMP schools are given a ‘mini-EGRA’ test every 3 months to track their progress, so are more accustomed to this type of testing format. Together, these factors sharply reduce the presence of zero scores and the likelihood of large differences. Participants from PLMP schools showed a significant reduction in the likelihood of getting a zero score on EGRA and its various subtasks. On average, we see a reduction of almost 21 percent in the number of children who get zero scores on the various EGRAs. These results suggest a favorable impact of the PLMP innovation in participating schools. Next, we examine results at the level of Class 2. TABLE 15. IMPACT OF THE PLMP INNOVATION IN TERMS OF THE LIKELIHOOD THAT THE PERCENT OF ZERO SCORES IS REDUCED BY THE INNOVATION. DATA FROM CLASS 2 Program Point of Assessment N Participants Percent 0 Scores Percent Reduction due to PLMP P value Non PLMP Math 1725 14.26 14.45 0.081 PLMP Math 1566 12.20 Non PLMP CBA 1719 3.43 7.29 0.685 PLMP CBA 1542 3.18 Non PLMP Words 487 34.91 21.46 0.008 PLMP Words 598 27.42 Non PLMP Text Reading 485 51.55 25.06 0.000 PLMP Text Reading 598 38.63 Non PLMP Reading Comprehension 486 51.03 19.30 0.001 PLMP Reading Comprehension 595 41.18 Non PLMP EGRA overall 487 12.32 47.08 0.001 PLMP EGRA overall 598 6.52 Table 15 shows a significant reduction in zero scores on all of the subtasks of EGRA indicating that substantial learning has taken place during the second year of basic education (Class 2). At the same time, 34.91 percent of all Class 2 children tested in non PLMP schools were unable to read a single word of the common words test. The result is somewhat better in PLMP schools but still stands at 27 percent. As in the case of Class 1, children from PLMP schools were significantly less likely to score at the zero level on the various subtests which make up the EGRA. The reduction in zero scoring is approximately 21 – 22 percent. 24 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste 3. High performance The third perspective on the use of learning assessments to compare the impact of the PLMP innovation is that of high performance. This assessment will help judge whether the PLMP innovation had equal impact among all students and not just some subset of students. For this purpose, we have defined high performance on the math assessment as achieving a minimum score of 65 percent. This level of performance is within a couple of percentage points of being one standard deviation above the mean. This level of performance identifies slightly less than 16 percent of a population when performance is normally distributed. (The Class 1 math scores are approximately normally distributed in the ranges above the mean but are overly represented towards the zero end of the scale below the mean.) The same standard was used for both Classes 1 and 2 even though it was known that math performance was lower in Class 2 than in Class 1. Table 16 presents the results. TABLE 16. DETAIL ON THE IMPACT OF PLMP ON EXCELLING IN MATH IN CLASSES 1 AND 2 (CBA) High Percent at Mean Score at Mean Score Increase in likelihood of Total Performer This Level This Level below This Level Being at this Level Class 1 Math Non PLMP 1603 292 18.22 78.35 29.94 32.18 PLMP 1414 341 24.12 81.31 30.23 Class 2 Math Non PLMP 1725 114 6.61 74.17 21.73 87.29 PLMP 1559 193 12.38 75.71 22.96 In Table 16, we see much the same pattern observed on other measures of testing though with one or two significant differences. Performance in Class 1 looks stronger with slightly more students than expected scoring above the threshold level in both programs (based on the assumption of normality). The difference between the two programs in the number of students achieving at the high level established is statistically significant (p < 0.0001). Again, we note that children in the PLMP schools were more likely (by 32.18 percent) to achieve the standard set for being a high performer. However, the difference in performance among all high scoring students in each program is not greatly different (81.31 vs. 78.35). The profile for Class 2 is notably different from that seen for Class 1 primarily in terms of the much lower percentage of children in each model who performed at the indicated level. Almost twice as many children from PLMP schools achieved at the high level established with the difference being highly significant statistically (p < 0.0001). School-level Findings The first section of the PLMP analysis focused on a comparison of PLMP and non PLMP schools by examining learning outcomes at the level of individual students. However, since the PLMP innovation is specifically designed to raise the quality and professionalism of the staff at individual schools, it is appropriate to examine impact at the school level by giving attention to variables and processes which are school-level rather than individual-level variables. For example, the variable of CLASS SIZE can be treated as a school-level variable. While it is true that the academic performance of a child can be affected by this variable, CLASS SIZE is not a characteristic of an individual person. The New Curriculum and the Professional Learning and Mentoring Program 25 The following two tables summarize a comparison of the two groups of schools—PLMP and non PLMP—on three separate groups of variables: (1) classroom practice and observation, and (2) test results. This approach to the analysis is important in understanding and evaluating the impact of the PLMP because of its primary focus on school staff. Ultimately, of course, we still want to know if PLMP improved student learning outcomes and, if it did, how this was accomplished. Table 17 compares the two groups of schools (Class 1 only) on 10 variables which describe classroom activity and practice. Comparing the two groups of schools on these variables will help identify how the PLMP innovation has led to changes in the performance of teachers which may, in turn, contribute to changes in learning outcomes TABLE 17. A COMPARISON OF VARIOUS CLASSROOM VARIABLES (CLASS 1 ONLY) BETWEEN PLMP AND NON PLMP SCHOOLS Non PLMP Schools PLMP Schools Variable N Mean StDev N Mean StDev Teacher Quality Index 51 58.76 14.14 57 62.23 10.22 Teaching Tetun Literacy 52 49.08 38.96 58 52.55 31.14 Teaching Port. Literacy 52 12.62 21.44 59 15.58 24.98 Instructional Minutes 52 177.40 72.70 59 196.51 55.10 Teaching math 52 40.38 31.54 59 42.25 25.99 New Curriculum Index 53 38.16 11.79 60 39.62 11.14 Personal Practice (min.) 52 26.71 20.96 59 35.69 27.89 Evaluation (min.) 52 14.02 18.75 59 12.50 19.67 Copying from Blackboard 52 25.46 23.73 59 21.54 18.52 Speaking Tetun in Class 52 136.11 60.24 59 140.86 64.89 Speaking Port. In Class 52 12.07 17.22 59 11.11 20.04 Speaking L1 in Class 52 15.63 30.73 59 10.55 28.28 Most of the differences between the two groups of schools on these variables are minor but a couple of them deserve comment. First, we find that the PLMP schools had consistently longer INSTRUCTIONAL TIMES than the non PLMP schools with the overall difference being close to 20 minutes per day. This is a gain of about 11 percent and seems to imply a somewhat higher level of commitment to instruction on the part of teachers participating in the PLMP innovation. Normally, one expects to see an increase in learning outcomes correlate with an increase in instructional time, so this will be an important variable to monitor in the subsequent school- level analysis. We also note that significantly more time was committed to PRACTICE ACTIVITIES on the part of children in the PLMP schools. Since practice activity is also associated with improvements in learning, this may also be one of the ways in which the PLMP innovation has influenced educational behavior on the part of teachers and could also be used to show how much PLMP is helping teachers correctly use the New Curriculum. The one variable on which the PLMP schools were noticeably lower was that of the amount of time that teachers spent using the local language for instructional purposes. This could be viewed as either a positive or negative influence depending on one’s view of best educational practice. The difference, however, was not statistically significant (p = 0.396). 26 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste Comparisons based on test results One of the primary objectives of this assessment effort was to measure the impact of the PLMP innovation. Evidence has already been presented based on the evidence of achievement at the level of individual children. Two separate measures of achievement were used for this purpose — EGRA and a specially developed CBA. For ease of reference, we present again the earlier findings on test results. TABLE 18. SCHOOL-LEVEL COMPARISONS OF PERFORMANCE AMONG PLMP AND NON PLMP SCHOOLS ON 4 POINTS OF ASSESSMENT IN CLASSES 1 AND 2 Class 1 Non PLMP Schools PLMP Schools N Mean StDev. N Mean StDev. Effect size P-values EGRA 57 11.29 10.07 70 17.80 12.89 .556 0.002 CBA 58 38.63 15.63 70 50.78 18.87 .699 0.000 Math 58 36.53 15.72 70 45.43 20.61 .480 0.008 Reading Comp. 57 9.39 13.24 70 18.77 17.88 .505 0.001 Class 2 EGRA 57 31.54 19.47 69 36.91 17.4 .306 0.106 CBA 58 33.99 12.19 70 41.96 13.75 .610 0.001 Math 58 23.15 12.40 70 30.71 14.62 .553 0.002 Reading Comp. 57 35.62 25.01 70 41.94 22.90 .265 0.140 In Table 18 we find that the PLMP schools outscored the non PLMP schools on every point of assessment in both Class 1 and Class 2. The advantage is slightly less in Class 2 but still sizable as measured by effect sizes. Six out of eight comparisons are at or close to the “moderate” level while two — EGRA and READING COMPREHENSION in Class 2 — are at or closer to the “small” level. Section B – Differences between PLMP and Non PLMP School Practices Use of instructional time in the classroom Table 19 compares the amount of time spent by PLMP teachers and non PLMP teachers in each of the classroom activities listed in the table. During the assessment work, classroom observers actually monitored the amount of time committed to each of these activities. Because the observation of a given classroom was limited to a single teaching day, the value or validity of the data rests on the breadth of the data (128 schools) rather than what was true for a given teacher on a given day in a given school. All values in the table are the mean number of minutes devoted to the named task during a typical instructional day. The New Curriculum and the Professional Learning and Mentoring Program 27 TABLE 19. COMPARING CLASSROOM BEHAVIORS OF PLMP AND NON PLMP TEACHERS (CLASS 1 ONLY) Non PLMP PLMP Percent Effect Classroom activities and behaviors Sig. schools Schools difference Size Teacher explaining or lecturing 24.57 29.43 19.8 0.287 .204 Using a visual aid 12.31 18.92 53.7 0.061 .360 Additional instruction in response to question .353 1.294 266.6 0.137 .285 Choral response – teacher and students 22.80 20.43 -10.4 0.517 .124 Copying from the blackboard 25.43 21.63 -14.9 0.347 .180 Individual practice of a skill 26.29 35.90 36.6 .045* .387 Group practice of a skill 15.11 14.74 -2.4 0.914 .021 Student presentation or performance .353 1.265 258.4 0.026* .432 Teacher giving positive feedback to students 8.10 9.78 20.78 0.503 .128 Teacher working one-on-one with student 12.48 11.36 -9.0 0.713 .071 Teacher asking short-answer questions 11.60 9.21 -20.6 0.388 .165 Teacher asking discussion-type questions 18.72 25.22 34.7 0.139 .284 Evaluation activities 14.24 12.34 -13.3 0.606 .099 Singing and games 13.02 13.99 7.5 0.680 .079 “Lao” (no structured activity) 27.47 27.30 .62 0.970 .006 Teacher provides summary of lesson 4.179 5.513 31.9 0.243 .224 Interval (break time) 26.93 31.46 16.8 0.202 .245 * Indicates statistically significant difference. Table 19 compares classroom behaviors between PLMP and non-PLMP teachers. It presents the mean time teachers allocated for 17 classroom activities. Only two activities — INDIVIDUAL PRACTICE OF A SKILL and STUDENT PRESENTATION OR PERFORMANCE — registered statistically significant differences between PLMP and non PLMP schools (at the level of Class 1). The second of these is statistically significant only because teachers in the non PLMP schools gave virtually no time at all to this activity. Overall, the time in PLMP was longer than the time in non-PLMP schools— 180.2 vs. 193.8. Otherwise, a few other observations are possible. PLMP teachers made more use of visual aids, spent less time having their students copying from the blackboard, and were more likely to ask open-ended questions of their students. While all of these appear to be approved tendencies within the world of education, none of them turned out to be strong predictors of performance on the EGRA in Class 1. PLMP and the New Curriculum An obvious question in this assessment project was whether the two innovations overlapped or benefitted from any type of synergy (beneficial interaction between the two innovations). Table 20 provides a detailed comparison between PLMP and non PLMP schools on each point of the rubric used in the assessment to measure implementation of the New Curriculum in the schools of Timor-Leste. The numbers in parenthesis after each item in the rubric indicate the weighting assigned to that item in the evaluation process. Items without a number all had a weight of 5. The combined total weight of all items was 74, as indicated in the last row. The column ‘sig’ shows statistical significance. 28 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste TABLE 20. COMPARING PLMP AND NON PLMP SCHOOLS ON USE OF THE NC IN THE CLASSROOM Non PLMP PLMP Percent Effect Indicators of use of the New Curriculum Sig. schools Schools difference Size School has a copy 5.00 5.00 0.00 -- -- Teacher has ready access to a copy 4.632 4.683 1.1 0.793 .049 Copy in the classroom 2.809 2.656 -5.4 0.378 .167 School supports the NC (6) 3.039 2.902 -4.5 0.189 .250 Teacher received training to use the NC (6) 4.267 4.742 11.1 0.161 .269 The NC was consulted during the day 3.529 3.729 5.7 0.543 .112 Classroom activities reflect influence of NC 2.813 2.712 -3.6 0.699 .074 Student learning reflects influence of NC (7) 4.001 4.169 4.2 0.598 .101 Objectives specified for the NC covered 2.313 2.305 -.3 0.921 .007 Content specified in the NC covered 2.822 3.034 7.5 0.458 .142 Learning activities specified in the NC used 3.467 3.797 9.5 0.280 .209 Overall use of the NC 3.009 3.259 8.3 0.298 .201 Teacher had carefully read the NC for the day 3.608 3.869 7.2 0.389 .166 Teacher’s performance improved by NC 3.266 3.441 5.4 0.518 .124 Overall NC Index (from 0 to 74) 38.330 39.440 2.9 0.609 .097 * Indicates statistically significant difference. Interestingly, there was not a statistically significant difference between PLMP and non PLMP schools on any of the 14 specific sub-measures of this rubric. There were, however, a few observable tendencies along with some fairly strong evidence that the New Curriculum is still not being as widely or as well implemented as intended. The item on which there is the greatest difference between PLMP and non PLMP schools is that of whether teachers received training on the use of the New Curriculum. It seems likely that this reflects the fact that PLMP mentors provided teachers in PLMP schools with additional training or orientation in the use of the New Curriculum. The last few rows in Table 20 generally summarize overall behavior of teachers with respect to the New Curriculum. These columns show a fairly definite tendency for PLMP teachers to make more consistent use of the New Curriculum though the difference is modest. The most striking feature of Table 20 is the extent to which the ratings of both sets of schools fall well below levels indicating full and serious commitment to use of the New Curriculum. For example, the ratings indicate that virtually all schools have a copy and virtually all teachers have access to a copy of the New Curriculum. However, only about 50 percent of teachers actually have a copy of the NC in their classroom for ease of reference. While this may be due to the fact that the school has only one copy of the NC, it may also be the case that a significant number of teachers are indifferent to the NC and how to use it in their teaching. This interpretation is supported by the further observation that only about 50 percent of schools were ranked as supporting use of the NC in basic education. Furthermore, the average score on the NC Index, a detailed rubric measuring use of the New Curriculum, was just 39 out of a possible 74. The New Curriculum and the Professional Learning and Mentoring Program 29 PLMP and a measure of teacher skill and quality Table 21 shows the results of a further general assessment of teacher skill and quality implemented during the study. The rubric used for this purpose contained 16 separate indicators of teacher quality and skill and every item received the same weighting in the rubric. TABLE 21. COMPARING PLMP AND NON PLMP SCHOOLS ON A MEASURE OF TEACHER SKILL AND QUALITY Non PLMP PLMP Percent Effect Indicators of teacher skill and quality Sig. schools Schools difference Size Teacher arrives well before class 3.108 2.716 -12.6 0.222 .392 Teacher is well prepared for class 3.714 4.053 9.1 0.096* .321 Teacher has good records including attendance 4.529 4.237 -6.4 0.238 .155 Teacher is comfortable in the classroom 4.059 4.226 4.1 0.421 .155 Teacher easily maintains control of the classroom 3.607 3.900 8.1 0.158 .272 Teacher enthusiastic and inspiring 3.715 3.983 7.2 0.274 .211 Teacher communicates effectively 3.813 3.900 2.3 0.713 .070 Teacher provides targeted instruction 3.074 3.125 1.7 0.854 .035 Teacher has thorough mastery of content 3.401 3.554 4.5 0.551 .115 Teacher knows how to teach new content 3.518 3.595 2.2 0.755 .060 Teacher gives children plenty of practice 3.681 4.036 9.6 0.145 .211 Teacher probes individual mastery of content 3.803 3.932 3.4 0.576 .107 Teacher understands each child’s progress 3.269 3.748 14.7 0.025* .435 Teacher understands the goals for the whole year 3.500 3.714 6.1 0.415 .156 Teacher understands patterns of child behavior 3.618 4.032 11.4 0.065* .414 Teacher’s classroom runs smoothly 3.718 4.102 10.3 0.077* .343 Overall Teacher Quality Index (80) 58.05 60.72 4.6 0.344 .071 * Indicates statistically significant difference. Table 21 compares measures of skill and teacher quality between PLMP and non-PLMP schools. Of 16 different measures, PLMP teachers have a positive and statistically significant indicator over non-PLMP teachers. These measures are: teacher is well prepared for class, teacher understands each child’s progress, teacher understands patterns of child behavior, and teacher’s classroom runs smoothly. 30 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste Section C – Summary of Findings on the Implementation of PLMP The PLMP innovation is the second of two innovations being investigated in this report. The research question for this innovation was similar to that for the New Curriculum: “Has school performance (measured in terms of learning outcomes) benefited from the implementation of PLMP?” A direct comparison between PLMP and non PLMP schools shows strong improvement in learning outcomes associated with the PLMP schools. The gains on some measures were as high as 50 percent which is very significant. The PLMP schools were especially notable for a tendency to reduce zero scoring in Class 1 on EGRA and its various subtasks. This signals that the mentoring component of the program has been effective in getting teachers to develop greater awareness of the needs of slower learners and ways to support them. Despite the good results, it is important to notice that the results may be have been affected by the non-random selection of the PLMP schools. The PLMP initiative appears to have had a positive impact on teachers and schools in terms of process issues such as more time on task in the classroom, improvements in teacher pedagogy, and gains in the level of accountability that teachers feel. Since we have only seen a first cycle of implementation, it is not possible to project the long-term impact of the initiative. Any substantial improvement in learning outcomes at the Class 1 level will produce positive results as children move on in school. At the same time, the ability of the PLMP initiative to bring about improvement in learning outcomes faces some of the same constraints faced by the New Curriculum, especially with respect to overly large class sizes in urban areas, problems of teacher absenteeism, etc. The New Curriculum and the Professional Learning and Mentoring Program 31 6. Discussion of Findings As noted in the early pages of the report, the two innovations which were the focus of the 2017 assessment had just finalized their initial implementation phase at the time of the assessment. However, even during this initial period, the research data provides ample evidence that the level of implementation of the New Curriculum in the 128 schools included in the assessment was rather modest. Similarly, the PLMP innovation, by its very nature — a focus on leadership development — can be expected to require time before its impact is likely to be fully evident. Specific Findings with Respect to the Three Research Questions The New Curriculum comparison of EGRA results for 2017 showed modest improvement compared to those of 2009 for Class • A 1, with the biggest gain being in reading comprehension; Despite the improvements, results in EGRA are low; • Class 1 students did well on the math portion of the CBA; • Schools with higher levels of implementation of the New Curriculum performed better. • The PLMP • P LMP schools’ results were higher in Class 1 compared to non PLMP schools on the EGRA; • PLMP schools also performed at a higher level than non PLMP schools on the CBA though the extent of the advantage in Class 2 was only about half of that shown for Class 1; • Performance in Class 1 was a very high predictor of performance in Class 2. This implies that effective learning in Class 1 is critical to success in Class 2 and, presumably, beyond. In the field of early basic education, this is known as the Matthew Effect (Stanovich, 1986) which posits that if steps are not taken to increase the likelihood of success for children in Class 1, the likelihood of improving the longer term outcomes will remain low; • PLMP schools showed lower number of students obtaining zero scores on the EGRAs; • PLMP schools showed stronger performance on some measures of classroom practice including especially more time on task teaching, more effective interaction with students, and better use of the New Curriculum; Factors impacting implementation of the two innovations cross all classrooms included in the evaluation, the average instructional day was only 66 percent of that • A specified by national standards. Only 7.5 percent were reported as being in session for the required 4.5 hours. Observers reported that the New Curriculum was being utilized in 90 percent of classrooms. They also • reported that the New Curriculum was only effectively being used about 55 percent of the time. 32 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste A Synthesis of Major Findings The primary focus of this study was an evaluation of two innovations early in the process of their implementation in Timor-Leste. Analysis of the impact of the two innovations identified a small number of factors which must be taken into consideration when doing further planning regarding the two innovations. School quality, the New Curriculum, the PLMP, and the many variables of classroom practice The Ministry (personal communication) has been aware for some time that there are schools which show low levels of learning. It was also well known that these were frequently — but not always — located in remote areas, areas where a local language was the dominant language, and/or where children suffer from malnutrition. Low performance in these cases was somewhat understandable. What was more puzzling were cases in which a low-performing school was next door to a high-performing school in less remote or even urban or semi-urban environments. The two innovations were implemented, in part, in an effort to solve and attempt to address this phenomenon. The analysis demonstrates that at least a little progress has been made, though the problem is far from solved. The New Curriculum has put a potentially powerful tool in the hands of teachers to improve learning. The first year of implementation of the PLMP provides positive evidence of the ability of this innovation to improve teachers’ ability to make proper use of the New Curriculum. However, a single year of implementation is not sufficient to provide a good understanding of the potential of either or both innovations. The New Curriculum speaks to the language issue, but it is early on in the implementation and more time may be needed. The PLMP innovation addresses the issue of teacher professionalization, but the task is massive and will take years to fully solve. As currently conceived and implemented, the PLMP innovation is expensive (which is true for any innovation based on a mentoring approach). In this evaluation, the many variables of classroom practice and activity only occasionally appeared as making a difference in learning outcomes. This could be due to the preliminary nature of the study, or it could be due to more fundamental limitations at the level of instruction which will need to be addressed in the future. The data is clear in showing that low performance is still common in too many schools. Future evaluations will need to attend to this feature of education in the country and the extent to which the problem is or is not being solved by the innovations in place. The New Curriculum and the Professional Learning and Mentoring Program 33 7. Recommendations The study shows a strong need to improve the quality of instruction. The data shows a large amount of variation between schools as well as a large amount of variation within some schools. This variation is linked to limitations either in teachers’ understanding of the role of a teacher or their ability as teachers. The following recommendations address steps which can be taken to improve teacher quality. School internships or learning from schools and teachers who are already successful Since it is quite clear that some schools are successful and some teachers are effective, it is recommended that the Ministry develop a school exchange program, whereby less successful schools could visit and learn from more effective schools. An internship program in which new, inexperienced, or ineffective teachers and administrators are assigned to a six month or one-year internship in exemplary schools to learn how to ensure effective education would also be a potentially effective avenue to explore. Introduce accountability into the system of educational delivery It is highly recommended that the Ministry introduce a system of accountability into the system of basic education in Timor-Leste. Specifically, there is a need to address the following: 1. Significant levels of teacher absenteeism; 2. The failure to use the full instructional day; 3. The presence of teachers who are unable to function as teachers; 4. Lack of ability or unwillingness to use the New Curriculum. (It is important to note that there are some skilled and experienced teachers who produce strong results without visible reference to the New Curriculum.) Tweaks in classroom practice Two elements of classroom practice showed up in the data as impacting outcomes. The first of these was that of summarizing lessons. It should be relatively easy to implement this activity throughout the schools of the country, especially as this activity is already built into the New Curriculum lesson plans. Personal practice time. Personal practice, also known as homework or seatwork, gives children a chance to master or internalize what was just taught by the teacher. It is recommended that the Curriculum Department in the Ministry give attention to how to support personal practice time by means of printing more workbooks, worksheets, or other means of helping teachers strengthen this part of their instruction. 34 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste Final Comments about the Two Innovations Motivating the Evaluation The New Curriculum Statistically or analytically, the variable created in this evaluation as a measure of the degree of implementation of the New Curriculum was frequently ambiguous — sometimes positive, sometimes negative, and sometimes a non-factor. Conceptually, the New Curriculum has a great deal of merit. In practice, the measurable impact of the New Curriculum is overshadowed by: (i) early and incomplete implementation; and (ii) continued weaknesses in teacher quality. Once these two limitations are addressed, one could expect the New Curriculum to make a solid contribution to basic education in Timor-Leste. Since the innovation had completed just one year of implementation at the time of the evaluation, it is too early to draw any conclusions about its longer-term effectiveness. The supporting role played by the PLMP provides evidence that the New Curriculum can, indeed, improve learning outcomes. The PLMP The primary contribution of the PLMP to date appears to be in helping teachers improve pedagogical practice at the level of the classroom. It also seems to have imposed a certain amount of accountability at the school level. These are worthy and needed contributions. At a superficial glance, it could be argued that the role and function of the PLMP could be taken over by national institutions and personnel. At present, however, we believe that taking this step would be premature. Some time is needed before local or national institutions are ready to supply the technical expertise in education that has come with the PLMP program, especially that which is provided by the international mentors. There is some anecdotal evidence that much of the beneficial impact of the PLMP comes from the work of the international mentors. It is recommended that MoEYS, development partners, and project staff work together to develop one or more strategies for strengthening this component of the PLMP, including an intentional component focused on the development of Timorese mentors. The New Curriculum and the Professional Learning and Mentoring Program 35 8. References and Resources Part 1 – The New Curriculum EGRA and EGRA findings: RTI International. 2009. Early Grade Reading Assessment 48. World Bank, Office of Human Development. https://s3.amazonaws.com\inee-assets\resources\EGRA_Toolkit_Mar09.pdf EGRA 2017: RTI International. 2016. The Early Grade Reading Assessment (EGRA) Toolkit, Second Edition. Research Triangle Park, NC: RTI International. Peabody Picture Vocabulary Test: Dunn, Lloyd M. and Douglas M. Dunn. 2007 (4th ed.). Peabody Picture Vocabulary. Pearson Clinical. http://images.pearsonassessments.com/images/assets/ppvt-4/2013-PPVT-Tech-RPT.pdf BICS and CALP: Originally proposed by: Cummins, J. 1981. The role of primary language development in promoting educational success for language minority students. In California State Department of Education (ed.), Schooling and Language Minority Students. A Theoretical Framework. Los Angeles: California State Department of Education. Cited in: Baker, C. 2011 (5th ed). Foundations of Bilingual Education and Bilingualism, 174. Bristol: Multilingual Matters. Part 2 – Professional Leadership and Mentoring Program Improving education in developing countries: Evans, David K, and Anna Popova. 2015. What really works to improve learning in developing countries? An analysis of divergent findings in systematic reviews. Policy Research Working Paper 7203. World Bank Group, Africa Region. https://openknowledge.worldbank.org/bitstream/handle/10986/21642/WPS7203.pdf?sequence=1 Effect size: Cohen, Jacob. 1988 (2nd ed.). Statistical Power Analysis for the Behavioral Sciences, 40, 79-80. Hillsdale, NJ: Lawrence Erlbaum. Ellis, Paul D. 2009. Thresholds for interpreting effect sizes. Hong Kong: Hong Kong Polytechnic University http://www.polyu.edu.hk/mm/effectsizefaqs/thresholds_for_interpreting_effect_sizes2.html Accessed Sept. 20, 2017. 36 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste Mastery learning: Proponents of Mastery Learning require students to achieve at least 80% on tests before moving on. See the original article by: Bloom, B. S. 1971. Mastery learning. In J.S. Block (ed), Mastery Learning: Theory and Practice, 47-63. New York: Holt, Rinehart and Winston. Guskey, T.R. 2010. Lessons of mastery learning. Educational Leadership 68 (2):52-57. http://www.ascd.org/publications/educational-leadership/oct10/vol68/num02/Lessons-of-Mastery- Learning.aspx Laska, John. 1988. A new paradigm for teaching and learning. New Education 10 (1 and 2):76-80. Class size and learning – Kom: Walter, S., and K. Trammell. 2010. The Kom experimental mother-tongue education project report for 2010, Unpublished research report. Available from the authors or SIL Cameroon. SES and learning: American Psychological Association. n/d. Education and socio-economic status. http://www.apa.org/pi/ses/resources/publications/education.aspx Mitchell Institute. 2015. Socio-economic disadvantage and educational opportunity persistently linked. http://www.mitchellinstitute.org.au/fact-sheets/socio-economic-disadvantage-and-educational- opportunity-persistently-linked/ Variation of SES: Mitchell Institute. 2015. Young people in rural and remote communities frequently missing out. http://mitchellinstitute.org.au/fact-sheets/young-people-in-rural-and-remote-communities-frequently- missing-out/ SES and egalitarianism: Guinote, Ana, J. Cotzia, S. Sandhu, and P. Siwa. 2015, January. Social status modulates prosocial behavior and egalitarianism in preschool children and adults. Proceedings of the National Academy of Sciences 112 (3):731-736. Time on task: Cotton. Kathleen. 1989, November. Educational time factors. School Improvement Research Series, Close- Up #8, sponsored by the Office of Educational Research and Improvement, Department of Education, Contract Number 400-86-0006. http://educationnorthwest.org/sites/default/files/EducationalTimeFactors.pdf Support for the importance of knowing the mother tongue: Sparks, R.L., J. Patton, L. Ganschow, N. Humbach, J. Javorsky. 2006, March. Native language predictors of foreign language proficiency and foreign language aptitude. Annals of Dyslexia 56 (1):129-160. https://link.springer.com/article/10.1007/s11881-006-0006-2 Proctor, C. P., August, D., Carlo, M. S., & Snow, C. (2006). The intriguing role of Spanish language vocabulary knowledge in predicting English reading comprehension. Journal of Educational Psychology, 98(1), 159- 169. http://dx.doi.org/10.1037/0022-0663.98.1.159 http://psycnet.apa.org/record/2006-02666-013 Cummins, J. 1979. Linguistic interdependence and the educational development of bilingual children. Review of Educational Research, 49 (2), 222-251. Quote: “a cognitively and academically beneficial form of bilingualism can be achieved only on the basis of adequately developed first language skills” (p. 222). The New Curriculum and the Professional Learning and Mentoring Program 37 Support for the importance of understanding the language of instruction: UNESCO. 1953. African languages and English in education. Education Studies and Documents 11. Paris: Author. “A child should first learn to read and write in the language spoken in his home.... When this foundation has been laid he can acquire a full command of his own and, if necessary, of other languages; without it, there is danger that he will never achieve a thorough command of any language” (p. 67). Matshazi, M. J. 1987. Mother tongue literacy: The importance of learning to read and write in one’s mother tongue. In M. Gayfer (Ed.), Literacy in industrialized countries: A focus on practice. Convergence 20: 3–4, 50–53. Matshazi (1987), an African educator who himself had been required to study in a second language, pled for mother tongue literacy on the basis that, “Learning to talk in your mother tongue is an arduous task. Learning to read and write in that language is even more difficult. But learning to read and write in a language other than your own is an experience that requires an enormous capacity for endurance” (p. 52). International Reading Association. 2001, April/May. Second language literacy instruction: A position statement of the International Reading Association. Reading Today 6. Newark, Delaware: International Reading Association. Literacy learning is easiest when schools provide initial literacy instruction in the child’s home language. Such instruction is consistent with building on a child’s own strengths and connecting unfamiliar material with the familiar to maximize learning efficiency. Literacy skills developed in the home language can then be applied to learning to read and write in a second language, which results in students who have become literate and gained proficiency in two (or perhaps more) languages. (p. 1) https://www.literacyworldwide.org/docs/default-source/where-we-stand/second-language-position- statement.pdf?sfvrsn=6 Re. class size: Whitehurst, Grover J. and Matthew M. Chingos. 2011. Class size: What research says and what it means for state policy. Washington, D.C.: Brown Center on Educational Policy, The Brookings Institution. Quote: It appears that very large class size reductions, on the order of magnitude of 7-10 students per class, can have significant long-term effects on student achievement and other meaningful outcomes. (p. 1) Re. Teacher quality: Pamela D. Tucker and James H. Stronge. 2005. Linking Teacher Evaluation and Student Learning. Alexandria, VA: Association for Supervision and Curriculum Development. Quote from Chapter 1, page 1: Years of research on teacher quality support the fact that effective teachers not only make students feel good about school and learning, but also that their work actually results in increased student achievement. Studies have substantiated that a whole range of personal and professional qualities are associated with higher levels of student achievement. For example, we know that verbal ability, content knowledge, pedagogical knowledge, certification status, ability to use a range of teaching strategies skillfully, and enthusiasm for the subject characterize more successful teachers.1  1See Darling-Hammond, L. (2000). Teacher quality and student achievement: A review of state policy evidence. Education Policy Analysis Archives, 8(1). Retrieved January 22, 2004 from http://olam.ed.asu. edu/epaa/v8n1/ and Stronge, J. H. (2002). Qualities of effective teachers. Alexandria, VA: Association for Supervision and Curriculum Development. http://www.ascd.org/publications/books/104136/chapters/The-Power-of-an-Effective-Teacher-and- Why-We-Should-Assess-It.aspx 38 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste Re. remoteness: James Bashaiza. July 2016. Africa: Performance – Why distance to school matters. New York Times. http://allafrica.com/stories/201607060021.html Quote: Charles Mutazihana, the head teacher of Kigali Parents School, says that long distances cause fatigue as the day commences which kills a learner’s concentration. “The lengthy footing of kids to school disorganizes their concentration in class. Some of them arrive at school sweaty, stressed and exhausted both physically and psychologically, which compromises their performance,” he says. Mutazihana adds that the fact that students are exposed to many things on their way to school affects their studies. Muhia, Nelson. 2016, October. Who would have thought that distance to school would have such an impact on quality education? Uganda: Africa Population and Health Research Center. http://aphrc.org/post/6016 Vuri, D. 2007, May. The effect of availability and distance from school on children’s time allocation in Ghana and Guatemala. University of Rome, Italy: Understanding Children’s Work Project Working Paper Series - UNICEF. http://www.ucw-project.org/attachment/standard_school_distance_vuri20110224_151422.pdf Gender: Lucy Lake. June 2016. The distance to school is not only about the distance you walk. Washington, D.C.: The Brookings Institution. https://www.brookings.edu/blog/education-plus-development/2016/06/08/the-distance-to-school-is- not-only-about-the-distance-you-walk/ Securing girls’ right to go to school is one thing; ensuring they can learn while they are there is quite another. Fuschia Heppworth. n/d. Why are boys under-performing in education? Gender analysis of four Asia-Pacific countries. East Asia and Pacific Regional UN Girls’ Education Initiative: UNICEF. https://www.unicef.org/eapro/report_why_are_boys_underperforming_FINAL.pdf United Nations Children’s Fund (UNICEF). 2015. Storrs, CT: University of Connecticut Model United Nations The Global Initiative on Out-of-School Children, 2010. All in school GENDER. New York: UNICEF and the UNESCO Institute for Statistics http://allinschool.org/barriers-to-education/gender/ Hawthorne effect: Mayo, E. 1933. The Human Problems of an Industrial Civilization. New York: Macmillan. Black, Thomas R. 1999. Doing Quantitative Research in the Social Sciences, p. 143. London, Sage Publications. The Matthew Effect: Stanovich, Keith E. 1986. “Matthew effects in reading: Some consequences of individual differences in the acquisition of literacy.” Reading Research Quarterly, 21(4), pp. 360-394. Low performance in math: Walter, S. L., 2012. “Assessment Results for the Government Teacher Training College (GTTC) in Fundong.” Research report presented to the Ministry of Education in Cameroon. Tangential Items of Possible Interest. In Timor-Leste, gender is a significant predictor of educational outcomes. Girls consistently outscored boys. The advantage ranged from a modest 3-7 percent in math, Tetun language, and Portuguese language to a more substantial 27 and 28 percent respectively on Class 1 and Class 2 EGRA. The New Curriculum and the Professional Learning and Mentoring Program 39 ANNEXES ANNEX 1. INSTRUMENTS EMPLOYED IN THE ASSESSMENT LIST OF INSTRUMENTS USED IN THE STUDY Student evaluations: • Prosedimentu atu halao Avaliasaun baseia ba Kurikulo (ABK) • Curriculum Based Assessment – Avaliasaun ba Progresu Edukasaun iha Ensinu Primaria Timor-Leste - Klase 2 – Avaliasaun ba Progresu Edukasaun iha Ensinu Primaria Timor-Leste - Klase 3 • EGRA assessment – EGRA lnstrusaun ba AvaliadÓr sira – EGRA instrument filled by student (including student data) Classroom Based Observation • Prosedimentu atu halao observasaun ba Aula (CBO) • Gia ba KÓdigu Mapeamentu Atividade iha Klase Laran • Matalan/Gia ba KÓdigu Mapeamentu Atlvldade Iha Klase Laran • Gia: Kompleta lnstromentu D Estimasaun “Sasukat ba Manorin” • lnstrumentu A • lnstrumentu B • Oinsá atu hatama dadus husi instrumentu C ba Baze Dadus • Instrument C (Dadeer) • Instrument C (Lokraik) • lnstrumentu D The New Curriculum and the Professional Learning and Mentoring Program 43 PROSEDIMENTU ATU HALAO AVALIASAUN BASEIA BA KURIKULO (ABK) 1. To’o iha eskola/sa la de aula mais sedu posivel, pelu menus minute 10 antes de hahu aula 2. Halo preparasaun ba administra teste 3. Fo num eru ba teste (se seidauk iha numeru) 4. Administra teste ba Segundo ano (2 Ano) no terseiru ano (3 Ano) dala ida deit ba estudante hotu nebe mak presente iha turma nla laran . 5. Kada ekipa sei administra deit teste ABK ba turma ida (1) 6. Rekolha fali teste nebe kompleta ona 7. Halao selesaun “ random/acak” ba avaliasaun EGRA 8. Administra teste EGRA ba estudante nain 8 uluk husi nain 10 nebe selesiona ona 9. Administra EGRA 10. Administra teste PPVT ba es tudante nain 8 nebe partisipa Iha EGRA. Kuando remata ho Teste : 1. Halo koresaun no pontuasaun ba teste EGRA 2. Halo koresaun no pontuasaun ba teste ABK 44 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste Avaliasaun ba Progresu Edukasaun lha Ensinu Primária Timor-Leste KLASE 2 Fevereiru - Malu 2017 The New Curriculum and the Professional Learning and Mentoring Program 45 Avaliasaun ba Progresu Edukasaun iha Ensinu Primária Timor- Leste: KLASE 2 Instrusaun: LA BE LE hakerek iha espasu ruma bainhira se idauk simu instrusaun. (AvaliadÓr: Ita-Boot presiza fÓ or ienta saun no es plika saun ba estudante hodi kompl eta informasaun iha pájina ne’e.) Estudante nia naram kompletu : ________________________________________________________________________ Estudante nia tinan: ____________________________ Estudante nia seksu: _______________________ Eskola: _________________________________________ Bainhira sei iha Klase 1, estudante ne’e eskola iha ne’ebé: _____________________________________ Se estudante ne’e liu hosi EPE B, hakerek EPE ne’e nia naran: __________________________________ Para iha ne’e no hein instrusaun atu hahÚ teste. Labele hakerek iha kraik. Ema ne’ebé fó teste ka fó valór mak prienxe seksaun ida-ne’e. Ema ne’ebé fó teste nia naran: _____________________________________________________________ Data hala’o teste: __________________________________ Sumáriu valór teste: Seksaun 1 – Lee (pergunta 14): _____________ loos _____________ sala _____________ la koko Seksaun 2 – Matemátika (pergunta 19): _____________ loos _____________ sala _____________ la koko Seksaun 3 – Lian Tetun (pergunta 7): _____________ loos _____________ sala _____________ la koko TOTAL _____________ loos _____________ sala _____________ la koko Estatus formuláriu teste Husi sé Data Marka: ______________________________________ ______________________________________ _____ _____ Rezultadu rejistu: ______________________________________ ______________________________________ _____ _____ Validade: ______________________________________ ______________________________________ _____ _____ 46 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste Dada liña hodi apár liafuan ho ilas. Ezemplu: bibi busa manu 1. (27) ponte sapatu 2. (28) ai hun kanuru 3. (29) uma The New Curriculum and the Professional Learning and Mentoring Program 47 Lee ho kompresaun Tau sírkulu ba liafuan ne’ebé mak deksreve ema tuir ilas ne’e. 4. (45) a. Haksoit b. Hakerek c. Dansa 5. (46) a. Labarik feto ne’e fase ropa. b. Labarik mane ne’e halo kareta. c. Mane ne’e halo uma. 6. (47) a. Labarik mane ne’e duni bibi. b. Labarik ne’e duni busa. c. Labarik ne’e han haas. 7. (48) a. Apa fase hela motor. b. Apa dudu hela motor. c. Apa sae hela motor. 48 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste Lee Istória Favor lee istória ne’e no responde ba pergunta iha okos. Tau sírkulu ba resposta ne’ebé ó hanoin loos. Maria labarik feto ida. Maria iha alin ida naran Rino. Nia inan naran Nina. Nina ba merkadu sosa hahán ba nia família. Nia sosa fos no manu ida ba nia família. Nia sosa rebusadu rua ba nia oan ida-idak. Nia mos sosa manu tolun sanulu resin rua. 8. (32) Nia sosa saida ba nia oan sira? a. fos b. rebusadu c. manu tolun 9. (33) Nina sosa manu tolun hira? a. 10 b. 12 c. 2 10. (34) Rino ne’e se? a. kios nain b. Nina nia laen c. Maria nia alin Toni toos nain ida hela iha foho. Toni nia feen naran Angelita no sira iha oan mane nain neen. Oan mane boot naran Paulo. Toni kontente tanba nia iha oan mane sira ne’ebé bele ajuda nia servisu. Toni kuda batar no koto atu nia família bele haan. Nia mos kuda tomate no kafee atu bele fa’an iha merkadu. Tinan-tinan, Paulo ajuda nia man silu batar. Ne’e servisu todan. 11. (49) Toni nia servisu saida? a. toos nain b. hanorin iha eskola c. kondutor kareta 12. (50) Toni iha oan feto na’in hira? a. la iha b. rua c. neen 13. (51) Tanbasa Toni kuda kafee? a. atu fó han manu b. atu fa’an iha merkadu c. atu fó nia belun 14. (52) Toni nia oan mane boot naran... a. Jose b. Paulo c. Roberto The New Curriculum and the Professional Learning and Mentoring Program 49 Matemátika Aumenta ka hamenus objetu sira hodi hetan nia totál. 15. (35)     +    = ________ 16. (36)       -   = ________ Responde uza adisaun ka subtrasaun. 17. (38) 9 + 7 = _________ 19. (53) 35 - 10 = _________ 18. (40) 10 - 8 = _________ 20. (54) 49 - 7 = _________ Hakerek númeru ne’ebé falta ka tuir mai iha kada liña. 21. (41) Númeru ne’ebé mak tuir mai? 10,15, 20, 25, ______ 22. (42) Númeru ne’ebé mak tuir mai? 2, 4, 6, 8, ______ 23. (43) Hakerek númeru ne’ebé falta? 20, 30, ___, 50, 60 24. (55) 35 26. (57) 76 +14 -33 25. (56) 18 27. (58) 51 +9 -19 28. (59) Tau sírkulu ba númeru ne’ebé mak boot liu númeru 41? a. 41 b. 50 c. 31 50 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste 29. (60) Tau sírkulu ba símbolu ne’ebé deskreve relasaun entre númeru sira ne’e: 97 99 a. > b. < c. = 30. (61) Iha númeru 716, númeru 1 indika saida? a. atus b. nulu c. unidade 31. (62) Iha númeru sira ne’e, númeru ne’ebé falta pár ka impar? 3 4 5 6 ___ 8 9 a. pár b. impar c. seluk tan Lee pergunta ho didiak no tau sírkulu ba resposta ne’ebé loos. 32. (63) Diego sosal sabraka 8 iha merkadu. Iha dalan fila ba uma, Diego no nia maun haan tika sabraka 3. Hea sabraka hira mak Diego lori ba uma? a. 5 b. 3 c. 11 33. (64) Maria sosa manu tolun 22 iha kios ida no nia sosa manu tolun 18 iha kos seluk. Nia sosa manu tolun hamutuk hira? a. 16 b. 20 c. 40 The New Curriculum and the Professional Learning and Mentoring Program 51 Lian Tetun Dada liña hodi apár liafuan Tetun ho ilas. 34. (65) liman 36. (67) fahi kadernu 35. (66) hudi livru Tau sírkulu ba resposta ne’ebé loos. 37. (69) Ami ___________ bola iha kampo. a. tuur b. tebe c. haksoit d. tuda 38. (70) Iha tempu loron ami hotu ba eskola, maibe iha tempu _______________ ami hotu deskansa. a. tinan b. fulan c. kalan d. ulan Tiu Jose lori hudi sabraka ba fa’an iha merkadu. Nia bá merkadu iha loron Sabadu. Nia fa’an hodi sustenta nia familia. 39. (73) Tiu Jose fa’an saida iha merkadu? a. fa’an tomate b. modo c. sabraka d. fa’an hudi no sabraka 40. (74) Nia fa’an hodi halo saida? a. sosa kareta b. fó ba nia oan c. sosa motor d. Sustenta familia 52 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste Avaliasaun ba Progresu Edukasaun lha Ensinu Primária Timor-Leste KLASE 3 Fevereiru - Maiu 2017 The New Curriculum and the Professional Learning and Mentoring Program 53 Avaliasaun ba Progresu Edukasaun iha Ensinu Primária Timor- Leste: KLASE 3 Instrusaun: LA BE LE hakerek iha espasu ruma bainhira se idauk simu instrusaun. (AvaliadÓr: Ita-Boot presiza fÓ or ienta saun no es plika saun ba estudante hodi kompl eta informasaun iha pájina ne’e.) Estudante nia naram kompletu : ________________________________________________________________________ Estudante nia tinan: ____________________________ Estudante nia seksu: _______________________ Eskola: _________________________________________ Bainhira sei iha Klase 1, estudante ne’e eskola iha ne’ebé: _____________________________________ Se estudante ne’e liu hosi EPE B, hakerek EPE ne’e nia naran: __________________________________ Para iha ne’e no hein instrusaun atu hahÚ teste. Labele hakerek iha kraik. Ema ne’ebé fó teste ka fó valór mak prienxe seksaun ida-ne’e. Ema ne’ebé fó teste nia naran: _____________________________________________________________ Data hala’o teste: __________________________________ Sumáriu valór teste: Seksaun 1 – Lee (pergunta 14): _____________ loos _____________ sala _____________ la koko Seksaun 2 – Matemátika (pergunta 19): _____________ loos _____________ sala _____________ la koko Seksaun 3 – Lian Tetun (pergunta 7): _____________ loos _____________ sala _____________ la koko TOTAL _____________ loos _____________ sala _____________ la koko Estatus formuláriu teste Husi sé Data Marka: ______________________________________ ______________________________________ _____ _____ Rezultadu rejistu: ______________________________________ ______________________________________ _____ _____ Validade: ______________________________________ ______________________________________ _____ _____ 54 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste Seksaun 1. Lee Tau sírkulu ba liafuan ne’ebé mak deksreve ema tuir ilas ne’e. 1. (46) a. Labarik feto ne’e fase ropa. b. Labarik mane ne’e halo kareta. c. Mane ne’e halo uma. 2. (47) a. Labarik mane ne’e duni bibi. b. Labarik ne’e duni busa. c. Labarik ne’e han haas. Lee istória tuir mai ne’e no responde ba pergunta. Toni toos nain ida hela iha foho. Toni nia feen naran Angelita no sira iha oan mane nain neen. Oan mane boot naran Paulo. Toni kontente tanba nia iha oan mane sira ne’ebé bele ajuda nia servisu. Toni kuda batar no koto atu nia família bele han. Nia mos kuda tomate no kafee atu bele fa’an iha merkadu. Tinan-tinan, Paulo ajuda nia aman silu batar. Ne’e servisu todan. 3. (73) Toni nia servisu saida? a. halo to’os b. hanorin iha eskola c. kondutor kareta d. tesi ai 4. (74) Toni iha oan mane na’in hira? a. laiha b. rua c. neen d. lima 5. (75) Tanbasa Toni kuda kafee? a. atu fó han manu b. atu fa’an iha merkadu c. atu fó nia belun d. atu hemu iha uma 6. (76) Toni nia oan mane boot naran ... a. Jose b. Roberto c. Paulo d. Pedro The New Curriculum and the Professional Learning and Mentoring Program 55 Gramatika no Komprensaun Lee 7. (77) Tau sírkulu ba liafuan sira ne’ebé mak (substantivu) naran propiu? “Jose ba Dili atu sosa telefone ho batria iha Timor Plaza.” a. Jose, Dili, telefone b. Jose, Timor, Dili c. Jose, battria, Dili d. Timor, telefone, battria 8. (78) Tau sírkulu ba adjetivu sira iha fraze tuir mai. “Feto ki’ik hatais vestidu kapaas.” a. hatais, kapaas b. feto, kapaas c. vestidu, ki’ik d. ki’ik, kapaas 9. (79) Tau sírkulu ba liafuan sira iha orden alfabetu. a. uma, asu, faru, lilin b. asu, faru, lilin, uma c. asu, uma, faru, lilin d. lilin, asu, faru, uma 10. (80) Tau sírkulu ba liafuan tuir mai. “Presu telephone ______________?” a. sé b. hira c. tanba d. oinsa 11. (81) Tau sírkulu ba fraze orden. a. Ha’u ba iha merkadu c. Lori kopu ba fo o nia apa b. Ha’u nia sapatu iha ne’be? d. Ha’u lori kopu ba ha’u nia apa Lee istoria no hatan ba pergunta. Sr. Sanchez hela iha suku Lehara iha Distrito Manatutu ho nia feen no oan nain haat. Sr. Sanchez hanesan toos nai’n ida ne’ebé mak kuda batar no hare’ba nia familia atu han no kafé atu faan iha merkadu. Tinan kotuk hanesan tinan ne’ebé mak diak ba kafé, nune’e Sr. Sanchez planu atu sosa motor ida atu uza hodi ba nia toos. Ida ne’e sei ajuda nia wainhira nia ba merkadu hodi sosa material sira ba nia familia. Nia mos iha planu atu sosa kapasete ida atu seguru ba nia hodi asegura husi asidente ruma. 12. Sr. Sanchez nia planu atu sosa saida hafoin nia hetan osan husi faan kafé? a. bisikleta ida b. motor ida c. uma foun ida d. telefone ida 56 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste 13. Tansa mak Sr. Sanchez hakarak sosa motor ida? a. atu haksolok hamutuk ho nia kolega sira b. atu sa’e hodi ba tasi ibun c. atu ba nia toos no merjadu d. atu lori kafé ba merkadu 14. Tuir istoria ne’e, dalan diak ida atu asegura ita wainhira sa’e hela motor mak; a. sa’e lori neneik liu b. sa’e iha tempu loron deit c. uza kapasete ida d. sa’e mesak deit Seksaun 2. Matemátika 15. 25 19. 318 +14 562 +777 16. 18 +9 20. 37 X5 17. 56 -34 21. 1.019 +7.222 18. 41 -19 22. Tau sírkulu ba númeru ne’ebé mak boot liu númeru 41. a. 40 b. 50 c. 31 d. 14 Tau símbulu < > = ne’ebé deskreve relasaun entre númeru sira ne’e: 23. 97 99 24. 2.019 2.910 25. 1/3 1/4 The New Curriculum and the Professional Learning and Mentoring Program 57 Tau sírkulu ba resposta ne’ebé loos. 26. Iha númeru 56.2177, númeru 6 indika saida? a. atus b. rihun c. unidade d. nulu 27. Iha sekuensia númeru ne’e, númeru ne’ebé falta pár ka impar? 3 4 5 6 ___ 8 9 a. pár b. impar c. seluk tan d. iguala 28. Diego sosal sabraka 8 iha merkadu. Iha dalan fila ba uma, Diego no nia maun haan tika sabraka 3. Hea sabraka hira mak Diego lori ba uma? a. 5 b. 3 c. 11 29. Maria sosa manu tolun 12 iha kios ida no nia sosa manu tolun 18 iha kos seluk. Nia sosa manu tolun hamutuk hira? a. 6 b. 20 c. 30 30. Perimetru husi retangulu tuir mai ne’e hira: 10 m a. 25 meter b. 50 meter c. 15 meter 5m d. 30 meter 31. Sr. Avila iha oan nain haat: Ali ho idade 31, Anna ho idade 40, David ho idade 277, no Julia ho idade 26. Sr. Avila nia oan sira ne’e ida ne’ebé mak indika moris tu-tuir malu, husi moris uluk too moris ikus? a. Ali, Anna, David, Julia b. David, Anna, Ali, Julia c. Anna, David, Julia, Ali d. Anna, Ali, David, Julia 58 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste Seksaun 3. Lian Tetun Maria ho nia familia hela iha knua ida besik mota. Maria iha maun ida naran Anton no alin ida naran Joana. Maria ho nia maun alin sira ba eskola. Maria nia mestre naran Domi. Maria agora klase tolu no Joana iha pre-eskolar B. Maria ho Joana gosta halimar bola. Sira nain rua halimar hamutuk ho belun sira. 32. Maria nia mestre naran saida? a. Jose b. Antonio c. Domi d. Maria 33. Maria nia alin iha klase saida? a. klase 1 b. pre-eskolar B c. klase 2 d. pre-eskolar A 34. Maria ho nia familia hela iha ne’ebé? a. knua ida besik tasi b. knua ida do’ok mota c. knua ida besik mota d. knua ida iha eskola okos Tau sírkulu ba pontu 35. Tansa nia nehan moras ___ ! . ? , 36. Adelina la’o ba eskola ho nia belun sira. a. bainhira b. sé c. tanba d. oinsa The New Curriculum and the Professional Learning and Mentoring Program 59 Seksaun 4. Lian Portugés Dada liña husi liafuan Portugés ida-idak hodi apár ho ilas. 37. cadeira 39. cama água 38. peixe gatos Hili fraze ne’ebé deskreve ka hatete loloos tuir saida mak iha ilas. 40. Ilas ne’e kona-ba saida? a. O homem está a trabalhar no seu jardim b. O homem vai ao mercado com o seu carro c. O homem está a andar de bicicleta 41. Ilas ne’e kona-ba saida? a. O menino está a ir para a escola b. O menino está a jogar á bola c. O menino está a ler 60 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste EGRA Instrusaun ba Avaliadór sira Januariu 2017 The New Curriculum and the Professional Learning and Mentoring Program 61 Instrusaun Jeral: Avaliasaun ida ne’ebé inportante atu kria relasaun ida no konfiansa ho labarik sira; Ezemplu, ko’alia kona-ba tópiku sira ne’ebé labarik sira iha interese. Labarik sei komprende avaliasaun tuir mai ne’e hanesan jogu ida ho situasaun sériu. lmportante atu lee DE’IT sesaun sira ne’ebé iha kaisa laran ne’e ho lian maka’as maibé neineik no mós klaru. Bainhlra avaliasaun remata, fó agradese ba labarik ida-idak ne’ebé oferese nia tempu hodi esforsu sira nia an, no mós fó prezente ba sira ho lapis ida. Sesaun 2. Fó Kuñesementu Kona-ba Letra Nia Naran Ho Nia Son Hatudu ba labarik kona-ba letra sira iha pájina dahuluk (Formuláriu ba labarik) no lee intrusaun sira tuir mai: Iha pájina ne’e, iha letra sira ho nia alfabetu. Halo favór, ko’alia mai ha’u letra sira ida-idak nia NARAN ho letra boot ne’ebé mak ó bele temi-la’os letra ninia son sira maibé letra nia naran. Ezemplu, letra ne’e nia naran (hatudu ba B) nia son mak “BE”. Mai ita prátika: Dehan mai ha’u letra ne’e nia naran [hatudu ba M]: Sé labarik hatán loos dehan: Di’ak, letra ne’e nia naran ho son mak “ME” Sé labarik hatán laloos dehan: Letra ne’e nia naran ho son mak ;’ME” Agora koko ho letra seluk: Dehan mai ha’u letra nia naran [ hatudu ba L]: Sé labarik hatán loos dehan: Di’ak, letra ne’e nia naran ho son mak ‘’ELE’’ Sé labarik hatán laloos dehan: Letra ne’e nia naran ho son mak “ELE” Ó komprende ona sá mak ó halo? Bainhira ha’u dehan “Hahú”, favór dehan sai letra nia naran hotu ne’ebé mak ó hatene. Lee letra sira ne’e iha pajina naruk, komesa husi risku dahuluk nia okos. [Hatudu tetra iha !iii.a dahuluk tuir mai fó nia ezemplu]. Prontu? Komesa. Bainhira labarik komesa lee letra dahuluk avaliador akompaña. Haree no akompana alunus lee salah iha letra ida ne’ebé no tau marka barra (/) iha liafuan ne’e. Kuandu labarik sira lee fali (Kuriji) letra ne’e, konta letra ida ne’e loos. Nonok nafatin, ezetu ho situasaun hanesan; Bainhira labarik nonok liu detik/segundu 3, dehan sai letra ne’e nia naran ba labarik no tau marka iha letra ne’e laloos, hafoin kontinua hatudu fali letra tuir mai no dehan “Favor bá oin.” 62 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste Depois to’o iha detik/segundu 60, “PARA”. Marka letra ne’ebé lee ikus ho sinal kolxete (]). Regr,a atu hapara ezersísiu: Se labarik la fó resposta ida ka rua ne’ebé iha lina ida dahuluk, dehan “Obrigada barak”, ba ezersísiu ne’e, marka sinal ida iha okos hafoin pasa fali ba ezersísiu tuir mai. Sesaun 4. Identifika Liafuan Ne’ebé Iha Familiar Hatudu ba labarik sira papél ida ho liafuan familiar sira iha pájina daruak (Formuláriu ba labarik. Dehan:) Iha ne’e iha liafuan balu. Ha’u hakarak ó atu lee liafuan sira mai ha’u ne’ebé mak ó bele lee (Labele soe letra maibé lee de’it). Ezemplu, liafuan ida ne’e: “Busa’’. Mai ita koko: Favór lee liafuan ne’e [Hatudu ba liafuan ne’e “Biti”]: Sé labarik lee ho loos ita dehan: Di’ak, liafuan ida ne’e “Biti” Sé labarik lee laloos ita dehan: Liafuan ida ne’e “Biti” Agora ita koko fali liafuan seluk: Favór lee liafuan ida ne’e [Hatudu ha liafuan “Leten”) Sé labarik lee ho loos ita dehan: Di’ak, liafuan ida ne’e “Leten” Sé labarik lee laloos ita dehan: Liafuan ida ne’e “Leten” Bainhira ha’u dehan “Hahu”, favór temi sai liafuan nia naran hotu ne’ebe mak ó hatene. Lee liafuann sira iha pájina nia naruk, komesa hosi risku dahuluk nia okos. [Hatudu ba iha liafuan dahuluk iha liña dahuluk depois fó ezemplu]. Prontu? Komesa. Bainhira labarik komesa lee liafuan dahuluk iha testu laran avaliador akompaña. Haree no akompaña alunus lee salah iha liafuan ida ne’ebé no tau marka barra (/) iha liafuan ne’e. Kuadu labarik kuriji ka lee fali, konta ida ne’e loos. Nonok nafatin, ezetu ho situasaun hanesan; Bainhira labarik nonok liu segundu/detik 3, dehan sai liafuan nia naran no marka liafuan ne’e laloos, hafoin kontinua hatudu fali liafuan tuir rnai dehan “Favór ba oin.” Depois segundu/detik 60 ona dehan, “Para”, Marka liafuan ne’ebe lee ikus nee ho sinal kolxete (]). Reera atu hapara ezersísiu: Se labarik la fó resposta ida ka rua ne’ebé loos iha liña ida dahuluk, dehan “Obrigadu barak”, ba ezersísiu neie, marka kaixa ida iha okos hafoin pasa fali ba ezersísiu tuir mai. The New Curriculum and the Professional Learning and Mentoring Program 63 Sesaun 5. Dekodifikandu Ba Progresu Inventa Liafuan Sira Hatudu ba labarik sira papel ida ho liafuan inventa sira iha pájina datoluk (Formuláriu ba labarik). Dehan: Iha ne’e iha liafuan balu. Ha’u hakarak ó atu lee liafuan sira mai ha’u ne’ebé mak ó bele lee (Labele soletra maibé lee de’it). Ezemplu, liafuan ida ne’e: “uta’’. Mai ita koko: Favór lee liafuan ne’e [Hatudu ba liafuan ne’e “kut”]: Sé labarik lee ho loos ita dehan: Di’ak, liafuan ida ne’e “kut” Sé labarik lee laloos ita dehan: Liafuan ida ne’e “kut” Agora ita koko fali liafuan seluk: Favór lee liafuan ida ne’e [Hatudu ha liafuan “mab”) Sé labarik lee ho loos ita dehan: Di’ak, liafuan ida ne’e “mab” Sé labarik lee laloos ita dehan: Liafuan ida ne’e “mab” Bainhira ha’u dehan “HahÚ”, favór temi sai liafuan nia naran hotu ne’ebe mak ó hatene. Lee liafuann sira iha pájina nia naruk, komesa hosi risku dahuluk nia okos. [Hatudu ba iha liafuan dahuluk iha liña dahuluk depois fó ezemplu]. Prontu? Komesa. Bainhira labarik komesa lee liafuan dahuluk iha testu laran avaliador akompaña. Hare no akompaña alunus lee salah iha liafuan ida ne’ebé no tau marka barra (/) iha liafuan ne’e. Kuadu labarik kuriji ka lee fali, konta ida ne’e loos. Nonok nafatin, ezetu ho situasaun hanesan; bainhira labarik nonok liu segundu 3, dehan sai liafuan nia naran no marka liafuan ne’e laloos, hafoin kontinua hatudu fali liafuan tuir mai dehan “Favor bá oin.” Depois segundu 60 ona dehan, “PARA”, Marka liafuan ne’ebe lee ikus nee ho sinal kolxete (]). Reera atu hapara ezersísiu: Se labarik la fó resposta ida ka rua ne’ebé loos iha liña ida dahuluk, dehan “Obrigadu barak”, ba ezersísiu neie, marka kaixa ida iha okos hafoin pasa fali ba ezersísiu tuir mai. 64 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste Sesaun 6. Leitura No Komprensaun Ba Testu Hatudu istória simples ida iha pájina dahaat (Formuláriu ba labarik) no dehan: Iha ne’e iha istória badak ida. Ha’u hakarak ó atu lee maka’as. Bainhira lee hotu tiha, ha’u sei husu ó kona-ba saida mak ó lee tiha ona iha istória ne’e. Ó komprede ona kona-ba saÍda mak ó atu halo? Bainhira ha’u dehan “Komesa” entaun lee istória ne’e tuir buat ne’ebé o hatene. Prontu? Komesa. Bainhira labarik komesa lee liafuan dahuluk iha testu laran avaliador akompaña. Hare no akompaña alunus lee salah iha liafuan ida ne’ebé no tau marka barra (/) iha liafuan ne’e. Kuadu labarik kuriji ka lee fali, konta ida ne’e loos. Nonok nafatin, ezetu ho situasaun hanesan; bainhira labarik nonok liu segundu 3, dehan sai liafuan nia naran no marka liafuan ne’e laloos, hafoin kontinua hatudu fali liafuan tuir mai dehan “Favor kontinua.” Depois segundu 60 ona dehan, “PARA”, Marka liafuan ne’ebe lee ikus nee ho sinal kolxete (]). Reera atu hapara ezersísiu: Se labarik la fó resposta ida ka rua ne’ebé loos iha liña ida dahuluk, dehan “Obrigadu barak”, ba ezersísiu neie, marka kaixa ida iha okos hafoin pasa fali ba ezersísiu tuir mai. Bainhira labarik lee istória ne’e iha minutu 1 nia laran (Segundu 60) ka lee menus husi minutu 1 laran, tau testu ne’e iha labarik sira nia oin no husu hahusuk (Husu pergunta) dahuluk iha tabela liman loos. Fó tempu segundu 15 ba labarik atu hatán pergunda ida depois mak pasa fali ba pergunta seluk. Maibé se husu de’it hahusuk ba labarik kona-ba istória ne’ebé nia lee ona tuir marka sinal ( ] ) ne’ebé mak iha. The New Curriculum and the Professional Learning and Mentoring Program 65 Instrusaun Jerál: Avaliasaun ida ne’ebé inportante atu kria relasaun ida no konfiansa ho labarik sira; Ezemplu, ko’alia kona-ba tópiku sira ne’ebé labarik sira iha interese. Labarik sei komprende avaliasaun tuir mai ne’e hanesan jogu ida ho situasaun sériu. lmportante atu lee DE’IT sesaun sira ne’ebé iha kaisa laran ne’e ho lian maka’as maibé neineik no mós klaru. Bainhira avaliasaun remata, fó agradese ba labarik ida-idak ne’ebé oferese nia tempu hodi esforsu sira nia an, no mós fó prezente ba sira ho lapis ida. A. Data Avaliasun B. Naran intervistadór/a C. Naran no Lokazasaun Eskola D. Kódigu Eskola E. Natureza Eskola  1 = PLMP  2 = Non-PLMP  3 = Non-PLMP Privadeu F. Klase ne’e iha turma hira? G. Nivel Estudante  2 = 2a klase  3 = 3a klase H. Estudante nia naran I. Estudante nia data moris Data : Fulan : Tinan : J. Tinan hira (Idade) K. Jéneru  1 = Feto  2 = Mane L. Esperiensia Eskola Anterior (Se iha, Fó 1a klase: _______________________________ eskola nia naran)  2 = 2a klase _______________________________  3 = 3a klase _______________________________ M. Falta Loron hira? N. Lian Lian saida mak ó-nia inan-aman uza h odi koalia ba malu? _____________ Lian saida mak ó koalia ho o-nia inan-aman? _____________ 66 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste Sesaun 2. Fó Kuñesementu Kona-ba Letra Nia Naran Ho Nia Son Ezemplu: b m L 1 2 3 4 5 6 7 8 9 10 f E N S T d A O i L (10) h A G D R t U b e r (20) O L M e U a E T R w (30) g R B E i f M t s r (40) S T K N p A I W a E (50) a S W A M K O t n P (60) e A E s O F H u A t (70) R w H b S i G m i L (80) L a N O e o E r p X (90) N A K D d a O j e n (100) Marka sinal (X) iha kaisa tuir main ne'e sé ezersisiu la kontinua tanba labarik la hatán loloos iha liña dahuluk:  Sé labarik sira lee kompletu hotu iha ezersisiu ne'e maibé menus husi segundu 60 entau hakerek segundu hira mak sira uza hodi lee: _______________________ Halo nota ba total númeru letra ne'ebé sira lee durante iha tempu ezersisiu (Segundu 60): __________________ Halo nota ba letra ne'ebé sira lee LOOS durante tempu ezersisiu: __________________ Halo nota ba letra ne'ebé sira lee LALOOS durante tempu ezersisiu: _________________ Servisu di'ak! Mai ita kontinua sesaun tuir mai. The New Curriculum and the Professional Learning and Mentoring Program 67 Sesaun 4. Identifika Liafuan Ne'ebé Iha Familiar Ezemplu: busa biti leten 1 2 3 4 5 ba sai azúl buka mai (5) ajuda rua halai haree tun (10) busa no halimar leten ita (15) gosta hakarak sei agora tenke (20) favór sé kraik sira di'ak (25) obrigadu lee dansa hatene nia (30) haksoit ami toba semo bele (35) tanba matak kanta mai nafatin (40) barak Ó biti tuur hamoos (45) hitu hemu hamnasa ha'u hamutuk (50) Marka sinal (X) iha kaisa tuir main ne'e sé ezersisiu la kontinua tanba labarik la hatán loloos iha liña dahuluk:  Sé labarik sira lee kompletu hotu iha ezersisiu ne'e maibé menus husi segundu 60 entau hakerek segundu hira mak sira uza hodi lee: _______________________ Halo nota ba total númeru letra ne'ebé sira lee durante iha tempu ezersisiu (Segundu 60): __________________ Halo nota ba letra ne'ebé sira lee LOOS durante tempu ezersisiu: __________________ Halo nota ba letra ne'ebé sira lee LALOOS durante tempu ezersisiu: _________________ Servisu di'ak! Mai ita kontinua sesaun tuir mai. 68 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste Sesaun 5. Dekodifikandu Ba Progresu Inventa Liafuan Sira Ezemplu: uta kut mab 1 2 3 4 5 kib rel san lek pof (5) huz tup buf Fev maz (10) nuv gak duig Kos dit (15) jod eg wix Vom zad (20) fut zode hime nepe jabe (25) phune xo shi whe tade (30) staw plea fler esmala blor (35) klefa troba spail fel groaf (40) posnuva relime defe gartion prefute (45) unkaben odeda sanitoda fikadanu maful (50) Marka sinal (X) iha kaisa tuir main ne'e sé ezersisiu la kontinua tanba labarik la hatán loloos iha liña dahuluk:  Sé labarik sira lee kompletu hotu iha ezersisiu ne'e maibé menus husi segundu 60 entau hakerek segundu hira mak sira uza hodi lee: _______________________ Halo nota ba total númeru letra ne'ebé sira lee durante iha tempu ezersisiu (Segundu 60): __________________ Halo nota ba letra ne'ebé sira lee LOOS durante tempu ezersisiu: __________________ Halo nota ba letra ne'ebé sira lee LALOOS durante tempu ezersisiu: _________________ Servisu di'ak! Mai ita kontinua sesaun tuir mai. The New Curriculum and the Professional Learning and Mentoring Program 69 Sesaun 6. Leitura No Komprensaun Ba Testu Agora ha'u sei husu hahusuk (Pergunta) ruma kona-ba istória ne'ebé mak ó lee ona. Koko atu hatan pergunta ne'ebé ó sente bele atu hatán. Bondia. Ha'u nia naran 5 Maria Haú nia tinan hitu. Ha'u nia 14 Maria nia alin naran saida?  Loos  Laloos  La hatán alin naran José. (José) Nia tinan lima. Ha'u mos iha 22 Maria nia familia iha labarik  Loos  Laloos  La hatán bin ida. nain hira? (Tolu) Nia naran Ana. Ami hotu- 31 Maria hela iha ne'ebé?  Loos  Laloos  La hatán hotu hela iha Dili. (Dili) Ami bá eskola husi loron 40 Sira ba eskola loron hira?  Loos  Laloos  La hatán Segunda too loron Sábadu. (Segunda too sabadu) Ami hotu gosta lee istória. 51 Maria nia apa servisu Ha'u nia apa servisu nu'udár saida? mestre. (Professor/Mestre) Ha'u nia ama faan aifuan 58 Maria nia ama halo saida  Loos  Laloos  La hatán iha merkadu. iha Merkadu? (Faan aifuan) Marka sinal (X) iha kaisa tuir main ne'e sé ezersisiu la kontinua tanba labarik la hatán loloos iha liña dahuluk:  Sé labarik sira lee kompletu hotu iha ezersisiu ne'e maibé menus husi segundu 60 entau hakerek segundu hira mak sira uza hodi lee: _______________________ Halo nota ba total númeru letra ne'ebé sira lee durante iha tempu ezersisiu (Segundu 60): __________________ Halo nota ba letra ne'ebé sira lee LOOS durante tempu ezersisiu: __________________ Halo nota ba letra ne'ebé sira lee LALOOS durante tempu ezersisiu: _________________ Servisu di'ak! Mai ita kontinua sesaun tuir mai. 70 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste 27) Iha ó-nia uma, ó iha ............? (Hatán loos ka la'e ......) Loos La'e La Hatene Eletrisidade Jeleira TV Rádio Aparellu DVD Fgaun elétriku/gas Ai sunu (Prezisa atu hili ai) Bisikeleta Kareta Motor Hariis fatin/sentina iha uma laran Telefone (Mobile Phone) Bee torneira Simente Besi latasen Agradese ba labarik, ba ninia partisipaun tomak iha avaliasaun ida ne'e no sei fó lapis ida ba labarik ida-ida ne'ebé tuir intrevista ida ne'e. The New Curriculum and the Professional Learning and Mentoring Program 71 Ezemplu: b v L Sesaun 2 f e n S T d A O i L h a G D R t u b e r O L m e U a E T R w g R B E i f m t s r S T K N p A I W a E a s w A M K O t n P e A e s O F h u A t R w H b S i g m i L L a N O e o E r p X N A K D d a O j e n Ezemplu: busa biti leten Sesaun 4 ba sai azúl buka mai ajuda rua halai haree tun busa no halimar leten ita gosta hakarak sei agora tenke favór sé kraik sira di'ak obrigadu lee dansa hatene nia haksoit ami toba semo bele tanba matak kanta mai nafatin barak Ó biti tuur hamoos hitu hemu hamnasa ha'u hamutuk Ezemplu: uta kut mab Sesaun 5 kib rel san lek pof huz tup buf Fev maz nuv gak duig Kos dit jod eg wix Vom zad fut zode hime nepe jabe phune xo shi whe tade staw plea fler esmala blor klefa troba spail fel groaf posnuva relime defe gartion prefute unkaben odeda sanitoda fikadanu maful 72 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste Sesaun 6 Bondia. Ha'u nia naran Maria. Ha'u nia tinan hitu. Ha'u nia alin naran José. Nia tinan lima. Ha'u mos iha bin ida. Nia naran Ana. Ami hotu-hotu hela iha Dili. Ami bá eskola husi loron Segunda too loron Sábadu. Ami hotu gosta lee istória. Ha'u nia apa servisu nu'udár mestre. Ha'u nia ama faan aifuan iha merkadu. The New Curriculum and the Professional Learning and Mentoring Program 73 Prosedimentu atu halao observasaun ba Aula (CBO) Antes hahu aula 1. To'o iha sala minute 20 antes de hahu aula. 2. Informa kordenador e professor kona ba ita servisu. 3. Impresta kopia lista prezensa. Konta numeru estudante nebe inikut iha lista. 4. Husu atu bele hare i examina professor nia kopia husi reforma kurikulu. 5. Husu ba profesor atu bele hare planu de aula ba loron ohin. Antes de hahu aula, pojisiona ita nian an iha sala laran para bele regista aktividades iha sala laran durante aula, diak liu se ita kotuk para bele oberva ho diak. 6. Kuando aula hahu, komesa regista observasaun baseia ba mapa de tempo nebe iha ona. 7. Kontinua halo observasaun kada minute tolu e priense mapa de tempo nebe los. Depois de aula remata 8. Halo entrevista badak ho direitor kona ba professor nia utilijasaun ba reforma kurikulo. 9. Halo entrevista ho professor kona ba utilijasaun reforma kurikulo. 10. Husu professor atu esplika kona ba relasaun aula ohin nian ba kurikulu no mos hanusa mak aula ne bele suporta atingi objetivu kurikulu tinan ida nian. 11. Husu profesor atu deskreve progresu akademiku husi estudante ida-idak tuir lista presenza. 12. Administra teste Portuguese ba profesor. Depois de kompleta pontos hotu iha leten, kompleta intrumentu 3 tuir mai mak hanesan: 13. Kompleta instrumentu D. 14. Kompleta instrumentu A. 15. Kompleta instrument B. 74 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste Gia ba Kódigu Mapeamentu Atividade iha Klase Laran Atividade mapeamentu tempu nu'udar parte xave ida iha estudu obervasaun aula. Iha mapeamentu tempu, obervadór hato'o imformasaun kada minutu tolu kona-ba saida mak akontese iha aula laran. Wainhira lisaun dahuluk hahuu iha klase, obervador tenke tau pontu iha tabela besik ba periódu tempu nu'udar indikadór katak aula dadeer hahuu ona. Hofoin, kada minutu tolu, obervador sei halibur informasaun kona-ba saida mak akontense iha klase laran i depois passa informasaun iha mapa tempu tuir gia ne'ebé fó iha okos. Bainhira laiha mdansa liu kedas iha koluna entre minutu tolu ida ba minutu tolu seluk, la presiza hatama informasaun iha koluna ne'e. Presiza h atama informasaun ba koluna wainhira iha diferensa atividade entre minutu tolu dahuluk ho atividade iha minutu tolu tuir mai. Hatama informasaun foun uza kódigu ne'ebé esplika iha okos ne'e. Asegura hodi lee instrusaun ne'ebé aplika iha Koluna 4, Laos ba Atividade. Koluna 1 - Materia Iha kategoria rua hosi informasaun mak sei halo mapeamentu iha koluna Materia. 1. Dahuluk mak nu'udar indikadór ba hahuu no remata instrusaun ka aula. a. Hahú materia (H) b. Remata materia (R) c. Hakat tópiku ka materiá (HTM) d. Kontinua materia (K) 2. Daruak mak nu'udar identifikadór ba materia foka iha klase laran. Koluna ne'e rezerva ba de'it konteudu instrsaun ka konteudu aula. a. Dalen Tetun no Literasia (T) b. Dalen Portugés no Literasia (P) c. Matemátika (M) d. Siénsia Naturál (SN) e. Siénsia Sosiál (SS) f. Arte no Kultura (AK) g. Saúde (S) h. Edukasaun Fízika (EF) i. Relijiaun (RE) Uza letra sira no ifon hodi indika ba informasaun relevante. Haree izemplu sira iha okos. HT = hahuu sesaun ida iha literasia Tetun RP = hahuu sesaun ida iha dalen Portugés ka literasia ISN = intervalu durante sesaun iha Síensia Natural KSS = kontinua fali sesaun iha Síensia Sosial. The New Curriculum and the Professional Learning and Mentoring Program 75 Koluna 2 - Atividade sira Koluna ne'e rezerva ba identifikasaun klase no estudante nia atividade iha kualkér materia ne'ebé hanorln. Sei lalha valór ida se karik koluna ne'e relata kona-ba aprendizajen ka atividade seluk hanesan intervalu, disiplina klase, tempu bolu, ka merenda. Maibe, tetra ida de'it ne'ebe iclentiflka atividade mak sei tau iha lina. Uza kódigu tuir mai hodi identifika atividade: lnstrusaun relasiona ho atividade a. Manorin hanorin, esplika ka dernonstra abiidade foun (T) b. Manorin uza material vizual ka objetu hodi ajuda hanorin konseitu ka abilidade foun (V) c. Manorin responde ba estudante nia resposta liuhosi fó informasaun adisionál (RA) d. Manorin no estudante involve hamutuk iha atividade hatán hanesan koru (maneira repetisaun) (MEK) e. Estudante kopia materia hosi kuadru (KK) f. Estudante prátika atividade ka abilidade balu mes-mesak (PAM) g. Estudante prátika abilidade ka atividade balu iha grupu ki'ik (PAG) h. Estudante hanorin sira-nia belun ida-ida ka iha grupu ki'ik (EH) i. Manorin fó komentariu pozitivu ba estudante (KP) j. Manorin servisu ho estudante ida pur ida (El) k. Manorin fó pergunta ho reposta limitadu ba estudante ida-idak (PL) I. Manorin fó pergunta maklokek ba estudante ida-idak, laos resposta sim ka lae (PM) m. Teste ka avaliasaun balu la'o hela (AV) n. Estudante ho manorin hananu no halo jogu hamutuk (HJ) o. Manorin lao haleu klase hod! halo interasaun ho estudante (LI) p. Manorin halo sumálriu ba konteudu lisaun loron ohin nian (SK) Atividade laos instrusaun a. lntervalu (I) b. Tempu haan (H) c. Lista prezensa (L) d. Atividade disiplinar (D) e. lnterrupsaun husi atividade seluk (IA) f. Atividade administrativa rutina lao hela (fó ka aruma livru sira, nsst) (AA) g. La defini (LO) Koluna 3 - Lian lnstrusaun Objetivu hosi koluna ne'e mak atu monltoriza/haree utilizasaun Han iha klase laran. Akademikamente, ida ne'e rerere ba "codeswitching." Hatama letra ida ne'ebé indika ba manorin nia lian ne'ebé uza hela durante observasaun direta. Uza kódigu tuir mai. a. Tetun (T) b. Portugés (P) c. Indonesia (I) d. Lian lokál (L) 76 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste Koluna 4 - Estudante laos kona-ba servisu Koluna ne'e nia objetivu prinsipál mak atu monitoriza/haree kona-ba estudante iha klase laran ne'ebé la involve iha prosesu normal instrusaun no aprendizajen. Estudante hirak ne'e kar!k halimar hela ho brinkendu, tai ulun ba meza hodi toba dukur, ko'alia ka halimar hela ho belun seluk, hateke sai bal liur hosi janela hodi hatudu maneira laiha interese. Iha kauza pos(vel barak iha nivel involvimetu ne'e: Lian-estudante la komprende manorin nia lian; estudante ne'ebé akademikamente la hatene. tauk no lakohi atu partisipa; nesesidade espesiál - karik estudante mentalmente ncncik, hamlaha1 violensia, kolen, falta nutrisaun, rona la mós, labele haree, nsst., labele adapta ho eskola sosialmente ka psikolojikamente tanba razaun oi-oin. Problema iha ne'e mak labarik la hetan ajuda no dala-baral la bele aprende. lta hakarak halibur informasaun labarik hira mak sai evidensia tanba sira la partisipa. Hanesan indika iha periódu tempu, haree iha sala laran no konta numeru estudante ne'ebé ita hanoin katak sira la partsipa no hakerek nia númeru. Koluna ne'e nia objetivu daruak mak atu halibur informasaun kona-ba labarik nia prezensa durante loron ne'e. Tanba da\a-ruma labarik sira ne'ebé hela besik eskola, iha tendensia fila ba uma depois de intervalu ka depois de merenda. lta hakarak hatcne fatór ida ne'e nia forsa. Maibe ita sei la halibur inforrnasaun ba ida ne'e kada minutu tolu. !ta sei tau halibur informasaun kona-ba tempu espesifiku tuir mai: 1. Bainhira mak sesaun dahuluk iha klase hahuu iha dadeer (ka loraik). 2. Bainhira mak hahuu kada oras (liña ka oras relcvante halo malahuk). 3. Bainhira mak eskola remata Iha loron nla rohan. Iha tempu espesffiku, koluna haat sei iha informasaun kona-ba total númeru estudante iha klase laran duké númeru kona-ba estudante ne'ebé la partisipa iha atividade klase nian iha loron ne'e. The New Curriculum and the Professional Learning and Mentoring Program 77 Matadalan/Gia ba Kódigu Mapeamentu Atividade iha Klase Laran Koluna 1 - Materia i. (KP) Manorin fó komentariu pozitivu ba estudante j. (EI) Manorin servisu ho estudante ida pur ida  1. Dahuluk mak nu'udar indikador ba hahu no remata k. (PL) Manorin fó pergunta ho reposta instrusaun ka aula. limitadu ba estudante ida-idak a. (H) Hahú materia l. (PM) Manorin fó pergunta maklokek ba estudante b. (R) Remata materia ida-idak1 taos resposta sim ka lae c. (HTM) Hakat tópiku ka matéria m. (AV) Teste ka avaliasaun balu la'o hela d. (K) Kontinua materia n. (HJ) Estudante ho manorln hananu no halo jogu hamutuk oluna ne'e rezerva ba de'it konteudu 2. K o. (LI) Manorin lao haleu klase hodi halo instrusaun ka konteudu aula. interasaun ho estudante a. (T) Dalen Tetun no Literasia p. (SK) Manorin halo sumáriu ba konteudu lisaun b. (P) Dalen Portigés no Literasia loron ohin nian c. (M) Matemátika d. (SN) Siénsia Naturál Atividade laos instrusaun e. (SS) Siénsia Sosiál f. (AK) Arte no Kultura a. (I) lntervalu g. (S) Saúde b. (H) Tempu haan h. (EF) Edukasaun Fízika c. (L) Lista prezensa i. (Re) Relijiaun d. (D) Atividade disiplinar e. (IA) lnterrupsaun husi atividade seluk Ezemplu: f. (AA) Atividade administrativa rutina lao hela: fó ka HT = hahuu sesaun ida iha literasia Tetun aruma livru sira, nsst RP = remata sesaun ida iha dalen Portuges ka literasia g. (LD) La defini ISN = intervalu durante sesaun iha Siensia Natural KSS = kontinua fali sesaun iha Siensia Sosial. Koluna 3 -Lian lnstrusaun Koluna 2 - Atividade sira Hatama letra ida ne'ebé indika ba manorin nia lian ne'ebé uza hela durante observasaun direta. Uza a. (T) Manorin hanorin, esplika ka dernonstra kódigu tuir mai. abiidade foun b. (V) Manorin uza material visual ka objetu hodi a. (T) Tetun ajuda hanorin konseitu ka abilidade foun b. (P) Portugés c. (RA) Manorin responde ba estudante nia resposta c. (I) Indonesia liuhosi fó informasaun adisionál d. (L) Lian lokál d. (MEK) Manorin no estudante involve hamutuk iha atividade hatan hanesan koru (manelra repetisaun) Koluna 4 - Estudante laos kona ba e. (KK) Estudante kopia material hosi kuadru servisu f. (PAM) Estudante prátika atividade ka abilidade balu mes-mesak Estudante iha klase laran ne'ebé la involve (la partisipa) g. (PAG) Estudante prátika abilidade ka iha prosesu normal instrusaun no aprendizajen. atividade balu iha grupu ki'ik ida ka iha h. (EH) Estudante hanorin sira-nia belun ida­ grupu ki'ik 78 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste Gia: Kompleta lnstrumentu D Estimasaun "Sasukat ba Manorin" Item 1 - Manorin iha klase laran antes aula hahuu Kontextu: Iha nível baze iha edukasaun primária, bai-bain husu katak manorin tenke iha klase laran antes aula hahuu. Iha razaun balu ba ida ne'e. Labarik ki'ik sira presiza supervizaun atu prevene baku-malu, konflitu verbál no hahalok negativu sira seluk. Iha nasaun barak, normalmente sira-nia klase sira xave wainhira oras eskola remata atu nune'e manorin bele loke wainhira eskola tama hodl husik labarik sira tama. Aleinde ne'e, iha mós objetivu edukasional; karik iha preparasaun seluk hanesan hadia kadeira-meza sira, tau sasan sira ka ekipamentu ba aula sira, organiza livru sira, aruma klase laran. Nune'e mós, ho manorin nia prezensa antes eskola hahuu fó sinal ba labarik katak eskola ne'e importante, manorin iha interese ba labarik, no iha komitmentu atu hanorin iha loron ne'e. Hili opsaun sira: Loos (5) - Karik manorin to'o minutu 15 antes aula hahuu no halo preparasaun ho ativu. Karik liu hosi minutu 15 antes maibé lá involve an iha preparasaun ba loron ne'e (iz. Fuma, ko'alia lá para ho manorin seluk, tuur de'it iha eskritóriu, la' o ba mai laiha objetivu, nsst.), hili pontu avaliasaun tuirmai. To'o uluk liu de'it labarik (3)- Karik manorin to'o lá liu hosi minutu 15 antes aula hahuu KA manorin laiha atensaun espesifiku atu halo preparasaun ruma ka tau matan ba labarik sira ne'ebé to'o ona. To'o tarde uitoan (1) - Karik manorin to'o eskola tarde minutu 10 DEPOISDE aula hahuu. To'o tarde liu (0) - Karik manorin to'o eskola tarde liu minutu 10 DEPOISDE aula hahuu. Karik manorin to'o eskola tarde minutu 30 DEPOISDE aula hahuu, aumenta fitun rua (**) iha liña. Depois de hill tiha pontu avaliasaun, prienxe númeru iha parénteze iho liña mamuk iho liña ikus pontu avaliasaun. Item 2 - Manorin iha planu lisaun detallu. Kontextu: Iha programa formasaun profesór iha mundu tomak sempre fó atensaun ba importansia tenke iha planu lisaun wainhira hakarak sai manorin di'ak. Maski esplikasaun la hanesan hosi programa formasaun ida ba programa tormasaun seluk kona-na ba impórtansia hosi prosedura ne'e, esplikasaun hotu ba de'it ideia katak bele halo planu lisaun depoisde aula wainhira manorin iha tempu atu hanoin no halo planu ho kuidadu. Ne'e duni manorin bele mai iha klase laran atu hanoin kona-ba atividade ba loron tuir mai ho detallu no prontu fó atensaun ba hanorin. Planu lisaun di'ak no kompletu ne'e oinsá? Planu liasaun di'ak mak hanesan tuir mai: spesifiku kona-ba saida mak atu hanorin no atinji iha kada disiplina. 1. E Orariu ne'ebé hatudu bainhira mak hanorin kada disiplina no tempu hira ba kada disiplina. 2. Ba kada disiplina, fó tempu ba komponente ka atividade tuir rnai: 3. a. Revizaun ba materia ne'ebé hanorin ona (Iha loron ida antes) b. Atu hanorin ka hatudu materia foun c. Atu fó ba estudante atu pratika konteudu foun d. Atu haree estudante nia kunesimentu kona-ba materia foun e. Ba revizaun, sumáriu, no haforsa The New Curriculum and the Professional Learning and Mentoring Program 79 4. tividade ne'ebé rnanorin ho estudante tenke pratika atu hatene konteudu foun. A 5. Atividade atu sukat kuñesimentu 6. Atividade atu revlza no haforsa 7. Karik iha, servisu uma no klase. Hili opsaun slra ne'e: Loos (5) -karik manorin nia planu liasaun ohin loron nian hanesan ho deskrisaun iha leten. Manorin halo tuir planu lisaun. Jeralmente prepara duni maibe la kompletu (4) - karik pontu balu iha okos ne'e loos: Manorin iha planu lisaun rnaibé la kompletu. – Manorin halo tuir planu lisaun rum a iha loron tomak, maibé la hakerek iha planu lisaun loron ne'e nlan. – Parte balu hosi planu lisan kompletu maibé balu la kompletu ka la dezenvolve. – Manorin iha planu lisaun di'ak maibé la halo tuir. – Preparasaun lá tuir planu (3) - karik deklarasaun ruma okos ne'e loos. Manorln halo tuir planu lisaun ida iha loron ne'e maibé la hakerek. – Manorin hakerek planu ruma lalais wainhira estudante halo servisu iha sira-nia meza leten ka iha – intervalu. Preparasaun laiha flu (1) - manorin laiha planu lisaun no halo planu durante nia hanorin hela. Dalabarak manorin soi hos/tópiku, laiha preparasaun, laiha ligasaun (0) - Hili opsaun ne'e karik sesaun ohin la organlza, laiha ligasaun no konfuzaun ka (2) manorin lakon tempu barak ba tópiku ka disiplina laos ba kurrikulu no laiha konsistensia ho materia ne'ebé hanorin antes, ka (3) manorin la uza tempu ho di'ak: lakon tempu liu ba disiplina ida ka rua de'it hodi lakon disipllna sira seluk. Depois de hili tiha pontu avaliasaun, prienxe númeru Iha iha lina mamuk. Item 3 -Lista prezensa atual no kompletu. Kontextu: Lista prezensa di'ak hatudu katak manorin Iha atensaun ba detalle sira no Iha hakarak atu tau matan ba sira­ nia servisu. Lista prezensa tulun manorin sira identifika se mak iha problema tanba razaun ruma no manorin bele tulun atu rezolve. Iha lista di'ak hatudu katak manorin Iha komitmentu atu hadia kualidade edukasaun. Hili opsaun sira ne'e Loos (5) - karik lista prezensa kompletu no atual. Laduun (4) - karik lista prezensa kompletu maibe ba de'it loron ida ka loron rua liu bá, ka karik menus hosi loron rua laiha lista prezensa. Falta lista signifikate (3) - karik loron barak laiha lista prezensa ka loron 3 ka liu ona laiha lista prezensa Lá adekuadu (2) - karik lista prezensa la kompletu liu ka hanesan "inventa de'it" atu hatudu hanesan kompletu Laiha (0) - karik laiha ka la hetan lista prezensa. Depoisde hili tiha pontu avaliasaun, prienxe númeru iha iha liña mamuk. 80 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste Item 4 - Manorin livre iha klase no iha nia knaar hanesan manorin. Kontextu: Manorin organiza ambiente ba klase no ba estudante. Karik manorin nervozu, hirus, ka tensaun, ka manorin hatudu hahalok la hanesan bai-bain, estudante sei hatudu hahalok tauk no dalaruma la kontraldu. Manorin diák komunika sentidu de konfiansa, bai-bain, domin, no hakarak atu hanorin. Estudante sira, maski ida ne'ebé ki'ik liu sei sente no hatudu hahalok hanesan manorin. Evidensia katak manorin livre iha klase laran: – Manorin la nervozu ka tauk; – Manorin hamnasa ka kumprimenta estudante winhira sira tama klase laran; – Manorin komunika ho setudante ho livre no laiha ameasa ka violensia barak; – Manorin prontu atu hanorin hahu lisaun wainhira tempu toó ona atu hanorin; – Manorin prontu ba saida de'it mak akontese; – Manorin kontente ho ninia servisu nu'udar manorin; – Manorin iha planu lisaun ne'ebé di'ak - hakerek no mentál - halo tuir; – Manorin jere estudante ho di'ak; – Estudante respeita no hatán ba manorin ho prontidaun. Hili opsaun sira ne'e Loos (5) - Halo tuir hotu lista iha leten. Dalaruma loos (4) - Dalaruma manorin halo tuir lista iha leten. La hatene (3) - Manorin halo tuir mais ao menus metade hosi lista iha leten no seluk la halo tuir hotu. Dalaruma la loos (2) - Karekterístika iha lista iha leten barak liu mak LA LOOS iha manorin ne'e. La loos liu (0) - Manorin ne'e la livre liu no karik hatudu de'it karakterístikca livre balu iha klase laran. Depoisde hili tiha pontu avaliasaun, prienxe númeru iha iha lina mamuk. Item 5 - Manorin kontrola duni klase. Kontextu: Prosesu hanorin no aprende di'ak liu mak ha ambiente favorável ba hanorin no aprende. Ne'e katak jere klase ho di'ak ho disturba ituan ne'ebé preven estudante ni akonsentrasaun atu aprende. Disturbu mosu iha forma oi-oin; barullu, estudante nakar, hosi liur, koa'lia maka'as, labarik la'o bá mai, manorin nia asaun derrepente, nsst. Fraze, "kontrola klase" la signifika katak manorin intimida ka hatauk estudante sira atu koa'lia ka halo buat ruma. Maibé, ne'e katak manorin bele mantein kondisaun di'ak ba hanorin no aprende. Hili opsaun sira ne'e Loos (5): karik manorin sempre ka organiza bebeik etudante hodi konsentra ba aprende. Dalaruma loos (4): karik iha disturbu rua ka tolu ne'ebé halo estudante laiha atensaun ba objetivu aprendizajen ohin loron nian ka lisaun nian. La hatene (3): karik iha disturbu ida ne'ebé halo estudante laiha atensaun ba objetivu aprendizajen ohin loron nian ka lisaun nian. Dalaruma la loos (3): karik iha akontesimentu ne'ebé disturbu estudante no halo manorin tenke bolu bebeik estudante nia antensaun duké konsentra ba hanorin. La loos liu (0): karik iha disturbu barak no manorin la konsege jere ka kontrola klase no rezulta barullu no disturbu barak. The New Curriculum and the Professional Learning and Mentoring Program 81 Depoisde hili tiha pontu avaliasaun, prienxe númeru iha iha liña mamuk. Item 6 - Manorin haksolok, komunika livre ho estudante, no involve estudante ho másimu iha saida de'it mak atu hanorin no atu aprende. Kontextu: "Enerjia" mak dinámika pozitivu iha ambiente aprendizajen. Termu "enerjia" bele ferere ba funsaun klase: iha interese, atividade, haksolok, antisipasaun, susesu, iha korajen. Importante ba manorin atu lori enerjia ba iha klase laran no bele halo ne'e iha maneira oi-oin: esperiensia iha interese no haksolok kreatividade, domin ba estudante no iha korajen maka'as, rekunese estudante nia aproveitamentu. Laos estudante hotu-hotu bele hatán hanesan ba manorin nia attitude sira ne'e. Estudante balu responde di'ak ba hahalok haksolok no domin. Balu responde ba hahalok haforsa ka rekuñesimentu aproveitamentu. Balu motiva ho ideia foun no kreatividade. Manorin di'ak identifika labarik ida-idak nia resposta no responde ho loloos. Kestaun boot iha ne'e mak oinsá manorin kria ambiente dinámika pozitivu ba aprendizajen iha klase laran no estudante responde ho pozitivu ba ambiente ne'e. Hili opsaun sira ne'e Loos (5): sentidu iha interese no haksolok ka antisipasaun ba eskola evidente duni iha estudante nia leet. Manorin haree estudante ho di'ak no sira bele responde. Ne'e hatudu wainhira manorin bele kria ambiente pozitivu. Dalaruma loos (4) Iha tempu balu lta-Boot bele sente enerjia dinamika pozitivu iha klase laran maibe laos bebeik. La hatene (3): susar atu hateten karik karakteristika hirak ne'e ezisti regulár maski dalaruma lta-Boot haree sinal balu hosi enerjia sira. Dalaruma la loos (2): manorin nia "enerjia tuun'' ka iha interese ka ksolok uituan de'it ba klase la ran. Aleinde ne'e, hili opsaun ida ne'e karik lta-Boot haree estudante barak baruk ka laiha interese. La loos liu (O): lta-Boot haree dinamika "kontrariu" Iha klase laran. Estudante barak liu baruk no laiha interese ka manorian laiha inspirasaun no laiha korajen atu enkoraja estudante sira. Depoisde hili tiha pontu avaliasaun, prienxe númeru iha iha liña mamuk. Item 7 - Manorin uza maneira efetlvu hodi ko'alia no komunika ho estudante. Kontextu: Lian no komunlkasaun irnportante tebes iha edukasaun. Karik estudante la rona ka komprende manorin Iha klase laran, ne'e katak efetividade mós tuun. Iha razaun barak tanba sa mak estudante slra susar atu rona manorln: manorin ko'alia neneik liu, barullu rnaka'as iha klase laran, labarik balu susar atu rona, manorin murmura ka laiha sentidu liu. Konserteza, karik manorin ko'alia iha lian ne'ebé estudante la komprende ka komprende uitoan de'it, ne'e kria inefetividade boot ida tan iha instrusaun durante aula laran. Manorin tenke ko'alia maka'as atu estudante bele rona maibé laos hakilar. Manorin tenke hatene katak estudante balu presiza repete esplikasaun tanba sira la ko'alia ka uza lian instrusaun ho diak. 82 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste Hili opsaun sira ne'e Loos (5): Manorin komunika ho efetivu. Nia lian maka'as natón, maibé laos hakilar. Buat ne'ebe nia hateten klaru no Iha ligasaun. Dalaruma iha repetisaun atu asegura estudante rona saida mak nla hateten. Manorin uza lian ne'ebé estudante hatene di'ak. Dalaruma loos (4): Buat ne'ebé hateten iha leten loos maibe dalaruma estudante la komprende saida mak manorin hateten. La hatene (3): Liuhosi rona ba manoirin no haree ba estudante Ita-Boot bele haree iha komuniksaun klaru ka lae, susar ba lta·Boot atu hateten. Ita-Boot haree bebeik tempu ne' ebe estudante la rona ka la komprende saida mak manorin koko atu komunika. Dalaruma la loos (2): Ita-Boot haree ho klaru tempu ne'ebé manorin la komunika ho di'ak tanba barullu Iha klase laran, iha liur, estudante la komprende lian1 ka manorin la ko'alia maka'as natón ka la moos atu estudante bele komprende. La loos liu (O): Komunikasaun efetivu sai problema boot Iha klase laran. Dala barak estudante la rona ka la komprende saida mak manorln hateten. Depoisde hili tiha pontu avaliasaun, prienxe númeru iha iha liña mamuk.. Item 8 - Manorin Iha komprensaun kona·ba "figura jerál" ba tinan Ida ne'e. Kontextu: Organiza instrusaun klaru liu-llu iha nivel baze atu bele hatene objetivu aprendizajen iha tinan ida laran. Loos duni katak objetivu sira iha ona kurrikulu ka iha livru sira, maibé manorin mós presiza hatene ho di'ak abilidade ka kuñesimentu saida mak estudante presiza hatene iha tinan ida nia laran. Abilidade ka kuñesimentu sira sei ajuda manorin atu hatene ka hasoru situasaun ne'ebé mak estudante la hatene no atu asegura katak estudante iha duni abilidade báziku. Karik manorin la hatene figura jerál kona-ba abilidade saida mak estudante ida-idak tenke iha durante tinan ne'e, fásil atu husik hela estudante iha kotuk no la konsentra ba saida mak presiza atu hatene, liu-liu hirak ne'ebe difisil liu. Manorin di'ak iha figura klaru kona-ba saida mak estudante tenke aprende iha tinan ida ne'e nia laran no esforsu atu bele atinji objetivu ba estudante sira hotu iha klase laran. Hill opsaun sira ne'e Loos (5): iha manorin nia esplikasaun kona-ba estudante nia progresu iha tlnan ida ne'e, manorin esplika klaru no di'ak kona-ba salda mak estudante tenke aprende no atinji iha tinan ida ne'e. Dalaruma loos (4): manorin iha ideia di'ak natón de'it kona-ba saida mak atu atinji iha tinan ida ne'e inklui llsaun ida ne'e hanesan parte hosi figura ka planu jerál ne'e. La hatene (3): lta-Boot la haree katak manorin iha ideia klaru ba figura jerál ne'e ka lae, maibe haree ba razaun balu hatudu katak manorin hatene, Dalaruma la loos (2): Manorin foka ba estudante nia situasaun agora no lá liga aprendizajen agora ba objetivu jerál tinan Ida nian. La loos liu (O): Manorin laiha kuñesimentu kona-ba figura jerál no hatene de'lt situasaun agora. Depoisde hili tiha pontu avaliasaun, prienxe númeru iha iha liña mamuk. The New Curriculum and the Professional Learning and Mentoring Program 83 Item 9 - Manorin fó instrusaun tarjetu de'it ba estudante individual ka grupu ne'ebé presiza. Kontextu: Aspeitu fundamental ida iha edukasaun mak laos estudante hotu-hotu aprende ho tempu hanesan. Manorin ho esperiensia limitadu bai-bain falla iha ábltu ida katak hanorin de'it estudante ne'ebé hatene ka iha interese. Ne'e intensaun la loos. Ne'e akontese tanba manorin hakarak haree evidensla katak aprendizajen akontese duni. Manorin mós hanoin katak labarik tenke aprende materia ne'ebé hanorin tiha ona. Situasaun hanesan ne'e akontese iha nasaun sei dezenvolve ne'ebé manorln "aprende" uza forma hanesan grupu nia resposta nu'udar lnterasaun manorin-estudante (refere ba hatan hakilar hamutuk, hatán hanesan koru, hatán Iha grupu, ka hatan tanba dekór de'it) Modelu interasaun ne'e hanesan: Manorin husu: "2 + 5 hamutuk hira?" Estudante matenek hatan: "7" Estudante sira seluk hat.in tuir: "7" Manorin dehan: "Di'ak, ema hotu dehan 7." Manorin husu hotu-hotu hamutuk iha llan maka'as: "7" Manorin interpreta hatan hanesan koru katak hotu-hotu hatene oinsa sura 2+5 hetan 7. Hakfodak boot liu tan mak manorin uza modelu ne'e bebeik, dalaruma durante ti nan ida tornak, no la hatene katak labarik barak mak la aprende saida mak 2+5! Manorin di'ak hatene katak deklarasaun públiku ba resposta (7 iha kazu ne'e) signifika katak estudante bele rona lalais no repete saida mak estudante seluk hateten. Sira tenke servisu hamutuk ka husu estudante ida-idak hodi haree sira bele aumenta númeru rua hamutuk ka lae. lnstrusaun tárjetu katak manorin presiza haree estudante ida-idak nia kuñesimentu no fó ajuda ba estudante ida-idak karik sira la aprende ho di'ak. Iha realidade, iha peskiza bazeia ba evidensia hatudu katak depende ba metodolojia hanorin hanesan ne'e bai· bain rezulta estudante pursentu 10-20 de'it (liu-liu estudante matenek) mak aprende materia ne'ebé "hanorin tiha ona." Manorin di'ak fó atensaun adisional ba estudante pursentu 80-90 ne'ebe la aprende (instrusaun tárjetu). Hili opsaun sira ne'e Loos (5): Manorin halo teste regulár ba estudante ida-idak nia kunesimentu ba materia ne'ebé fó ona NO fó instrusaun tárjetu adisional wainhira iha estudante balu sedauk aprende. Dalaruma loos (4): Manorin dalaruma fó instrusaun tárjetu maibé laos beibeik. La hatene (3): lta-Boot haree iha esforsu ba instrusaun tárjetu maibé laiha esforsu beibeik hodi hatene estudante nia kuñesimentu ka la fó instrusaun adisional. Dalaruma la loos (2): lta-Boot ladún haree teste ba estudante ida-idak ka laiha instrusaun tárjetu. La loos liu (0): lta-Boot la haree liu teste ba estudante ida-idak ka laiha instrusaun tárjetu. Depoisde hili tiha pontu avaliasaun, prienxe númeru iha iha liña mamuk. 84 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste Item 10 - Manorin Iha kuñesimentu ba material ne'ebé nia hanorin. Kontextu: Barak hanoin katak manorin Iha nivel ensinu báziku iha kuñesimentu tomak ba konteudu ne'ebé nia hanorin. Iha kontextu balu, karik ne'e loos; iha seluk konserteza la loos. Iha situasaun balu tanba sa manorin iha kuñesimentu limitadu. Nee inklui (1) Manorin la hatene di'ak L2 wainhira L2 mak lian instrusaun, (2) Manorin hanorin uza lian ne'ebé sei dezenvolve hela (3) Manorln la forte iha area sira (lz. Matematika), (4) Manorin la hanorin ho klean Evidensia tanba falta kuñesimentu depende ba situasaun. Karik manorin hanorin iha L2, dalaruma manorin Iha frakeza Iha gramátika, sintátika no vokabulariu. Dalabarak hakerek liafuan iha kuadru la loos. Karik manorin hanorin uza lian ne'ebé mak sei dezenvolve hela, bai-bain ita bele haree sala ka hakerek la hanesan. Karik manorin fraku Iha disiplina hanesan matematika ka siensia, manorin bele fó instrusaun sala ka koko atu subar sala liuhosi fó esplikasaun ba konteudu. Hili opsaun sira ne'e Loos (5): lta-Boot haree laiha falla iha lian no la haree manorin nia servisu ruma ne'ebé hatudu katak nia Iha kuñesimentu llmitadu ba materia. Dalaruma loos (4): lta-Boot haree iha sala uituan de'it iha lian ka iha kuñesimentu ba konteudu. La hatene (3): lta-Boot haree iha sala barak ka iha kuñesimentu limitadu ba materia maibé manorin hanoin katak ida ne'e bai-bain de'it. Dalarumo la loos (2): lta-Boot haree iha evidensia manorin nia kuñesimentu limitadu ba materia. La loos liu (0): Tuir lta-Boot nia opiniaun, manorin iha kuñesimentu llmitadu liu no nia la bele hanorln konteudu kurrikulu ho efetlvu. Depoisde hili tiha pontu avaliasaun, prienxe númeru iha iha liña mamuk. Item 11 - Manorin efetivu dunf hanorin konteudu foun. Kontextu: Tuir deflnisaun manorin nia servisu prinsipál mak atu hanorln kuñesimentu no abilidade foun. lta hatene ida ne'e bele halo ho maneira oi-oin ho nível susesu oi-oin. Modelu hanorin ne'ebé susesu liu inklui komponente tuir mai: (a) haree ba estudante nia kuñesimentu, abilidade no esperiensia uluk nian hodi halo sai referensia ba kuñesimentu no abilidade foun; (b) esplika1 hatudu, aprezenta konteudu foun iha maneira loloos atu estudante bele komprende; (c) fó oportunidade ba pratika no aplika kuñesimentu no abilidade foun atu estudante bele hahuu uza ka hatene kuñesimentu ka abilidade ne'e; no (d) reforsa kuñesimentu no abilidade foun ne'e liuhosl revizaun, teste, aplika iha maneira faun, haluan kuñesimentu ne'e iha esperiensia area foun. Iha nasaun barak, manorin foka maka'as liu iha etapa (b) introdus materia foun hodi la fó atensaun ka fó atensaun uituan de'it ba etapa seluk. Objetivu hosi avaliasau ne'e mak atu haree oinsa mak manorin uza modelu kompletu hodi introdus materia foun. Hili opsaun sira ne'e Loos (5): manorin uza aproximasaun oi-oin no klean hodi lntrodus konteudu foun hanesan deskereve iha leten. Dalaruma loos (4): dalaruma manorin uza modelu kompletu hodi introdus modelu foun ka hanesan deskreve iha leten. La hatene (3): manorin fó atensaun uituan de'it ba elementu balu hosi siklu aprendizajen kompletu ka hanesan deskreve iha leten. The New Curriculum and the Professional Learning and Mentoring Program 85 Dalaruma la loos (2): manorin husik liu elementu Ida ka liu hosi siklu aprendizajen kompletu. La loos liu (0): manorin uza etapa ida de'lt hosi siklu aprendizajen kompletu no laiha atensaun ba siklu seluk. Depoisde hili tiha pontu avaliasaun, prienxe númeru iha iha liña mamuk. Item 12 - Manorin fó oportunidade barak ba estudante atu pratika abilidade foun. Kontextu: Iha observasaun haree katak aplikasaun no prátlka mak komponente sira fraku liu iha ensinu báziku (no ensinu seluk). Ne'e akontese tanba raiaun barak; (a) manorin lakohi fó tempu no esforsu atu prepara material no atividade sira ba prátika no aplikasaun; (b) material sira ba prátika no aplikasaun laiha; (c) manorin troka atividade hanesan resposta hanesan koru, repete resposta ka kopia konteudu hosi kuadru; (d) manorin la komprende funsaun prátika no aplikasau iha siklu aprendizajen. Iha maneira barak estudante bele pratika no aplika kuñesimentu no abilidade faun. Ne'e inklui halo ezersisiu iha kadernu laran, servisu uma nian, servisu grupu, fó instrusaun ba pár no aprende, aplika projetu, nsst. Atividade ne'ebé de'it mak presiza kuñesimentu no abilidade foun, presiza prátika atu kuñesimentula lakon. Hili opsaun sira ne'e Loos (5): manorin fó bebeik oportunldade barak ba estudante ida-idak prátlka no aplika abilidade no kuñesimentu foun. Dalaruma loos (4): manorin fó oportunidade ba estudante atu prátika abilidade no kuñesimentu foun maibé laos bebeik. La hatene (3): oportunidade prátika laos sai toman ka manorin depende ba resposta hanesan koru ka repetisaun resposta hanesan estudante nia maneira prlnsipál. Dalaruma la loos (2): manorin introdus bebeik materia foun maibe lalha tempu ba apllka ka prátika. Hili opsaun ne'e karik manorin UZA DE'IT resposta hanesan koru ka repetisaun resposta hanesan mekanizmu ba estudante nla resposta ka evidensia hosl aprendlzajen. La loos liu (0): manorin hanorin hanesan diskursu, koa'lia, hakerek iha kuadru, ka hatudu konteudu foun la ho estudante nia resposta ka aplikasaun. Depoisde hili tiha pontu avaliasaun, prienxe númeru iha iha liña mamuk. Item 13 - Manorin koko estudante nia kuñesimentu ba materia liuhosi revizaun, teste, pergunta maklokek, projetu Individuál; aprezentasaun, nsst. Kontextu: Manorin di'ak haree fali konteudu ne'ebe estudante sira hotu aprende ona. (la realístiku karik ejize estudante hotu-hotu tenke domina hotu konteudu faun ne'ebé fo tiha ona tanba abilidade no kapasidade Iha klase la ran la hanesan inklui estudante balu ho nesesidade espesial no seluk ne'ebe lakohi atu aprende). Iha nasaun barak, bai-bain bele hetan manorin balu nunka haree fali estudante nia aproveitamentu. Iha razaun oi-oin; (a) manorin la hatene oinsá atu halo, (b) manorin lakohi atu hamoe estudante; (c) manorin la sente Iha responsabilidade ba estudante nia kuñesimentu-manorin barak hateten, "manorin hanorin de'it, depende ba estudante atu aprende," (d) manorin nia estudante baralk liu, tempu la to'o atu prova estudante ida pur ida; (e) manorin hanoin katak resposta hanesan koru ka repetisaun resposta mak buat Ida ne'ebé presiza ba estudante ida-idak nia aprendizajen. 86 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste Manorin di'ak buka maneira oinsa bele prova estudante ida-idak liuhosi husu pergunta, haree individuál nia hahalaok ho kuidadu, forma grupu ki'ik hodi haree ida-idak nia kuñesementu, fó teste, haruka estudante servisu uza livru ezersisiu, nsst. Buat ne'ebé presiza duni iha aspeitu manorin nia attitude mak nesesldade atu manorin fó instrusaun adisional ka atividade aprendizajen adisionál ba estudante sira ne'ebé tanba razaun balu aprende neneik ka sira ne'ebé la komprende buat ne'ebé hanorin ona kona-ba konteudu foun. Ida ne'e konsitu kona-ba instrusaun tárjetu iha item 9. Hili opsaun sira ne'e: Loos (5): karik lta-Boot haree manorin prova estudante ida-idak nia kuñesementu ho regular. Dalaruma loos (4): karik dalaruma manorin halo prova maibé laos bebeik. La hatene (3): Ita-Boot susar atu hateten karik manorin prova duni bainhira haree liuhosi nia knaar iha klase laran. idak Dalaruma la loos {2). hosi lta-Boot nia observasaun dalaruma lta-Boot haree manorin la prova estudante ida­ nia kuñesementu. La loos liu (0): karik lta-Boot nunka haree manorin fó prova. Depoisde hili tiha pontu avaliasaun, prienxe númeru iha iha liña mamuk. Item 14 - Manorin iha komprensaun di'ak kona-ba labarik ida-ldak nia progresu no bele ldentifika etapa tuir estudante nia nesesidade. Kontextu: Bai-bain iha edukasaun ema dehan, "Manorin di'ak hatene nia estudante." Loos duni, bele interpreta deklarasaun ida-ne'e iha maneira oi-oin. Ne'e bele signifika, izemplu, manorin presiza hatene estudante ida­ idak nia naran no hatene sira pesoalmente. Bele mós signifika katak manorin presiza hatene estudante ida· idak nia esperiensia edukasaun antes no oinsa mak sira halo ona. Maibé interpretasaun komun liu mak manorin presiza iha kuñesementu di'ak ba estudante lda-idak nia progresu ka lalaok iha klase laran liu-liu kona-ba kuñesementu konteudu foun. Iha ne'e bele lnklui ideia kona-ba manorin nia kuñesementu ba estudante ida-ldak nia aprendizajen no presiza ajusta sira-nia instrusaun ho loloos atu haluan posibilidade susesu ba estudante lda-idak. Bele realiza deklarasaun dehan katak, "Manorin di'ak hatene nla estudante" bainhira númeru estudante iha klase laran ki'ik-25 ka tuun tan. Iha nasaun slra ho númeru estudante barak iha klase laran, susar atu atinji. susar liután bainhira oras eskola badak ka bainhlra manorin iha servisu seluk depoisde eskola (komún Iha nasaun balu). Maibé, maski Iha sala ho númeru estudante uitoan, Iha posibilidade manorin foka liu ba introdusaun materia foun duke foka ba estudante nia aproveitamentu. Resposta baze ba item ne'e se mai hosi intervista ho manorin depoisde aula duké hosi observasaun aula (maski resposta balu bele mai hosi observasaun aula). Hili resposta sira ne'e Loos (5): Karik manorln Iha kuñesementu di'ak ba estudante ida-idak nla progresu akadémika iha klase laran (ka pelumenus ba estudante sira ne'ebé mai eskola ho regular). Dalaruma loos (4): Karik manorin iha komprensaun di'ak ba estudante balu nia progresu akadémika iha klase laran maibé laos ba estudante hotu-hotu. No mós karik manorin iha kuñesementu jeral de'it ba estudante ida-­ idak nia servisu no oinsa estudante iha eskola. The New Curriculum and the Professional Learning and Mentoring Program 87 La hatene (3): Karik Ita-Boot la bele haree hosi intervista ho manorin kona-ba sira-nia hatene estudante nia aproveitamentu ka lae, ka karlk Ita-Boot hanoin manorin naran siik de'it ka inventa de'it buatruma kona-ba estudante ida-idak nia aproveitamentu. Dalaruma la loos (2): Karik manorin hatene estudante balu nia aproveiamentu no iha ideia uituan de'it ka la hatene kona-ba sira seluk. La loos liu (0): Karik manorin la hatene estudante slra hotu no foka de'it ba hanorin konteudu faun. Depoisde hili tiha pontu avaliasaun, prienxe númeru iha iha liña mamuk. Item 15 - Manorin komprende estudante no sira-nia hahalok no bele rezolve problema sira ho loloos, justu no pozitivu. Kontextu: Manorin uiton de'it mak iha treinamentu ka iha esperiensla atu bele komprende estudante nia hahalok Iha klase laran, liu-liu estudante ne'ebé iha problema. Sira ne'ebé iha ona treinamentu ka esperiensia mos sei iha balu la komprende estudante nia hahalok. Iha tempu hanesan, klaru katak kontrola estudante fízikamente de'it hanesan baku sira, haruka sira hamriik iha kantu, ka hakilar slra, hatudu katak laiha komprensaun ba motivasauh ka saida mak halo estudante halo sira-nia hahalok. Manorin presiza aprende no aplika estratejia atu rezolve estudante ho problema tuir estudante nla nesesidade no karakterlstika. lzemplu, estudante balu presiza de'it sinál ruma ka hateke de'it ba sira, atu bele kurriji sira-nia hahalok ladiak. Balu presiza ko'alia ho lian normál hodi fó hanoin sira. Iha estudante balu mak susar atu manorin bele komprende no haree sira no teste ida ida ne'e atu sukat manorin nia abilidade. Hili opsaun sira ne'e Loos (5): Karik durante observasaun haree manorin komprende prinsipiu ne'e no halo esforsu hodi fó asaun koresaun tuir estudante lda-ldak nia nesesidade. Dalaruma loos (4): Iha maneira balu manorin uza hasoru estudante ho problema maibé nia gosta llu uza estratejia ida ka rua hanesan hakilar ka haruka estudante sai hosi klase laran. La hatene (3): karik lta-Boot la haree manorin iha komprensaun ba konseitu ne'e ka la haree manelra ruma oinsá haree estudante ho problema. Dalaruma la loos (2): karlk manorin uza metodolojia dlslpllna hanesan de'it ba estudante hotu-hotu la haree ba violasaun ka estudante nia emosaun. La loos liu (O): karik manorin haree ba estudante nia hahalok ladiak hanesan de'it hodi la haree ba razaun hosi estudante nia hahalok ka estudante nla nesesidade. Depoisde hili tiha pontu avaliasaun, prienxe númeru iha iha liña mamuk. 88 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste Item 16 - Manorin nia klase la'o ho di'ak no produtivu no disturba uitoan de'it. Kontextu: Parte ida ne'e hanesan kombinasaun hosi item hotu iha leten. Bele haree hanesan avaliasaun jerál Ida ba manorin nia efetividade. Hili opsaun sira: Loos (5): karik lta-Boot haree no deskreve prosesu hotu iha klase laran ho di'ak. Dalaruma loos (4): karik klase la'o ho di;ak no produtivu durante tempu tomak. La hatene (3): karik lta-Boot laiha konfiansa katak lta-Boot la bele fó julgamentu hotu ba item ka parte ne'e. Dalaruma la loos (2): karik iha problema barak Iha klase la ran, disturba bebeik, ka karik manorin la bele halo klase la'o ho di'ak no produtivu. La loos liu (0): karik klase la la'o ho diak no haree hanesan aprendizajen laiha. Depoisde hili tiha pontu avaliasaun, prienxe númeru iha iha liña mamuk. The New Curriculum and the Professional Learning and Mentoring Program 89 Instrumentu A Utilizasaun Kurikulu Nasional (KN) Siklu 1: Klase 1 and 2 Antes atu halo observasaun klase: aree saida rnak hakerek iha KN ba loron 5 antes loron nebee mak ita atu vizlta. – H Haree di-diak saida mak hakerek iha KN ba loron nebee mak ita atu halo observasaun. – Hakerek rezumu ba konteudu lisaun sira nebee mak mestri/a atu hanorin iha loron nee. (lta sel presiza – informasaun nee, se lae susar atu hataan pergunta tulr mai kona ba Kurikulu Nasional.) Husu atu haree mestri/a nia planu lisaun, no analiza lalais. 1. Mestri/a iha planu lisaun ka lae? Sin (3) Lae (0) _____ 2. Planu lisaun nee hatudu katak mestri/a uza KN ka lae? Sin, klaru (5) Dala ruma (2) Lae (0) _____ Halo observasaun iha klase laran. Bainhira sai hosi eskola, hataan pergunta tuir mai: 3. Iha kopia KN iha klase laran ka? Sin (1) Lae (0) _____ 4. Mestri/a hanorin uza KN ka lae? Sempre (5) Dala Barak (4) Dala Ruma (2) ladún (1) Nunka (0) _____ 5. Ba área konteúdu ida-ida, planu lisaun tuir pontu espesifiku kalisaun ne'ebé mak hakerek iha KN ka lae? Sempre (5) Dala barak (3) ladún (1) Nunka (0) _____ 6. Alunus sira halao atiividade balurak nebee hakerek iha KN ka? Sempre (5) Dala barak (3) ladún (1) Nunka (0) _____ Alunus sira hatudu katak sira aprende materia nebee mak tama iha KN ba loron t antes iha halao observasaun klase ka? Sempre (5) Dala barak (3) ladún (1) Nunka (0) _____ 7. Lisaun ohin loron tuir KN iha aspetu tuir mai ka lae? a. Tuir objetivu aprendizajen prinsipál? Sin (3) Dalaruma (1) Lae (0) _____ b. Hanorin materia foun: Di'ak (5) Maizumenus (3) Ladún di'ak (1) Lae (0) _____ c. Atividade aprendizajen nian ohin loron foti hosi KN ka lae? Maioria (5) Balu (3) Uitoan (1) Lae (0) _____ 8. Ita-nia hanoin kona-ba manorin uza KN iha klase laran. Di'ak loos (5) Di'ak (4) Maizumenus (2) La uza KN (0) _____ 90 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste Intervista 9. Husu manorin, nia lee seksaun KN kona ba lisaun ohin loron nian ka lae? Nia estuda matéria ne'e ka lae? a. Sin, manorin estuda didi'ak lisaun KN ba ohin loron (5) b. Sin, manorin matéria ne'e, maibé la estuda ho didi'ak (3) c. Manorin lee liu duni matéria ne'e, maibé la domina (2) d. Manorin la lee matéria ne'e (0) _____ 10. Saida mak ita aprende hanorin KN nian? Hili resposta ne'ebé deskreve manorin nia resposta di'ak liu: a. Manorin fó resposta ne'ebé klaru, ho detallu (5) b. Manorin temi buat 3 ka ne'ebé liga ba KN, maibé resposta la detallu (3) c. Manorin nia resposta hatudu katak nia ladún domina KN (1) d. Manorin nia resposta hatudu katak nia la uza KN (0) _____ Manorin uza KN (Sura hamutuk valór hotu-hotu hosi leten, hakerek totál iha-ne'e): _____ Pergunta ba manorin 11. Husu manorin, saida mak formadór sira bele ajuda atu manorin sira uza KN ho di'ak liután. Hakerek rezumu ba sira-nia resposta: 12. Formasaun hira ona mak Ita tuir ona kona-ba uza KN? 13. Dala ruma Ita hasoru malu ho maluk manorin sira seluk atu ko'alia konao-ba uza planu lisaun no oinsá bele tulun malo halo atividade ruma? Imi ajuda malu atu halo planu lisaun ka atu halao atividade ruma ka? 14. Se hatán sin, nune'e imi hasoru malu dala hira? 15. Maizumenus imi hasoru malu dala hira iha fulan kotuk? 16. Lista atividade hirak-ne'ebé imi hasoru malu durante ne'e. Pergunta ba koordenadór sira 17. Oinsá Ita-nia hanoin kona-ba Kurríkulu ne'e? Hatán ho fraze ida de'it. 18. Durante ne'e ita halo saida atu enkoraja manorin sira para uza Kurríkulu? 19. Oinsá ita monitora atu haree se manorin sira uza Kurikulu ne'e ka lae? The New Curriculum and the Professional Learning and Mentoring Program 91 Instrumentu B Sala: Ekipamentu no Preparasaun 1. Observasaun ba klase hira: klase ___________ Se multi klase (klase saida? _______________ ) Se multi klase, númeru estudante hira: klase 1 ____________ klase 2 ____________ Impresta lista prezensa klase nian. 2. Númeru estudante ne'ebé hahú iha inísiu? ________ Se multi klase hira? klase 1 _________ klase 2 __________ 3. Mane? ______________ Feto? _______________ a. Klase 1: feto _________ mane _________ b. Klase 2: feto _________ mane _________ Ekipamentu no preparasaun 4. Estudante, kadeira no meza iha: Hili deskrisaun. Kade estudante ha kadeira no meza (5) Estudante balu la iha kadeira no meza (4) Estudante barak la iha kadeira no meza (2) Estudante barak liu mak la iha kadeira no meza (1) 5. Oinsá organiza meza no kadeira: Liña naruk Sírkulu grupu ki'ik Tuur namkari 6. Tau materiál sira-ne'ebé prepara ona husi Kurríkulu Nasionál Ministériu Edukasaun nian mak hanesan mapa, diagrama, kartolina Alfabetu Tetun no portugés, oráriu, kalendáriu, korreiu amizade, katinu dixiplina ida-idak, kartás, mini biblioteka, nst. (barak) (5) (balu) (4) ida ka rua (2) la iha (0) 7. Tau kahur estudante sira-ne'ebé ativu no la ativu, nakar no defisiénsia. Sin (4) maizumenus (2) Lae (0) 8. Ekipamentu iha klase sufisiente no kondisaun di'ak. Sin (4) maizumenus (2) Lae (0) 92 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste Oinsá atu hatama dadus husi intrumentu C ba Baze Dadus (Hare instrusaun: "Prepara dadus observasaun klase ba base dadus") Eskola ____________________________ Klase Manorin ___________________________ Estudante mane sira-ne'ebé iha ona eskola laran bainhira klase hahú ________ Estudante feto sira-ne'ebé iha ona eskola laran bainhira klase hahú ________ Oras klase hahú nian ___________ Koluna 1 Dixiplina Totál Minutu hanorin Literasia Tetun ____________ (0) la halo Totál Minutu hanorin Literasia Portug'es ____________ (0) la halo Totál Minutu hanorin ALO ____________ (0) la halo Totál Minutu hanorin Matemátika ____________ (0) la halo Totál Minutu hanorin Siénsia Naturál ____________ (0) la halo Totál Minutu hanorin Siénsia Sosiál ____________ (0) la halo Totál Minutu hanorin Arte no Kultura ____________ (0) la halo Totál Minutu hanorin Saúde ____________ (0) la halo Totál Minutu hanorin Edukasaun Fiziku ____________ (0) la halo Totál Minutu hanorin Relijiaun ____________ (0) la halo Oras klase remata nian _______________ Oras remata tuir kurríkulu _______________ Koluna 2 Instrusaun relasiona ho atividade a. Manorin hanorin, explika ka demonstra abilidade foun (HED) _____ b. Manorin uza materiál vizuál ka objetu hodi ajuda hanorin konseitu ka abilidade foun (V) _____ c. Manorin responde ba estudante nia resposta lihosi fó informasaun adisionál (RA) _____ d. Manorin no estudante envolve hamutuk iha atividade hatán hanesan koru (maneira repitisaun) (MEK) _____ e. Estudante kopia matéria hosi kuadru (KK) _____ f. Estudante prátika atividade ka abilidade balu mesmesak (PAM) _____ g. Estudante prátika abilidade ka atividade balu iha grupu ki'ik (PAG) ____ h. Estudante hanorin sira-nia belun ida-ida ka iha grupu ki'ik (EH) _____ i. Manorin fó komentáriu pozitivu ba estudante (KP) _____ j. Manorin servisu ho estudante ida-ida (EI) _____ k. Manorin fó pergunta ho reposta limitadu ba estudante ida-idak (PL) _____ l. Manorin fó pergunta maklokek ba estudante ida-idak, la'ós resposta sin ka lae (PM) _____ m. Teste ka avaliasaun balu la'o hela (AW) _____ n. Manorin ho estudante hananu no halo jogu hamutuk (HJ) _____ o. Manorin la'o hale'u klase hodi halo interasaun ho estudante (LI) _____ p. Manorin halo sumáriu ba konteúdu lisaun loron ohin nian (SK) _____ The New Curriculum and the Professional Learning and Mentoring Program 93 Atividade la'ós instrusaun a. Intervalu (I) _____ b. Tempu han (F) _____ c. Lista Prezensa (L) _____ d. Atividade dixiplina (D) _____ e. Interrupsaun husi atividade seluk (IA) _____ f. Atividade administrativa rutina la'o hela (fó ka aruma livru sira, nst) (AA) _____ g. La defini (LD) _____ Koluna 3 Lian Instrusaun a. Tetun (T) _____ b. Portugés (P) _____ c. Indonézia (I) _____ d. Lian lokál (LL) _____ Koluna 4 - Estudante ne'ebé la envolve Tuku 8 Estudante ne'ebé la envolve _______ Númeru estudante ne'ebé prezente iha klase ______ Tuku 9 Estudante ne'ebé la envolve _______ Númeru estudante ne'ebé prezente iha klase ______ Tuku 10 Estudante ne'ebé la envolve _______ Númeru estudante ne'ebé prezente iha klase ______ Tuku 11 Estudante ne'ebé la envolve _______ Númeru estudante ne'ebé prezente iha klase ______ Tuku 12 Estudante ne'ebé la envolve _______ Númeru estudante ne'ebé prezente iha klase ______ Tuku 13 Estudante ne'ebé la envolve _______ Númeru estudante ne'ebé prezente iha klase ______ NOTA: Se klase remata molok tuku 12 ka 13, sei la hatama buat ruma ba dadus ba hirak-ne'e. 94 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste Instrument C (Dadeer) Observasaun Sala - Uza Oras Data _______________________ Observador _________________________________________________ Eskola ______________________________________________ Municipaliti _____________________________________ Klase ______________________ Eskola PLMP Sim Lae Estudante nebee tama? Feto ________ Mane _________ Estudante nebee fila uluk? Feto ________ Mane _________ Oras Subj. Act. Lian La Oras Subj. Act. Lian La Oras Subj. Act. Lian La envolve envolve envolve 8:00 10:06 12:09 8:03 10:09 12:12 8:06 10:12 12:15 8:09 10:15 12:18 8:12 10:18 12:21 8:15 10:21 12:24 8:18 10:24 12:27 8:21 10:27 12:30 8:24 10:30 12:33 8:27 10:33 12:36 8:30 10:36 12:39 8:33 10:39 12:42 8:36 10:42 12:45 8:39 10:45 12:48 8:42 10:48 12:51 8:45 10:51 12:54 8:48 10:54 12:57 8:51 10:57 1:00 8:54 11:00 1:03 8:57 11:03 1:06 9:00 11:06 1:09 9:03 11:09 1:12 9:06 11:12 1:12 9:09 11:15 1:15 9:12 11:18 1:18 9:15 11:21 1:21 9:18 11:24 1:24 9:21 11:27 1:27 9:24 11:30 1:30 9:27 11:33 9:30 11:36 9:33 11:39 9:36 11:42 9:39 11:45 9:42 11:48 9:45 11:51 9:48 11:54 9:51 11:57 9:54 12:00 9:57 12:03 10:00 12:06 The New Curriculum and the Professional Learning and Mentoring Program 95 Instrument C (Lokraik) Observasaun Sala - Uza Oras Data _______________________ Observador _________________________________________________ Eskola ______________________________________________ Municipaliti _____________________________________ Klase ______________________ Eskola PLMP Sim Lae Estudante nebee tama? Feto ________ Mane _________ Estudante nebee fila uluk? Feto ________ Mane _________ Oras Subj. Act. Lian La Oras Subj. Act. Lian La Oras Subj. Act. Lian La envolve envolve envolve 12:30 2:36 4:42 12:33 2:39 4:45 12:36 2:42 4:48 12:39 2:45 4:51 12:42 2:48 4:54 12:45 2:51 4:57 12:48 2:54 5:00 12:51 2:57 5:03 12:54 3:00 5:06 12:57 3:03 5:09 1:00 3:06 5:12 1:03 3:09 5:15 1:06 3:12 5:18 1:09 3:15 5:21 1:12 3:18 5:24 1:15 3:21 5:27 1:18 3:24 5:30 1:21 3:27 5:33 1:24 3:30 5:36 1:27 3:33 5:39 1:30 3:36 5:42 1:33 3:39 5:45 1:36 3:42 5:48 1:39 3:45 5:51 1:42 3:48 5:54 1:45 3:51 5:57 1:48 3:54 6:00 1:51 3:57 1:54 4:00 1:57 4:03 2:00 4:06 2:03 4:09 2:06 4:12 2:09 4:15 2:12 4:18 2:15 4:21 2:18 4:24 2:21 4:27 2:24 4:30 2:27 4:33 2:30 4:36 2:33 4:39 96 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste Instrumentu D Kalkula De'it "Manorin nia Dezempeñu" Kalkuladu, organiza, prepara 1. Manorin iha ona klase laran prepara antes aula ohin loron hahuu. Loos (5) To'o uluk de'it estudante (3) To'o tarde uitoan (1) To'o tarde liu (0) _____ 2. Manorin iha planu lisaun no hatene ho di'ak saida no oinsá mak atu hanorin. Loos (5) Jeralmente prepara maibé ladún di'ak (4) Laiha planu ba preparasaun (3) Laiha preparasaun (1) Manorin sai hosi tópiku, la prepara, laiha ligasaun (0) _____ 3. Lista prezensa atuál. Loos (5) Ladun (4) Falta dadus barak (3) La adekuadu (2) Laiha (0) _____ Prezensa iha klase, personalidade and maneira 4. Manorin livre iha klase ho iha ninia knaar hanesan manorin. "Livre" katak sa? Manorin kalma, la brutu, laiha nervozu/a, ka la tauk. Manorin haree no kumprementa estudante wainhira sira to'o. Manorin hatudu sentidu konfiansa no prontidaun. Manorin hakarak atu hahuu aula. Estudante hatudu respeitu ba manorin. Loos (5) Dalaruma loos (4) La hatene (3) Dalaruma la loos (2) La loos liu (0) _____ 5. Manorin kontrola duni klase. Manorin onestu no seguru. Estudante komprende saida manorin hakarak hosi sira no sira hatán lalais. Loos (5) Dalaruma loos (4) La hatene (3) Dalaruma la loos (2) La loos liu (0) _____ 6. Manorin kontente, komunika livre ho estudante, no involve estudante ho másimu iha saida de'it mak atu hanorin no atu aprende. Loos (5) Dalaruma loos (4) La hatene (3) Dalaruma la loos (2) La loos liu (0) _____ 7. Manorin uza maneira efetivu hodi ko'alia no komunika ho estudante. Loos (5) Dalaruma loos (4) La hatene (3) Dalaruma la loos (2) La loos liu (0) _____ Fó instrusaun 8. Manorin fó instrusaun tárjetu de'it ba estudante individual ka grupu ne'ebé aprende neneik liu. Loos (5) Dalaruma loos (4) La hatene (3) Dalaruma la loos (2) La loos liu (0) _____ 9. Manorin la domina konteudu lisaun ne'ebé nia hanorin. Loos (5) Dalaruma loos (4) La hatene (3) Dalaruma la loos (2) La loos liu (0) _____ The New Curriculum and the Professional Learning and Mentoring Program 97 10. Manorin efetivu duni hodi hanorin konteudu foun. (Ida ne'e atinji lihosi liga konteudu foun ba buat ne'ebe estudante hatene ona, liuhosi kontestu barak, no liuhosi fó izemplu natón no demonstra atu estudante bele komprende konteudu foun). Loos (5) Dalaruma loos (4) La hatene (3) Dalaruma la loos (2) La loos liu (0) _____ 11. Manorin fó oportunidade barak ba estudante atu prátika abilidade foun mes-mesak ka iha grupu liuhosi ezersisiu iha kadernu, servisu uma, aprezentasaun no hatán pergunta maklokek, nsst. Loos (5) Dalaruma loos (4) La hatene (3) Dalaruma la loos (2) La loos liu (0) _____ 12. Manorin avalia labarik ida-idak nia aproveitamentu liuhosi halo revizaun, teste, pergunta maklokek, projetu individual, nsst. Loos (5) Dalaruma loos (4) La hatene (3) Dalaruma la loos (2) La loos liu (0) _____ 13. Manorin iha komprensaun di'ak kona-ba labarik ida-idak nia progresu no bele identifika etape ne'ebé tuir estudante nia nesesidade. Loos (5) Dalaruma loos (4) La hatene (3) Dalaruma la loos (2) La loos liu (0) _____ 14. Manorin iha komprensaun kona-ba "figura jerál" ba tinan ida ne'e. Ne'e katak, manorin bele fó deskrisaun jerál kona-ba de'it mak presiza atu aprende iha tinan ida ne'e no komprende oinsá lisaun sira nia konteudu bele atinji rezultadu aprendizajen iha tinan tomak. Loos (5) Dalaruma loos (4) La hatene (3) Dalaruma la loos (2) La loos liu (0) _____ Jestaun klase no administrasaun 15. Manorin komprende labarik no sira-nia hahalok no bele rezolve problema sira ho loloos, onestu no pozitivu. Loos (5) Dalaruma loos (4) La hatene (3) Dalaruma la loos (2) La loos liu (0) _____ 16. Manorin nia klase la'o ho di'ak no produtivu no disturbu uitoan de'it. Loos (5) Dalaruma loos (4) La hatene (3) Dalaruma la loos (2) La loos liu (0) _____ 98 Using EGRA for an Early Evaluation of Two Innovations in Basic Education in Timor-Leste