Belarus SABER Country Report STUDENT ASSESSMENT 2015 Key Policy Areas for Student Assessment Status 1. Classroom Assessment Classroom assessment in Belarus is supported by formal, system-level documents that provide comprehensive information on what students are expected to learn and how classroom assessment should be conducted by teachers. Mechanisms are in place to monitor the quality of classroom assessment practices, and teachers engage in positive assessment practices in the classroom. At the same time, there is a need to improve the quality and availability of resources that support teachers ’ classroom assessment practices and to ensure that high-quality training opportunities are available to teachers on a regular basis. 2. Examinations Centralized Testing is an examination program that is used to select students into higher and specialized secondary education in Belarus. The program is overseen by the Republic Institute for Knowledge Control (RIKC), is adequately staffed, and is well resourced through government funding. The examination is well aligned with other assessment activities in the country, and extensive quality assurance procedures are used to help ensure its technical quality. At the same time, there is a need to better ensure equitable access to the examination, including by students with disabilities, and to introduce mechanisms to monitor the impact of the examination on the education system and on students. 3. National Large-Scale Assessment (NLSA) Belarus’s National Large-Scale Assessment exercise—the Republican Monitoring of Education Quality (“Republican Monitoring”)—has been administered annually since 2003. The Republican Monitoring exercise aims to assess the degree to which students have acquired the content taught under the formal educational program. The Republican Monitoring exercise is fully standardized at the system level, and sufficient funding is provided through national and local budgets to cover all core assessment activities. The Department of Education Quality Monitoring, which oversees the assessment, has most of the necessary resources to carry out its responsibilities, although there is room for improvement in the availability of computers for technical staff, software, and servers. 4. International Large-Scale Assessment (ILSA) Belarus has never participated in an ILSA but is taking active steps to participate in the 2018 administration of the Program for International Student Assessment (PISA). RIKC will be responsible for PISA 2018 activities in the country. RIKC has extensive experience carrying out assessments within Belarus. RIKC specialists, as well as staff from the National Institute of Education who will help with PISA 2018 activities, are well prepared to commence work on the ILSA. To maximize the benefits of participating in PISA 2018, the country may wish to consider doing a detailed needs assessment and developing a plan to address areas where they will need particular technical support. THE WORLD BANK Table of Contents Introduction ................................................................................................................................................3 What Is SABER–Student Assessment? ........................................................................................................3 Education in Belarus ....................................................................................................................................5 Classroom Assessment in Belarus ...............................................................................................................7 Examinations in Belarus ..............................................................................................................................9 National Large-Scale Assessment in Belarus .............................................................................................12 International Large-Scale Assessment in Belarus .....................................................................................16 Appendix 1: Assessment Types and Their Key Differences .......................................................................18 Appendix 2: Summary of the Development Levels for Each Assessment Type .......................................19 Appendix 3: Methodology for Assigning Development Levels .................................................................20 Appendix 4: Stakeholders Consulted during Completion of the SABER–Student Assessment Questionnaires in Belarus ................................................................................ 21 Appendix 5: SABER–Student Assessment Rubrics for Belarus ..................................................................23 Acknowledgments .................................................................................................................................... 66 References.................................................................................................................................................66 2 BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 Introduction SABER–Student Assessment methodology The SABER–Student Assessment framework is built on Belarus has focused on increasing student learning the available evidence base for what an effective outcomes by improving the quality of education in the assessment system looks like. The framework provides country. An effective student assessment system is an guidance on how countries can build more effective important component of efforts to improve education student assessment systems. The framework is quality and learning outcomes because it provides the structured around two main dimensions of assessment necessary information to meet stakeholders’ decision- systems: the types/purposes of assessment activities making needs. To gain a better understanding of the and the quality of those activities. strengths and weaknesses of its existing assessment system, Belarus decided to benchmark this system Assessment Types and Purposes using standardized tools developed under the World Bank’s Systems Approach for Better Education Results Assessment systems tend to comprise three main (SABER) program. SABER is an evidence-based program types of assessment activities, each of which serves a to help countries systematically examine and different purpose and addresses different information strengthen the performance of different aspects of needs. These three main types are classroom their education systems. assessment, examinations, and large-scale, system- level assessments. What Is SABER–Student Assessment? Classroom assessment provides real-time information to support ongoing teaching and learning in individual SABER–Student Assessment is a component of the classrooms. Classroom assessments use a variety of SABER program that focuses specifically on formats, including observation, questioning, and benchmarking student assessment policies and paper-and-pencil tests, to evaluate student learning, systems. The goal of SABER–Student Assessment is to generally on a daily basis. promote stronger assessment systems that contribute to improved education quality and learning for all. Examinations provide a basis for selecting or certifying students as they move from one level of the education system to the next (or into the workforce). All eligible National governments and international agencies are students are tested on an annual basis (or more often increasingly recognizing the key role that assessment if the system allows for repeat testing). Examinations of student learning plays in an effective education cover the main subject areas in the curriculum and system. The importance of assessment is linked to its usually involve essays and multiple-choice questions. role in the following areas: 1. Providing information on levels of student learning and achievement in the system Large-scale, system-level assessments provide 2. Monitoring trends in education quality over feedback on the overall performance of the education time system at particular grades or age levels. These 3. Supporting educators and students with real- assessments typically cover a few subjects on a regular time information to improve teaching and basis (such as every three to five years), are often learning and sample-based, and use multiple-choice and short- 4. Holding stakeholders accountable for results. answer formats. They may be national or international in scope. Appendix 1 summarizes the key features of these main types of assessment activities. 3 BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 Quality Drivers of an Assessment System Table 1: Framework for Building an Effective Assessment System, with Indicator Areas The key considerations when evaluating a student assessment system are the individual and combined quality of assessment activities in terms of the adequacy of the information generated to support decision making. There are three main drivers of information quality in an assessment system: enabling context, system alignment, and assessment quality. Enabling context refers to the broader context in which the assessment activity takes place and the extent to which that context is conducive to, or supportive of, the assessment. It covers such issues as the legislative or policy framework for assessment activities; The indicators are identified based on a combination of institutional and organizational structures for criteria, including the following: designing, carrying out, or using results from the assessment; the availability of sufficient and stable x Professional standards for assessment sources of funding; and the presence of trained x Empirical research on the characteristics of assessment staff. effective assessment systems, including analysis of the characteristics that differentiate between the System alignment refers to the extent to which the assessment systems of low- versus high- assessment is aligned with the rest of the education performing nations and system. This includes the degree of congruence x Theory—that is, general consensus among experts between assessment activities and system learning that it contributes to effective assessment. goals, standards, curriculum, and pre- and in-service teacher training. Levels of Development Assessment quality refers to the psychometric quality The World Bank has developed a set of standardized of the instruments, processes, and procedures for the questionnaires and rubrics for collecting and assessment activity. It covers such issues as design and evaluating data on the three assessment types and implementation of assessment activities, analysis and related quality drivers. interpretation of student responses to those activities, and the appropriateness of how assessment results are reported and used. The questionnaires are used to collect data on the characteristics of the assessment system in a particular country. The information from the questionnaires is Crossing the quality drivers with the different then applied to the rubrics to judge the development assessment types/purposes provides the framework level of the country’s assessment system in different and broad indicator areas shown in table 1. This areas. framework is a starting point for identifying indicators that can be used to review assessment systems and plan for their improvement. Rubrics are used to evaluate data collected using the standardized questionnaires. The goal of the rubrics is to provide a country with some sense of the development level of its assessment activities compared to best or recommended practice. For each indicator, the rubric displays four development levels: Latent, Emerging, Established, and Advanced. These levels are artificially constructed categories chosen to represent key stages on the underlying continuum for 4 BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 each indicator. Each level is accompanied by a age of six. In accordance with statute 2 of the Code of description of what performance on the indicator looks the Republic of Belarus on education in the Republic of like at that level. Belarus, general basic education is mandatory, and this includes elementary education (grades 1–4) and basic x Latent is the lowest level of performance; it education (grades 5–9). Upon completion of grade 9 represents absence of the desired attribute. (generally at age 15), students must pass exit x Emerging is the next level; it represents partial examinations to receive a certificate of completion of presence of the attribute. basic education. Students can continue their secondary x Established represents the acceptable minimum education in institutions of general secondary standard. education (grades 10–11), institutions of professional- technical education, or institutions of secondary- x Advanced represents the ideal or current best specialized education. Students who continue through practice. general secondary education and complete grade 11 must pass exit examinations to receive a certificate of A summary of the development levels for each general secondary school completion. In May 2015, assessment type is presented in appendix 2. Belarus joined the Bologna Process as a way to further enhance its higher education system and align it with In reality, assessment systems are likely to be at international standards. different levels of development in different areas. For example, a system may be Established in the area of The education system in Belarus faces several examinations, but Emerging in the area of large-scale, challenges despite having made major progress. Since system-level assessment, and vice versa. Although its independence in 1991, Belarus has expanded access intuition suggests that it is probably better to be to education at the preprimary and secondary levels. further along in as many areas as possible, the However, slowed economic growth and an aging and evidence is unclear as to whether it is necessary to be declining population have increased attention to the functioning at Advanced levels in all areas. Therefore, need for the country to increase efficiency within the one might view the Established level as a desirable education system and workers’ productivity. In minimum outcome to achieve in all areas but aspire response to these needs, efforts to improve education beyond that only in those areas that most contribute quality and optimize the education system have been to the national vision or priorities for education. In line introduced. One such initiative has focused on with these considerations, the ratings generated by the increasing financial efficiencies within the system as rubrics are not meant to be additive across assessment schools are consolidated and students are transported types (that is, they are not meant to be added to create from areas with a smaller student population to areas an overall rating for an assessment system; they are with a larger student population under the National meant only to produce an overall rating for each Development Program for General Secondary assessment type). The methodology for assigning Education (2007–16). Although the school development levels is summarized in appendix 3. consolidation program has increased efficiencies, it has not been undertaken in parallel with quality improvements. For example, schools that have Education in Belarus remained open and are receiving new students from areas with lower populations have not been provided with additional quality-enhancing inputs, such as Belarus is an upper-middle income country in Eastern improved learning environments, learning materials, Europe. GDP per capita (current $) is $8,040. Between or modern infrastructure, equipment, and technology. 2001 and 2008, GDP growth averaged 8.3 percent annually. However, growth has slowed considerably In response to these and other challenges, Belarus is since then, to approximately 1.6 percent in 2014, in prioritizing the reform of learning environments to part because of a reduction in export demand and enhance quality by improving schools’ infrastructure access to external borrowing. and expanding access to laboratories and information technology. In addition, to improve the quality of Although preschool education is not compulsory in education, Belarus is interested in improving its Belarus, 99 percent of children attend early childhood assessment system to have better data on student development programs before starting school at the 5 BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 performance that can inform further education reforms and policies. Detailed information was collected on Belarus’s student assessment system using the SABER–Student Assessment questionnaires and rubrics to benchmark it against best practices. Specifically, a local consultant with in-depth knowledge of, and experience with, the education system in Belarus oversaw the completion of the four SABER-Student Assessment questionnaires: one questionnaire each for Classroom Assessment, Examinations, National Large-Scale Assessment, and International Large-Scale Assessment. The data to complete these questionnaires were obtained through interviews and focus groups with key stakeholders 1 and a review of existing official and technical documents. The information in the completed questionnaires was then applied to the SABER–Student Assessment rubrics (one rubric for each assessment type), and the conclusions of this report were determined on the basis of this analysis. It is important to remember that these tools primarily focus on benchmarking a country’s policies and arrangements for assessment activities at the system or macro level. Additional data would need to be collected to determine actual, on-the-ground practices in Belarus, particularly by teachers and students in schools. The following sections discuss the findings by each assessment type, accompanied by suggested policy options. The suggested policy options were determined in collaboration with key local stakeholders based on Belarus’s immediate interests and needs. Detailed, completed rubrics for each assessment type are provided in appendix 5. 1 The list of stakeholders consulted is provided in appendix 4. 6 BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 Classroom Assessment in Belarus Schools report individual student performance on classroom assessment in all subject areas to students Level of Development: ESTABLISHED and their parents through schoolwide and class- specific parent-teacher conferences. Student grades Several formal, system-level documents provide are recorded in a classroom journal and in the guidelines for classroom assessment in Belarus. For student’s grade journal (dnevnik). Parents are required example, the documents “Educational Standards for to review the student’s grade journal every week and School Subjects, 2009” and “Educational Programs for every quarter. There is currently a move toward Extracurricular Subjects, 2009–2013” offer useful maintaining the classroom journal and the student’s guidance for classroom assessment activities by grade journal in electronic form. Parents can also specifying what students are expected to learn in receive information on their child’s performance in the different subjects at different grade levels. These classroom through SMS. documents do not, however, describe the level of performance that students are expected to reach in According to the data provided in the completed each subject area (that is, how well students should SABER–Student Assessment questionnaire, extensive learn something). Another official document, resources are available to teachers for their classroom “Educational Standards for School Subjects” (2012, assessment activities and for developing competency 2015), not only outlines what students are expected to in classroom assessment. All teachers receive learn, but also specifies the level of performance that textbooks that contain questions and sample material they should demonstrate in each subject area. All of for conducting classroom assessment activities. these documents are publicly available. Compilations of materials to be used for conducting continuous, intermediate, and final assessment have Other official and publicly available documents provide been published for all school subjects in grades 1–11 more specific guidance for classroom assessment and are available for purchase. All subject teachers activities in general secondary education, including receive funds specifically to acquire the describing what should be assessed, criteria for scoring methodological literature. Diagnostic assessment student work, formats and methods for carrying out modules for all subjects in grades 1–11 are available on classroom assessment activities, mechanisms for the national education portal. All participants in the creating scores and marks, and record keeping.2 education process, including students, parents, and teachers, have access to them. Although the sample Most teachers use classroom assessment information questions for classroom assessment activities and to diagnose student learning issues, provide computer-based resources are considered to be of high continuous feedback to students on their learning, plan quality, textbooks are considered to be of medium further instruction, and evaluate student performance. quality. In addition, although preservice teacher Additionally, classroom assessment information is training in classroom assessment is available to most required to be used for certification of student teachers, it is neither of high quality nor available on an achievement and as an input for selection to the next annual basis. Teacher self-education opportunities are educational level; guidance for these uses is provided available to all teachers. For example, in institutions of in official documentation made available to schools. general secondary education teachers independently According to the data provided in the completed select topics for self-education. Specifically, teachers SABER–Student Assessment questionnaire, a marginal review the educational, pedagogical, psychological, number of teachers in Belarus engage in poor and methodological literature related to their chose classroom assessment practices, with some teachers topic; develop lessons on their selected topic; present engaging in grade inflation and overemphasizing the and discuss the topic at methodological meetings that assessment of memorization/recall of information. 2 These documents include “Assessment Norms for Ungraded Learning Stage I of General Secondary Outcomes of Educational Activity of Students in Educational Education,” and “Rules for Certification of Pupils at Subjects,” “On Organization of Work of General Secondary Mastering Content Educational Programs of General Education Institutions to Monitor and Evaluate the Secondary Education.” Outcomes of Learning Activity of Students during the 7 BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 bring together subject teachers; and publish their b. Institute annual in-service teacher training methodological findings in methodological subject opportunities on classroom assessment journals, in the newspaper Nastaunitskaya Gazeta, on (including via live and prerecorded courses the websites of their education institutions, and on delivered via computer) to ensure that their personal websites. Teachers present their teachers have the ability to hone their publications on and experience with their chosen self- classroom assessment knowledge and skills education topic during an examination that they take every year as needed. to confirm their existing level of teaching qualification c. Ensure that distance learning options are or to receive the next level of teaching qualification. Teachers also benefit from groups that are formed available to all teachers. within the education institutions that provide a forum for teachers to discuss various topics in education, such as methodological approaches in various subject areas and assessment of student learning. A number of systems are in place to monitor the quality of classroom assessment practices, including as a part of school inspection and teacher supervision and performance evaluation. In addition, high-level reviews of the quality of education in Belarus include a focus on classroom assessment, and government funding is awarded to researchers and practitioners for research and development on classroom assessment practices. An external moderation system consists of subject-area commissions at the district level that objectively evaluate teachers’ review of examination questions of students who are being considered to receive an attestation of general secondary education with an award of a gold (or silver) medal. At the same time, a more wide-reaching moderation system, or a moderation system where teachers from different schools meet to review assessment tasks, scoring criteria, and students’ work, with the purpose of ensuring the validity and comparability of student scores and grades, is not in place. Suggested policy options: 1. Increase the quality and availability of resources that support teachers’ classroom assessment practices. For example: a. Improve the quality of student textbooks that provide support for classroom assessment. 2. Ensure that high-quality training opportunities are available to all teachers on a regular basis. For example: a. Review preservice teacher training options for building competencies in classroom assessment to ensure they are of high quality and available on an annual basis. 8 BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 effectiveness. Classroom teachers do not perform any Examinations in Belarus examination-related tasks. Level of Development: ESTABLISHED Opportunities are available to interested stakeholders to learn about the Centralized Testing program, Results from Belarus’s national examination program, including university courses/workshops on the content “Centralized Testing,” are used to select individuals and skills measured by the examination, nonuniversity into higher and specialized secondary education courses and workshops on examination topics, funding institutions. Centralized Testing can be taken by those for attending international programs, courses and individuals who have completed the educational workshops that cover topics relevant to the program of general secondary education, of the examination, presentations, and publications (e.g., professional-technical education that provides articles, collections of materials for Centralized vocational qualification and general secondary Testing). According to the data provided in the education, or of specialized secondary education as completed SABER–Student Assessment questionnaire, confirmed by a document of completion of such opportunities are generally of high quality and are education. Results are also used to inform policy and available to students studying in higher education evaluate interventions aimed at improving student institutions, existing full-time staff in the examination learning. The Council of Ministers of the Republic of unit (RIKC), university professors, general secondary Belarus Resolution no. 714, “On Approval of the school educators, and specialists in the education Regulation on the Procedure for Organizing and management bodies. At the same time, there are no Holding Centralized Testing” (June 6, 2006), is the university graduate programs on student assessment formal, publicly available policy document that that include topics relevant to the examination (e.g., authorizes the examination program. Guidance on test design, administration), university various aspects of Centralized Testing is provided in courses/workshops on examination topics other than Ministerial Orders and Resolutions. For example, the the content and skills measured by the examination guidance on the rules of computing examination scores (e.g., test design, administration), or internships/short- and the organizational rules at the centers carrying out term employment in the unit running the examination the Centralized Testing is provided within the Order of that would formally equip individuals to work on the the Ministry of Education of the Republic of Belarus Centralized Testing program. Resolution no. 323 of April 22, 2015. Extensive materials are available to students to The Republican Institute for Knowledge Control prepare for the examination. Most students have (RIKC)—accountable to the Ministry of Education, from access to the official framework document explaining which it is institutionally separate—has been what is measured on the examination as well as responsible for many elements of the examination examples of the types of questions that are on the since 2000. examination, provided by either RIKC or some other source. Throughout the school year, three rounds of The Centralized Testing program is well aligned with practice versions of the Centralized Testing are carried classroom assessment and National Large-Scale out, and higher education institutions provide courses Assessment activities in the country in terms of the to help students prepare for the Centralized Testing. content and skills being measured. The Centralized According to the data provided in the completed Testing program measures student achievement in SABER–Student Assessment questionnaire, these relation to the official learning standards and materials are considered to be of medium quality. In curriculum in Belarus. This focus is ensured through general, students do not face barriers to participating officially mandated internal and external alignment in the Centralized Testing program (there are no reviews that take place in all examination rounds. ethnic, racial, linguistic group identification, gender, or location—e.g., rural or hard-to-reach areas—barriers). Overall, the Centralized Testing program is well Individuals with vision, hearing, or musculoskeletal resourced and is sufficiently funded by the disorders do not participate in the Centralized Testing government. Staff possess relevant qualifications for because of a lack of appropriate accommodations or their jobs, and no major issues are seen with their alternative assessment options. Instead, such individuals can participate in entrance examinations in 9 BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 the institutions where they would like to be considered According to the data provided in the completed for admission, with the format of the examination SABER–Student Assessment questionnaire, although being determined by the institution, taking into the use of unauthorized materials such as prepared account the psychological development of the answers and notes, collusion among candidates via candidate. mobile phones, passing of papers, or the equivalent has taken place, the overall credibility of the Extensive quality assurance procedures are in place to examination was not compromised as a result of these ensure the technical quality of the Centralized Testing individual improprieties. program. Specifically, the following formal quality assurance procedures are in place: After receiving their examination results, students may x All proctors or administrators are trained apply to tertiary education institutions, secondary according to a protocol specialized institutions, or vocational education x A standardized manual is available for institutions in Belarus. The examination results are not examination administrators officially recognized by educational institutions or x Questions/items/tasks are piloted before the employers in other countries. Students who do not do official examination administration well on the examination are not allowed to repeat the x There are external observers and grade but may retake the examination in a future year. x There are internal observers. Suggested policy options: The examination is fully standardized at the system level in that its design, administration, scoring, and 1. Ensure equitable access to the examination. For reporting are the same for all students. To monitor the overall impact of the Centralized Testing program on example: education quality and learning levels in the system, a. Improve the quality of the preparation RIKC prepares an Annual Report, which provides materials available to students by reviewing information on the most recent round of examination existing materials, determining the areas that results. need to be improved, and implementing these improvements The Order of the President of the Republic of Belarus b. Increase the options available to students with provides for the creation of a commission on the disabilities for taking the examination, control of the preparation and implementation of particularly students with vision or hearing entrance examinations into institutions of higher and disorders or musculoskeletal disabilities by, for secondary specialized education. The commission example, adjusting the physical space (e.g., oversees the preparation and implementation of installing ramps for students using Centralized Testing every year. At the same time, focus wheelchairs) and providing examination groups or surveys of key stakeholders do not take place, and funding for validation studies or formats specifically adapted for students with independent research on the Centralized Testing these disabilities. program is not available. 2. Increase opportunities for key stakeholders to Sufficient documentation on the methods and learn about the examination. For example: procedures used during the Centralized Testing a. Build a pipeline of qualified assessment program is made available to the public and key specialists who could work on the examination stakeholders. Specifically, test specifications and in various capacities by information on the construction of questions and tasks i. Introducing university graduate programs, is publicly available, while information on the scoring courses, or workshops on assessment and of questions or tasks, reliability, and scaling is prepared testing topics relevant to the examination for the Ministry of Education but not made available to ii. Providing opportunities to university the public. students to do internships or apprenticeships at RIKC and gain hands-on 10 BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 practical experience in carrying out examination-related activities (create confidentiality agreements with students to ensure examination security). 3. Introduce a variety of mechanisms to monitor the impact of the examination on the education system and students. For example: a. Form expert review groups b. Provide funding for independent research on the examination c. Conduct studies on the examination, such as predictive validity studies d. Conduct focus groups or surveys of key stakeholders. 11 BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 National Large-Scale Assessment (NLSA) x Grade 6: Mathematics, world history x Grade 7: Physics, biology in Belarus x Grades 5 and 8: Reading Level of Development: ESTABLISHED The Republican Monitoring exercise assesses students’ knowledge and skills in a particular subject in relation to the content of the educational programs in these Belarus’s National Large-Scale Assessment exercise— grades and subjects that students were to have Republican Monitoring of Education Quality acquired in the previous school year. For example, the (“Republican Monitoring”)—has been administered Republican Monitoring test for mathematics in grade 6 annually since 2003. The Republican Monitoring assesses students’ knowledge of mathematics content exercise aims to assess overall levels of student and skills that should have been covered in grade 5. achievement in relation to national education Given that, starting in 2014, the Republican Monitoring standards and syllabi requirements in particular school exercise is carried out every year in different grades, subjects. The assessment results are used to inform and its objective is the assessment of the degree to policy, pedagogy, and text book design; monitor which students in selected grades acquired the content of updated educational programs, comparative education quality; and evaluate interventions to analysis of the results of the Republican Monitoring is improve learning. not conducted. The Republican Monitoring exercise is carried out in The Republican Monitoring exercise is well aligned with accordance with the official Belarus Secondary School the official learning standards and curriculum, as well Education Development Program for 2007–16, as well as with classroom assessment activities and formal as the annual decrees of the Ministry of Education “On examinations. There are officially mandated reviews to Organization and Carrying Out Secondary School ensure that the Republican Monitoring exercise Education Monitoring in the Academic Year.” The latter measures what it is intended to measure, including offer specific guidance on funding, who and what regular internal and external reviews that take place should be assessed, and how results should be used, during all or almost all Republican Monitoring rounds. among other things. The Republican Monitoring Specifically, the instruments used for the Republican exercise also has been guided by the national Monitoring exercise are subject to internal review in secondary school education monitoring programs for the Department of Education Quality Monitoring, a 2006–9 and 2009–14. The national secondary school permanent unit at the National Institute of Education; education monitoring program for 2015–18 is being informal external review in laboratories of the implemented. Research and Development Centre of the National Institute of Education; and formal external review in the Ministry of Education (in accordance with the The Republican Monitoring was first implemented in decrees of the Minister of Education, all diagnostic and 2003, and until 2011 it covered grades 4, 9, and 11. In instructional materials for carrying out the Republican the 2012–13 and 2013–14 school years, it covered Monitoring exercise are to be coordinated in the grades 3, 8, and 10. Between 2003 and 2014, students’ Departments of the Ministry of Education). knowledge and skills were assessed in the subjects of mathematics and Belarusian and Russian languages. The assessment also covered biology, chemistry, and Students are exposed to the content and skills physics in grades 9 and 11 (in 2003–11) and in grades 8 measured by the assessment through regular course and 10 (in years 2012–14). In 2012 and 2013, instruction at school; they also have access to samples knowledge and skills in social studies were assessed in of previously administered Republican Monitoring grades 9 and 10. assessments. At the same time, preparatory information (e.g., official framework, approved sample questions) on the assessment is not provided to In the 2014–15 school year, students’ knowledge and schools ahead of the test administration. skills were assessed in the following subjects and grades: 12 BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 The Department of Education Quality Monitoring is the quality of education are also financed through responsible for the Republican Monitoring program, other government sources. including the assessment design, administration, data processing, and reporting. The Ministry of Education The Department of Education Quality Monitoring has ensures assessment quality and performs project most of the necessary resources to carry out its management activities. In accordance with the Order responsibilities in relation to the assessment program, of the Ministry of Education of the Republic of Belarus, although there is room for improvement in the committees that are responsible for education availability of computers for technical staff, software, management in an oblast (1) select the coordinator for and servers. The department is appropriately staffed, conducting the Republican Monitoring in institutions of and all staff possess the relevant qualifications for their general secondary education; (2) organize the roles. participation of external observers (specialists from the rayon/city educational and methodological departments, specialists from the institutes of Several opportunities are available to learn about the educational development) and the carrying out of the Republican Monitoring program, including university Republican Monitoring in regional education graduate programs, university and nonuniversity institutions; (3) organize technical support for the workshops, funding for participation in international implementation of the Republican Monitoring in courses on assessment, and opportunities to education institutions in the region (copying of participate in online conferences and webinars. materials, scanning of materials, carrying out a survey However, these opportunities are available to only a on the computer); and (4) provide verification and limited number of individuals. assessment of student responses in educational institutions and in the institutes of education quality. School coordinators, who are appointed by the director A representative random sample of public school of an educational institution, are responsible for the students in the target grades is selected to participate implementation of assessment activities in the in the Republican Monitoring program. Students in educational institutions. Temporary subject private institutions of general secondary education commissions, comprising subject teachers with the comprise 0.07 percent of students in institutions of highest qualifications and teacher-methodologists (at general secondary education in the Republic of Belarus the rayon level and in the institutes of educational and are therefore not included in the Republican development), are formed for the purposes of Monitoring sample. Starting in 2014, students from all verification and evaluation of students’ work. rural schools have an equal opportunity to be included Specifically, specially selected secondary school in the sample of the Republican Monitoring. This subject teachers (top-ranked or supervisors) review required increasing the number of external observers and evaluate the question responses of students by employing specialists from the rayon and city participating in the Republican Monitoring test. teaching offices of the institutes of educational Experts from the Department of Education Quality development. Monitoring developed scoring guides to facilitate teachers’ review and evaluation of students’ The Republican Monitoring program is fully responses. Some issues have been reported with the standardized at the system level: assessment design, effectiveness of these teachers (for example, some administration, scoring, and reporting are the same for teachers overlooked errors, which was noted by the all students in the same assessment round. Quality Department of Education Quality Monitoring of the assurance procedures are in place to help ensure the National Institute of Education staff who reviewed quality of the assessment, including requiring that all students’ work); however, the overall quality of the proctors and administrators be trained according to a assessment was not compromised as a result. protocol, the use of a standardized manual for NLSA administrators, the numbering of all answer sheets, Funding for the Republican Monitoring program is and the use of external observers (e.g., specialists from provided by national and local government budgets the rayon or city teaching offices of the institutes of and covers all core assessment activities. Research and educational development) as well as internal observers development activities on the topic of assessment of (e.g., NLSA staff observing at administration sites). At the same time, other desirable procedures, such as 13 BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 piloting of assessment questions, training of scorers, oversight committee or expert review groups to and double processing of data, have yet to be put in monitor the overall impact of the Republican place. The only inappropriate behavior that has been Monitoring program, and no funding is available for recorded as taking place is students copying from other independent research on the assessment. students. This inappropriate behavior did not affect the credibility of the Republican Monitoring, and a decision Suggested policy options: was made in 2014 to increase the number of different versions of the assessment from two to four to address this issue. 1. Review the design of the Republican Monitoring program in light of the information needs of key decision makers and the intended purposes and Additionally, documentation is limited on the methods uses of the assessment results. This review and procedures used during the Republican should cover: Monitoring exercise. Information on the evaluation and assessment of all questions is provided in the a. Primary purposes of the assessment in the “Recommendations for Evaluation and Assessment of eyes of key decision makers Students’ Work” document that is made available on b. Key desired uses of the assessment results, the website of the management of the monitoring of including the ability to compare scores over the quality of education at the National Institute of time and Education on the day that the Republic Monitoring c. Appropriateness of the current technical takes place. This information is made available to design of the assessment given these assessment coordinators in those educational purposes and uses. institutions that took part in the monitoring. Documentation on test specifications also exists. No 2. To enhance the existing Republican Monitoring documentation is available on how test questions are program, provide preparatory information about constructed, how the test is assembled, or how scaling the program to schools ahead of each test is done. administration. For example: a. A framework document explaining what is There is an official report for the Ministry of Education measured on the assessment on the results of the Republican Monitoring exercise, b. Examples of the types of questions that are on based on which recommendations for enhancing the assessment education quality are prepared and published. The c. Examples of the criteria used for scoring report provides information about the number of questions. students who acquired the content of the taught (and assessed) subject at a high, sufficient, average, acceptable, and low level. Official results of the 3. To improve the existing Republican Monitoring Republican Monitoring are made available at the program, enhance the communication of Ministry of Education, in education-specific scientific- assessment results by creating guidelines for methodological journals, as well as on the national how results should be communicated to key education portal, which publishes a summarized stakeholder groups. This should include: version with recommendations on how to improve the a. Guidelines for the content and format of management of education quality. National results at targeted publications for different the question/item level are also available through stakeholder groups (for example, policy Nastaunitskaya Gazeta. Schools (including teachers), makers, schools, parents) parents, and the local authorities have access to this b. Advice on the types of results that should be information. communicated to different stakeholder groups (for example, question-level data for Limited mechanisms are in place to monitor the impact teachers). of the Republican Monitoring program on quality and learning in the education system. Focus group discussions are held with secondary school teachers during most assessment rounds; however, there is no 14 BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 4. To enhance the existing Republican Monitoring program, continue to strengthen its technical quality by, for example: a. Piloting questions, items, and tasks before the official assessment administration (a decision has been made to conduct piloting) b. Training scorers to ensure high interrater reliability c. Documenting methods and procedures used in the assessment exercise, including information on i. Construction of questions, items, and tasks ii. Pilot activities and analysis of piloted questions, items, and tasks iii. Test assembly iv. Reliability and v. Scaling. 15 BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 program implemented by the Russian Centre for International Large-Scale Assessment International Cooperation in Education Development (ILSA) in Belarus (CICED). Level of Development: LATENT Suggested policy options: Until recently, Belarus had never participated in an 1. Enabling Context: According to best practices, the international large-scale assessment exercise. Now, following may assist Belarus in building its enabling however, the country is actively planning to take part context for carrying out ILSA activities in the mid- in the 2018 Program for International Student to long term: Assessment (PISA 2018). The Republican Institute for a. Prepare a formal policy document that Knowledge Control (RIKC) will be responsible for PISA authorizes ILSA activity and explains its role in 2018 activities in the country. Although PISA 2018 supporting improved education quality and represents the first time that RIKC will carry out an learning in Belarus, and make it available to the international large-scale assessment exercise, the general public. institute has prior experience carrying out assessments because its main function is to carry out the Centralized b. Develop a medium- to long-term funding plan Testing program in Belarus. Funding support for PISA for ILSA activities and ensure that sufficient 2018–related activities in Belarus will be provided funding is available to carry out all ILSA through a World Bank project loan. activities, particularly core activities essential to the technical integrity of the assessment and the utility of the results. In addition to RIKC staff, specialists who work at the c. Conduct a needs assessment of organizational National Institute of Education, the Academy of Post- diploma Education, institutes of educational resources (e.g., computers, software, storage development, and local educational authorities are facilities, building security, communication prepared to work on PISA 2018, mainly because of their tools) and human resources (e.g., specialists, previous experience working on the Republican translators) that will be needed to carry out Monitoring of Education Quality program. These ILSA activities, particularly core ILSA activities. experts have degrees in pedagogy, social studies, and psychology, and many of them have completed 2. System Alignment: Belarus should focus on training courses relevant to PISA, including a “Basic enhancing the alignment between its ILSA activity Theory and Methods of Pedagogical Measurements” and other components of its education system. For course. National education quality assessment system example, the country should: experts regularly participate in webinars conducted by a. Ensure that opportunities to learn about the the Higher School of Economics Russian Training ILSA (including its design, content, and skills Centre at the Institute of Education in Moscow and receive information on education quality assessment measured) are available to key stakeholder from Eurasian Association for Educational Assessment groups (including teachers) ahead of time. (EAOKO) news bulletins. b. Ensure that students have sufficient opportunities to be exposed to the content and skills measured by the ILSA ahead of time, Experts from the National Institute of Education also including through regular instruction at school, have experience conducting research in the area of and that they have practice responding to the educational assessment. In 2013 the institute, in association with the Russian National Training types of question formats that PISA uses. Foundation, conducted research to measure the information and communication competencies of 3. Assessment Quality: In the case of PISA and other grade 9 students in Belarus. The “Information and ILSAs, many aspects of assessment quality are Communication Competence Assessment of Grade 9 already monitored through standardized protocols Students in the Republic of Belarus” project was and centralized quality assurance mechanisms. implemented within the framework of a small grants However, countries have latitude in how they 16 BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 communicate and use their results. This is an area where it would be beneficial for the Belarus team to: a. Develop a plan for targeted communication of ILSA results to national stakeholder groups. b. Learn how other countries have communicated results to different stakeholders in ways that maximize understanding and use. c. Learn about and plan for policy- and pedagogically relevant analyses of the PISA data to inform future policy and planning in Belarus. 17 BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 Appendix 1: Assessment Types and Their Key Differences Classroom Large-scale assessment Examinations surveys National International Exit Entrance Purpose To provide To provide To provide To certify To select immediate feedback on feedback on the students as they students for feedback to overall health of comparative move from one further inform the system at performance of level of the educational classroom particular the education education system opportunities instruction grade/age system at to the next (or level(s) and to particular into the monitor trends in grade/age workforce) learning level(s) Frequency Daily For individual For individual Annually and Annually and subjects offered subjects offered more often more often on a regular on a regular where the system where the system basis (such as basis (such as allows for allows for every 3–5 years) every 3–5 years) repeats repeats Who is All students Sample or A sample of All eligible All eligible tested? census of students at a students students students at a particular grade particular grade or age level(s) or age level(s) Format Varies from Usually multiple Usually multiple Usually essay Usually essay observation to choice and short choice and short and multiple and multiple questioning to answer answer choice choice paper-and-pencil tests to student performances Coverage of All subject areas Generally Generally Covers main Covers main curriculum confined to a few confined to one subject areas subject areas subjects or two subjects Additional Yes, as part of Frequently Yes Seldom Seldom information the teaching collected from process students? Scoring Usually informal Varies from Usually involves Varies from Varies from and simple simple to more statistically simple to more simple to more statistically sophisticated statistically statistically sophisticated techniques sophisticated sophisticated techniques techniques techniques 18 BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 Appendix 2: Summary of the Development Levels for Each Assessment Type Assessment type LATENT EMERGING ESTABLISHED ADVANCED Absence of, or deviation On way to meeting Acceptable minimum Best practice from, the attribute minimum standard standard There is no system-wide There is weak system- There is sufficient There is strong system- institutional capacity to wide institutional system-wide institutional wide institutional support and ensure the capacity to support and capacity to support and capacity to support and quality of classroom ensure the quality of ensure the quality of ensure the quality of assessment practices. classroom assessment classroom assessment classroom assessment practices. practices. practices. Classroom Assessment There is no standardized There is a partially There is a stable There is a stable examination in place for stable standardized standardized standardized key decisions. examination in place and examination in place. examination in place and a need to develop There is institutional institutional capacity and institutional capacity to capacity and some strong mechanisms to run the examination. The limited mechanisms to monitor it. The Examinations examination typically is monitor it. The examination is of high of poor quality and is examination is of quality and is perceived perceived as unfair or acceptable quality and is as fair and free from corrupt. perceived as fair for corruption. most students and free from corruption. There is no NLSA in There is an unstable There is a stable NLSA There is a stable NLSA place. NLSA in place and a in place. There is in place and institutional need to develop institutional capacity and capacity and strong institutional capacity to some limited mechanisms to monitor run the NLSA. mechanisms to monitor it. The NLSA is of high National (or System-Level) Assessment quality and it. The NLSA is of quality, and its Large-Scale Assessment impact are weak. moderate quality, and its information is information is effectively used to disseminated, but not improve education. always used in effective ways. There is no history of Participation in an ILSA There is more or less There is stable participation in an ILSA has been initiated, but stable participation in an participation in an ILSA nor plans to participate there still is a need to ILSA. There is and institutional capacity in one. develop institutional institutional capacity to to run the ILSA. The capacity to carry out the carry out the ILSA. The information from the International Large-Scale ILSA. information from the ILSA is effectively used Assessment ILSA is disseminated, to improve education. but not always used in effective ways. 19 BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 Appendix 3: Methodology for Assigning set as ceiling scores; that is, the overall mean score for the particular assessment type cannot be greater than Development Levels the score for these key dimensions. These key variables include formal policy, regular funding, having a 1. The country team or consultant collects information permanent assessment unit, and the quality of about the assessment system in the country. assessment practices. 2. Based on the collected information, a level of development and score is assigned to each dimension in the rubrics: x Latent = 1 score point x Emerging = 2 score points x Established = 3 score points x Advanced = 4 score points 3. The score for each quality driver is computed by aggregating the scores for each of its constituent dimensions. For example: The quality driver “Enabling Context,” in the case of ILSA, has three dimensions on which a hypothetical country receives the following scores: Dimension A = 2 points; Dimension B = 2 points; Dimension C = 3 points. The hypothetical country’s overall score for this quality driver would be (2+2+3)/3 = 2.33. 4. A preliminary level of development is assigned to each quality driver. 5. The preliminary development level is validated using expert judgment in cooperation with the country team and the World Bank Task Team Leader. For scores that allow a margin of discretion (e.g., to choose between two levels of development), a final decision has to be made based on expert judgment. For example, the aforementioned hypothetical country has an “Enabling Context” score of 2.33, corresponding to a preliminary level of development of “Emerging or Established.” Based on qualitative information not captured in the rubric, along with expert judgment, the country team chooses “Emerging” as the most appropriate level. 6. Scores for certain key dimensions under “Enabling Context” (in the case of EXAM, NLSA, and ILSA) and under “System Alignment” (in the case of CLASS) were 20 BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 Appendix 4: Stakeholders Consulted x Chemistry teacher, leader of the oblast and during Completion of the SABER–Student rayon creative and methodological group of teachers of chemistry, Belozersk Secondary Assessment Questionnaires in Belarus School no. 1 x Chemistry teacher and methodologist, head Classroom Assessment Questionnaire of rayon methodological society of teachers of x Deputy Director for Educational Activities, science, Brest Liceum no. 1 n.a. A.S. Pushkin Secondary School no. 161, Minsk x Top-rank chemistry teacher, head of x Head of Department for Methodology of methodological society of teachers of science, Education in the Sphere of History and Social oligorsk Gimnazium no. 3 Studies and Socio-Cultural Sphere, National Institute of Education x Top-rank methodologist, in charge of professional improvement of teachers of x Deputy Head of Education Quality Monitoring mathematics, Department for Education and Department, National Institute of Education Methodology, Academy of Postgraduate x Head of Department of Auxiliary Historical Education Disciplines and Methods of Teaching History, x Top-rank methodologist, in charge of Belarus State Pedagogical University n.a. professional improvement of teachers of Maksim Tank foreign languages, Department for Education x Director, Domatkanovichy Secondary School and Methodology, Academy of Postgraduate of Kletsky Rayon Education x Deputy Director, Kartsevichy Secondary School of Nesvizh Rayon Examinations Questionnaire x Deputy Director, Slutsk Secondary School no. x Head of the General Secondary Education 2 Department, Ministry of Education of the x Director, Krupitskaya Secondary School of Republic of Belarus Minsk Rayon x Director, Republican Institute for Knowledge x Top-ranked methodologist in charge of Control professional improvement of chemistry x First Deputy Director, Republican Institute for teachers, Department for Education and Knowledge Control Methodology, Academy of Postgraduate x Head of the Department for Monitoring of the Education Quality of Education, National Institute of x Mathematics teacher, Head of Subject and Education Methodology Division, Secondary School no. x Graduate of the 2008 PEE Secondary School 161, Minsk no. 161, Minsk x History teacher, member of the national club, x Graduate of the 2011 PEE Secondary School “Crystal Stork,” Secondary School no. 161, no. 161, Minsk Minsk x Deputy Director for Education, Secondary National Large-Scale Assessment School no. 183, Minsk Questionnaire x Head of Secondary School Education x Winner of the national professional Department, Ministry of Education of the excellence competition, chairperson of the Republic of Belarus “Crystal Stork” club, Biology teacher, Smorgon x Head of Department of Education Quality Gymnasium no. 4 Monitoring, National Institute of Education x Mother of 2015 graduate of Nesvizh x Deputy Head of Department of Educational Secondary School no. 4 Quality Monitoring, National Institute of x Teacher of Russian language and literature, Education Nesvizh Secondary School no. 4 21 BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 x Leading researcher of Laboratory of Mathematics and Natural Science Education, National Institute of Education x Head of Department of Education Sociology and Continuing Education, Academy of Postdiploma Education x Head of Department of Cooperation with Education Sector Stakeholders, Grodno Oblast Institute of Education Development x Head of Department of Auxiliary Sciences of History and Methods of Teaching History, Maxim Tank Belarusian State Pedagogical University x Deputy Head of the Education Department, Moskovskiy District administration of Minsk x Head of the Pre-school and Secondary School Department, Moskovskiy District of Minsk x First Rank Administration Methodologist of teaching and methodology of SEU Academy of Postdiploma Education in charge of chemistry teachers’ continuing education x Deputy Director for Academic Work of SEU Minsk Grammar School no. 174 International Large-Scale Assessment Questionnaire x Head of Secondary Education Department, Ministry of Education of the Republic of Belarus x Director, Republican Institute for Knowledge Control x Head of Education Quality Monitoring Department, National Institute of Education 22 BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 Appendix 5: SABER–Student Assessment Rubrics for Belarus This appendix provides the completed SABER–Student Assessment rubrics for each type of assessment activity in Belarus. In each row of the rubric, the relevant selection is indicated by a shaded cell. The selection may include a superscript number that refers to the justification or explanation for the selection. The explanation or justification text is located in the “Development-level rating justifications” section at the end of each rubric. If a row includes a superscript, but no shading, this means that insufficient information was available to determine the relevant selection in the row. Belarus Classroom Assessment 23 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 2014 Rubric for judging development level of Classroom Assessment LATENT EMERGING ESTABLISHED ADVANCED Indicator Absence of, or deviation On way to meeting minimum Acceptable minimum Best practice from, the attribute standard standard Curriculum/Standards There was no official document There was an official document There was an official document at There was an official document at at the system level that outlined at the system level, but it the system level that provided the system level that provided what students were expected to provided limited and insufficient sufficient, but not extensive extensive and comprehensive learn. information on what students information on what students information on what students were expected to learn. were expected to learn. were expected to learn.1 Policy Document There was no document at the There was a document at the There was an official and publicly There was an official and publicly system level that provided system level that provided available document at the system available document at the system guidelines for classroom guidelines for classroom level that provided sufficient, but level that provided extensive and assessment. assessment, but it was either not extensive, guidelines for comprehensive guidelines for unofficial, not publicly available, classroom assessment. classroom assessment.2 or provided limited guidance. Resources There were no resources There were resources available There were sufficient high-quality There was an extensive number of available to teachers in the to teachers in the system for resources available to all or high-quality resources available to system for their use in classroom their use in classroom almost all teachers in the system all or almost all teachers in the assessment activities. assessment activities, but these for their use in classroom system for their use in classroom resources were not of high assessment activities. assessment activities. 3 quality and were limited in number or availability. (CONTINUED) 24 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 LATENT EMERGING ESTABLISHED ADVANCED Indicator Absence of, or deviation from, On way to meeting minimum Acceptable minimum standard Best practice the attribute standard Teacher There were no formal mechanisms There was a minimum number of There were sufficient formal There was an extensive number of Development at the system level that supported formal mechanisms at the system mechanisms of high quality at the high-quality, formal mechanisms at the development of teachers’ level that supported the system level that supported the the system level that supported the competencies in classroom development of teachers’ development of teachers’ development of teachers’ assessment. competencies in classroom competencies in classroom competencies in classroom assessment, or else the available assessment.4 assessment. formal mechanisms were not of high quality or were limited in their availability. Quality There were no formal mechanisms There was a minimum number of There were sufficient formal There were extensive formal Monitoring at the system level to monitor the formal mechanisms at the system mechanisms at the system level to mechanisms at the system level to quality of classroom assessment level to monitor the quality of monitor the quality of classroom monitor the quality of classroom practices. classroom assessment practices. assessment practices, including assessment practices, including inspection/supervision.5 inspection/supervision. Report to Schools were not required to Schools had minimum Schools were required to report Schools were required to report Stakeholders report individual student requirements to report individual individual student performance on individual student performance on performance on classroom student performance on classroom classroom assessments to the classroom assessments to a variety assessments to any stakeholders. assessments. student and their parents.6 of relevant stakeholders. (CONTINUED) 25 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 LATENT EMERGING ESTABLISHED ADVANCED Indicator Absence of, or deviation from, On way to meeting minimum Acceptable minimum standard Best practice the attribute standard Report Content Schools were not required to Schools were required to report Schools were required to report This option does not apply to this report individual student individual student performance on individual student performance on indicator. performance on classroom classroom assessments in one or classroom assessments in more than assessments in particular subject two subject areas. two subject areas.7 areas. Report Format There were no requirements for There were requirements for There were requirements for schools There were requirements for schools to use specific formats for schools to use specific formats for to use specific formats for reporting schools to use specific formats for reporting individual student reporting individual student individual student performance on reporting individual student performance on classroom performance on classroom classroom assessments to the performance on classroom assessments. assessments to the students and students and their parents, including assessments to students, parents, their parents, but the formats written reports and teacher/school and the school district, Ministry of specified did not include written meetings. There also were Education, or equivalent. These reports and teacher/school requirements for schools to report included written reports and (in meetings.8 this information to the school district, the case of students and parents) Ministry of Education, or equivalent teacher/school meetings. although the format for reporting to these entities was not specified. Required Uses There were no system-level Teachers were required to use Teachers were required to use Teachers were required to use requirements for teachers to use classroom assessment classroom assessment information in classroom assessment classroom assessment information. information, albeit in a minimal a sufficient number of ways. information in extensive ways.9 number of ways. (CONTINUED) 26 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 LATENT EMERGING ESTABLISHED ADVANCED Indicator Absence of, or deviation from, On way to meeting minimum Acceptable minimum standard Best practice the attribute standard Selection and At the secondary level, classroom This option does not apply to this At the secondary level, classroom This option does not apply to this Certification assessment information was not indicator. assessment information was indicator. required as an input for required to be used as an input for certification decisions or for certification decisions or for selection to the next level of the selection to the next level of the education system. education system.10 Positive Uses Classroom assessment information Classroom assessment information Classroom assessment information Classroom assessment information was used in positive ways by a was used in positive ways by some was used in positive ways by most was used in positive ways by all or marginal number of teachers. teachers. teachers.11 almost all teachers. Poor Practices All or almost all teachers engaged Many teachers engaged in poor Only some teachers engaged in A marginal number of teachers or in poor classroom assessment classroom assessment practices. poor classroom assessment no teachers engaged in poor practices. practices. classroom assessment practices.12 27 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 Classroom Assessment: Development-level rating justifications 1. “Educational Standards for School Subjects, 2009” is an official learning standards document at the system level that outlines what students at different grade or age levels are expected to learn, but does not specify the desired performance level. The document covers subjects such as language (Belarusian and Russian), mathematics, biology, geography, world history, history of Belarus, Russian literature, Belarusian literature. Official system-level documents also include “Educational Standards for School Subjects” (2012, 2015), which outlines what students at different grade or age levels are expected to learn and the desired performance level, and the “Educational Programs for Extracurricular Subjects, 2009–2013,” which outlines what students at different grade or age levels are expected to learn but does not specify the desired performance level. 2. Formal, official documents at the system level that provide guidance for classroom assessment and are available to the general public include: - “Assessment Norms for Outcomes of Educational Activity of Students in Educational Subjects”: this document describes (a) what should be assessed, (b) criteria for scoring student work, (c) formats and methods for carrying out classroom assessment activities, (d) description of significant and insignificant errors in student responses, (e) performance evaluation of students on a 10-point scale, (f) assessment tests and tasks scale to translate points for the test into marks, (g) mechanisms for issuing test scores for subjects. - Instructions and methodological letter “On Organization of Work of General Secondary Education Institutions to Monitor and Evaluate the Outcomes of Learning Activity of Students during the Ungraded Learning Stage I of General Secondary Education”: this document describes (a) what should be assessed, (b) principles of ungraded training, (c) a mechanism to evaluate the learning process, (d) types and forms of control under ungraded training, (e) methods of formation of adequate student self-esteem, (f) record keeping by teachers and students, and administration of educational institutions, (g) rights and responsibilities of students and teachers, and the legal representatives of pupils. - “Rules for Certification of Pupils at Mastering Content Educational Programs of General Secondary Education”: this document describes (a) uses of assessment information, (b) the form of certification in secondary education institutions, (c) procedure for promotion to the next grade, (d) procedure for revising the annual grade. 3. Extensive resources are available to teachers in the system for their classroom assessment activities. Available resources include: A. A document that outlines what students are expected to learn in different assessment domains at different grade/age levels B. A document that outlines the performance level(s) that students are expected to reach in different assessment domains at different grade/age levels C. Student textbooks that provide support for classroom assessment (all teachers receive textbooks that contain questions and sample material for conducting classroom assessment activities). D. Teacher guides (compilations of materials to be used for conducting continuous, intermediate, and final assessment have been published for all school subjects in grades 1–11 and are available for purchase; all subject teachers receive funds specifically to acquire the methodological literature). E. Scoring criteria or rubrics for grading student work (diagnostic assessment modules for all subjects in grades 1–11 are available on the national education portal). All participants in the education process, including students, parents, and teachers, have access to these resources. 28 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 F. Item banks/pools with examples of questions/tasks to be used for classroom assessment activities G. Computer-based classroom assessment resources (diagnostic assessment modules have been development for all subjects in grades 1–11 and made available on the national education portal. All stakeholders in the education process, including students, parents, and teachers, have free access to these modules). Resources A, B, and E are of high quality and available to all or almost all teachers Resource C is of medium quality and available to all or almost all teachers Resource D is of high quality and available to most teachers Resource F is of high quality and available to all or almost all teachers Resource G is of high quality and available to all or almost all teachers. 4. Extensive high-quality mechanisms are in place at the system level to support the development of teachers’ competencies in classroom assessment and are available on an annual basis. Available mechanisms include: A. Preservice teacher training that addresses competencies in classroom assessment. This mechanism is available to all or almost all teachers, although not on an annual basis. This mechanism is considered to be of medium quality. B. In-service teacher training that addresses competencies in classroom assessment. This mechanism is available to all or almost all teachers, though not on an annual basis, and is considered to be of high quality. C. Online resources on classroom assessment. This mechanism is available to all or almost all teachers on an annual basis and is considered to be of high quality. D. Opportunities to participate in conferences and workshops on classroom assessment. This mechanism is available to all or almost all teachers on an annual basis and is considered to be of medium quality. E. Opportunities to participate in the development or scoring of test questions for large-scale assessments. This mechanism is considered to be of high quality but is available to only a marginal number of teachers and not on an annual basis. F. Implementation of experimental and innovative projects on piloting and introduction of new technologies for evaluation of students’ achievements. This mechanism is considered to be of high quality but is available to only a marginal number of teachers and not on an annual basis. G. Distance learning for teachers in classroom assessment. This mechanism is considered to be of high quality and is available to only a marginal number of teachers on an annual basis. H. Self-education of teachers in classroom assessment. This mechanism is available to all teachers on an annual basis but is considered to be of medium quality. Although all or almost all (more than 90 percent) teachers can participate in A, B, C, D, and H, a marginal number (less than 10 percent) of teachers can participate in E, F, and G. 29 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 5. Formal mechanisms exist at the system level to monitor the quality of classroom assessment practices. Four mechanisms are found, which is less than extensive (five or six mechanisms) and more than minimal (one or two mechanisms). Mechanisms include the following: (a) Classroom assessment is a required component of school inspection/teacher supervision, (b) classroom assessment is a required component of a teacher’s performance evaluation, (d) regular, high-level reviews of the quality of education include a focus on classroom assessment, and (e) government funding is awarded to researchers and practitioners for research and development on classroom assessment practices. An external moderation system that consists of commissions in educational subjects at the district level that objectively evaluate the assessment of written examination questions of those students who are being considered to receive an attestation of general secondary education with an award of a gold (or silver) medal is also in place. At the same time, a more wide-reaching moderation system, or a moderation system where teachers from different schools meet to review assessment tasks, scoring criteria, and students’ work, with the purpose of ensuring the validity and comparability of student scores and grades is not in place, is not in place. 6. All or almost all schools report individual student performance on classroom assessment in all subject areas to students and their parents through school- wide and class-specific parent-teacher conferences. Student grades are recorded in a classroom journal and in the student’s grade journal (dnevnik). Parents are required to review the student’s grade journal every week and every quarter. There is currently a move toward maintaining the classroom journal and the student’s grade journal in electronic form. Schools are not required to report to school district, Ministry of Education, or equivalent. 7. Schools are required to report individual student performance in all subjects, including language and mathematics. 8. Schools report individual student performance on classroom assessment in all subject areas to students and their parents through schoolwide and class- specific parent-teacher conferences. Student grades are recorded in a classroom journal and in the student’s grade journal (dnevnik). Parents are required to review the student’s grade journal every week and every quarter. There is currently a move toward maintaining the classroom journal and the student’s grade journal in electronic form. Schools are not required to report individual student performance information to school district, Ministry of Education, or equivalent entity in any format, including written and electronic. 9. Classroom assessment is required to be used in an extensive number (four) of ways by teachers, specifically, to: - Diagnose student learning issues - Provide continuous feedback to students as part of instruction - Plan the next steps in instruction - Evaluate student performance 10. Classroom assessment information is required to be used for certification and as an input for selection to the next level within the education system. 30 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 11. Classroom assessment information is used in positive ways by most (50–90 percent) teachers. Specifically, teachers use classroom assessment information to: - Diagnose student learning issues (most teachers) - Provide continuous feedback to students as part of instruction (most teachers) - Plan the next steps in instruction (most teachers) - Evaluate student performance (all or almost all teachers) 12. Overall, a marginal number of teachers engage in poor classroom assessment practices. No teachers use assessment tools that are not aligned with the pedagogical/curriculum framework; a marginal number of teachers (less than 10 percent) apply uneven standards for grading student work; some teachers inflate grades; some teachers overemphasize memorization/recall of information; a marginal number of teachers (less than 10 percent) overemphasize the use of multiple choice/selection-type questions; an insignificant number of teachers (less than 5 percent) make errors in the scoring or grading of student work. 31 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 BELARUS Examinations 32 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 LATENT EMERGING ESTABLISHED ADVANCED Indicator Absence of, or deviation from, On way to meeting minimum Acceptable minimum standard Best practice the attribute standard Program No examination program existed at An examination program existed at A stable examination program had A stable examination program had Stability the system level. the system level, but it was not been in place for several years. been in place for 10 years or more.1 sufficiently stable. Clarity of There were no policy-mandated The examination had clear policy- The examination had clear policy- This option does not apply to this Purpose purposes of the examination. mandated purposes, but these did mandated purposes that included indicator. not include student certification or student certification, selection, or selection. both.2 Policy No policy document authorized the An informal/draft policy document A formal/official policy document A formal/official policy document Document examination program. authorized the examination authorized the examination authorized the examination program. program, but the document was program and was available to the not available to the general public. general public.3 Program No official document provided An official document provided An official document provided key This option does not apply to this Guidelines guidelines for the examination guidelines for the examination guidelines for the examination indicator. program. program, but it was missing some program.4 key guidelines. Stability of There was no unit with primary There was a unit(s) with primary There was a permanent unit(s) with This option does not apply to this Organization responsibility for running the responsibility for running the primary responsibility for running indicator. examination program. examination program, but the the examination program that had unit(s) was temporary or had been been in place for 5 or more years.5 in place for less than 5 years. (CONTINUED) 33 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 LATENT EMERGING ESTABLISHED ADVANCED Indicator Absence of, or deviation from, On way to meeting minimum Acceptable minimum standard Best practice the attribute standard Accountability There was no unit with primary The unit(s) with primary The unit(s) with primary The unit(s) with primary of Organization responsibility for running the responsibility for running the responsibility for running the responsibility for running the examination program, or else the examination program was examination program was examination program was unit responsible was not accountable to a clearly recognized accountable to a clearly recognized accountable to a clearly recognized accountable to a clearly recognized body within the examination unit. body within the same institution as external body.6 body. the examination unit. Organization The examination unit did not have The examination unit had some of The examination unit had most of The examination unit had all of the Resources the appropriate resources. the appropriate resources. the appropriate resources. appropriate resources.7 Qualifications There were no individuals Some of the individuals Most of the individuals responsible All or almost all of the individuals of Staff responsible for completing key responsible for completing key for completing key examination responsible for completing key examination activities. examination activities had the activities had the relevant examination activities had the relevant qualifications. qualifications. relevant qualifications.8 Effectiveness of There were no individuals The responsible individuals The responsible individuals The responsible individuals Staff responsible for completing key completed key examination completed key examination completed key examination examination activities. activities, but there were activities, with only some issues in activities, and there were no issues significant issues in how these how these activities were in how these activities were activities were completed. completed. completed.9 Source of There was no funding available for The source of funding for the The source of funding for the This option does not apply to this Funding examination activities. majority of examination activities majority of examination activities indicator. was loans, credits, grants, or was the government’s internal equivalent. funding sources or student fees.10 (CONTINUED) 34 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 LATENT EMERGING ESTABLISHED ADVANCED Indicator Absence of, or deviation from, On way to meeting minimum Acceptable minimum standard Best practice the attribute standard Activities There was no funding available for Funding was not sufficient to cover Funding was sufficient to cover all This option does not apply to this Funded examination activities. all core examination activities. core examination activities.11 indicator. Staff/Teacher There were no opportunities to Opportunities to learn about the There were sufficient high-quality Opportunities to learn about the Opportunity to learn about the examination. examination were minimal, or not opportunities to learn about the examination were extensive, of Learn of high quality, or did not benefit examination that were available to high quality, and benefited key all key stakeholder groups.12 key stakeholder groups. stakeholder groups. Teacher Teachers did not perform Teachers performed a minimal Teachers performed a sufficient Teachers performed an extensive 13 Participation examination-related tasks. number of examination-related number of examination-related number of examination-related tasks. tasks. tasks. Measuring It was not clear what the There was weak alignment The examination measured official The examination measured official What Is examination was intended to between the examination and learning standards or curriculum, learning standards or curriculum, Intended measure. what it was meant to measure, or and officially mandated reviews to and officially mandated reviews to there was no regular review verify this alignment took place verify this alignment took place process in place to verify that during most examination rounds. during all or almost all examination alignment existed. rounds.14 Alignment with The examination was poorly The examination was somewhat The examination was very aligned This option does not apply to this Other aligned with other types of aligned with other types of with other types of assessment indicator. Assessments assessment activities in the system. assessment activities in the activities in the system.15 system. (CONTINUED) 35 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 LATENT EMERGING ESTABLISHED ADVANCED Indicator Absence of, or deviation from, On way to meeting minimum Acceptable minimum standard Best practice the attribute standard Availability of There were no materials available Materials to prepare for the Materials to prepare for the Materials to prepare for the Preparation to students to prepare for the examination were available to examination were available to examination were available to all or Materials examination. some or a marginal number of most students.16 almost all students. students. Quality of There were no materials available Minimal material was available to Sufficient and high-quality material Extensive and high-quality material Preparation to students to prepare for the students to prepare for the was available to students to was available to students to Materials examination. examination, or the material prepare for the examination. prepare for the examination. available was not of high quality.17 Reasons for Not All or almost all individuals could Most or some individuals could not There were no non-examination- This option does not apply to this Taking the not take the examination due to take the examination due to one or relevant reasons that prevented indicator. Examination one or more non-examination- more non-examination-relevant individuals from taking the relevant reason(s). reason(s). examination.18 Quality No formal procedures were in Formal procedures to ensure the Formal procedures to ensure the Formal procedures to ensure the Assurance place to ensure the quality of the quality of the examination were quality of the examination were quality of the examination were examination. minimal in nature or not required. sufficient in nature and required. extensive in nature and required.19 Standardization The examination was not The examination was partially The examination was fully The examination was fully standardized at the system level. standardized at the system level, standardized at the system level, standardized at the system level, or minimal or no procedures were and sufficient procedures were in and extensive procedures were in in place to ensure standardization. place to ensure standardization. place to ensure standardization.20 (CONTINUED) 36 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 LATENT EMERGING ESTABLISHED ADVANCED Indicator Absence of, or deviation from, On way to meeting minimum Acceptable minimum standard Best practice the attribute standard Quality Many errors or delays in activities Errors or delays in activities Any errors or delays in activities Errors or delays in activities did not Processes took place that affected the affected the examination to a had only a minimal effect on the affect the examination.21 examination to a great extent. significant level. examination. Inappropriate Inappropriate behavior Inappropriate behavior took place Inappropriate behavior was low Inappropriate behavior, if any, was Behavior compromised the credibility of the and compromised the credibility of and did not compromise the marginal and did not compromise examination to a great extent. the examination somewhat. credibility of the examination. the credibility of the examination.22 Credibility of The results of the examination The results of the examination The results of the examination The results of the examination Results were perceived as credible by very were perceived as credible by were perceived as credible by most were perceived as credible by all or few stakeholder groups. some stakeholder groups. stakeholder groups. almost all stakeholder groups.23 Confidentiality There was no official policy to keep Confidentiality of student results There was an official policy to keep This option does not apply to this of Results student results confidential, and was partially accomplished. student results confidential, and indicator. student results were not kept student results were kept confidential in practice. confidential in practice.24 Official Examination results were not This option does not apply to this Examination results were officially This option does not apply to this Recognition of officially recognized by educational indicator. recognized by educational indicator. Results institutions or employers in other institutions or employers in other countries.25 countries. (CONTINUED) 37 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 LATENT EMERGING ESTABLISHED ADVANCED Indicator Absence of, or deviation from, On way to meeting minimum Acceptable minimum standard Best practice the attribute standard Post- No options were available to Minimal options were available to Sufficient options were available to Extensive options were available to Examination students after they had taken the students after they had taken the students after they had taken the students after they had taken the Options for examination. examination. examination. 26 examination. Students Methods and There was no documentation on There was minimal documentation There was sufficient and public There was extensive and public Procedures the methods and procedures used on the methods and procedures documentation on the methods documentation on the methods Documentation during the examination. used during the examination, or and procedures used during the and procedures used during the the documentation that existed examination.27 examination. was not public. Impact No mechanisms were in place to Minimal mechanisms were in place Sufficient mechanisms were in Extensive mechanisms were in Monitoring monitor the impact of the to monitor the consequences of place to monitor the impact of the place to monitor the impact of the examination. the examination, or the examination and the mechanisms examination and the mechanisms mechanisms took place in only took place all or almost all took place all or almost all some or a few examination examination rounds. examination rounds. rounds.28 Readiness to The system was weakly prepared to The system was somewhat The system was well prepared to This option does not apply to this Start an start an examination program in prepared to start an examination start an examination program in indicator. 29 Examination the future. program in the future. the future. Program 38 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 Examinations: Development-level rating justifications 1. “Centralized Testing” is an examination program that exists at the system level. This examination was administered for the first time more than 10 years ago (in 2003) and has since been administered annually. 2. Results from Belarus’s national examination program, “Centralized Testing,” are used to select individuals into higher and specialized secondary education institutions. Centralized Testing can be taken by those individuals who have completed the educational program of general secondary education, of the professional-technical education that provides vocational qualification and general secondary education, or of specialized secondary education as confirmed by a document of completion of such education. Centralized Testing is also used to inform policy and for evaluating interventions aimed at improving student learning. 3. The Resolution of the Council of Ministers of the Republic of Belarus no. 714, “On Approval of the Regulation on the Procedure for Organizing and Holding Centralized Testing” (June 6, 2006), is the formal, publicly available, system-level policy document that authorizes the examination program. 4. A number of formal, publicly available system-level documents provide guidelines on the examination, including what should be assessed, who should be assessed, and how the results should be communicated to stakeholders. These documents include the Resolution of the Council of Ministers of the Republic of Belarus no. 714, “On Approval of the Regulation on the Procedure for Organizing and Holding Centralized Testing” (June 6, 2006); “Rules of Admission of Persons for the 1st Stage of Higher Education (as amended by Presidential Decree of 03.20.2014, no. 130)”; the Resolution of the Council of Ministers of the Republic of Belarus no. 19, “On Conducting the Centralized Testing in Academic Subjects in 2015” (April 13, 2015); and the Order of the Minister of Education of the Republic of Belarus no. 323, “On Organization of Centralized Testing in 2015” (April 22, 2015). 5. The unit with primary responsibility for running the examination program is the Republican Institute for Knowledge Control (RIKC), a permanent agency that took charge of the examination more than 10 years ago. Specifically, since 2000, RIKC has been responsible for examination design, administration, project management, data processing, and research and development. RIKC is accountable to the Ministry of Education. The Ministry of Education also provides guidance on the rules of computing examination scores and the organizational rules at the centers carrying out the Centralized Testing. The Ministry of Education is subordinate to the Council of Ministers of the Republic of Belarus. 6. RIKC, which has the primary responsibility for running the Centralized Testing program, is accountable to the Ministry of Education, from which it is institutionally separate. 7. RIKC currently has all of the appropriate resources, including appropriate computers for all technical staff, software, building security, storage facilities, computer servers, and communication tools (phone, email, Internet). 39 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 8. All or almost all (more than 90 percent) of the individuals currently responsible for completing key examination activities have relevant qualifications. Individuals responsible for completing key examination activities include: - Permanent staff from the examination unit - Temporary staff hired to support a particular examination cycle (e.g., examination administrators) 9. No issues are found with the effectiveness of any of the staff, or else minor issues were identified that have had no consequences for the quality of specific examination activities or for the overall quality of the examination. 10. Funding is available for examination activities. The source of funding for the majority of examination activities is government funding. Funding for examination activities supports examination design, administration, data processing, data reporting, project management, and research and development activities. These activities are supported by the government budget. Data processing is further supported by RIKC extra-budgetary funds. 11. Funding is sufficient to cover all core examination activities as well as research and development. 12. There are sufficient opportunities to learn about the examination. Opportunities to learn did not include university graduate programs. Opportunities to learn about examinations include university courses/workshops on the content and skills measured by the examination; non-university courses/workshops on examination topics; funding for attending international programs/courses/workshops on student assessment that cover topics relevant to the examination; presentations; instruction and methodology meetings with members of organizing committees working in venues of centralized testing; publications (articles, collections of materials for centralized testing). According to the data provided in the completed SABER–Student Assessment questionnaire, opportunities are generally of high quality and are available to students studying in higher education institutions, existing full-time staff in the examination unit (RIKC), university professors, general secondary school educators, and specialists in the education management bodies. The following opportunities were not available in the system to learn about the examination: university graduate programs (master’s or doctorate level) on student assessment that include topics relevant to the examination (e.g., test design, administration); non-university courses/workshops on the content and skills measured by the examination (e.g., courses on curriculum); university courses/workshops on examination topics other than the content and skills measured by the examination (e.g., test design, administration); internships/short-term employment in the unit running the examination. 13. Teachers do not perform examination-related tasks. 40 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 14. The Centralized Testing program is intended to measure official learning standards and curriculum. In practice, the examination is very aligned with what it is intended to measure. Officially mandated reviews to ensure that the examination measures what is intended to measure take place in the form of regular external reviews and regular internal reviews, both of which take place in all or almost all examination rounds (more than 90 percent). 15. The Centralized Testing program is closely aligned with Classroom Assessments and National Large-Scale Assessments in Belarus in terms of the content being measured. 16. Materials were available to students to prepare for the examination. On average, these materials were available to most (50 to 90 percent) students. Available materials included (a) the official framework document explaining what was measured on the examination; (b) examples of the types of questions that were on the examination, provided by RIKC; (c) examples of the types of questions that were on the examination, provided by an entity other than RIKC; (d) information on how to prepare for the examination, provided by RIKC; and (e) information on how to prepare for the examination, provided by an entity other than RIKC, such as by higher education institutions. A, b, and d were available to all or almost all students; c and e were available to some students. 17. Although the number of materials available to prepare for the examination was extensive, material content was of “medium” quality overall. 18. Individuals with the following profiles do not face barriers that might prevent otherwise eligible individuals from taking the examination: ethnic, racial, or linguistic group identification; gender; or location (e.g., rural or hard-to-reach areas). Individuals with vision, hearing, or musculoskeletal disorders do not take part in the Centralized Testing program. Instead, such individuals can participate in entrance examinations in the institutions where they would like to be considered for admission, with the format of the examination being determined by the institution taking into account the psychological development of the candidate. 19. Formal quality assurance procedures were in place to ensure the quality of the examination. These procedures were extensive in number (six or seven), and all procedures were required to take place every examination round. The following formal quality assurance procedures were in place to ensure the quality of the examination: - All proctors or administrators were trained according to a protocol - There was a standardized manual for examination administrators - Questions/items/tasks were piloted before the official examination administration - There were external observers - There were internal observers 41 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 - A process was outlined by which control of the organization and conduct of Centralized Testing is performed by the State Control Commission for the preparation and conduct of entrance examinations to institutions of higher and secondary specialized education, as approved by a Presidential Edict. No double processing of data was in place. 20. The examination was fully standardized at the system level. Assessment design, administration, scoring, and reporting were the same (or equivalent) for all students in the examination round. Extensive procedures (five or six) were in place to ensure the standardization of the examination at the system level including: - Examination papers and tasks were the same or equivalent for all students - Examination administrators were trained to ensure that all students took the examination under the same conditions - Quality control monitors/observers were used to ensure the same administration conditions in all locations where the examination was administered - The same scoring criteria were used to correct the examination questions/items/tasks - Examination results were computed using the same procedures for all students - Examination results were reported to all students in the same way 21. No issues affected the examination round. There were instances of errors in the printing of test booklets; however, this did not affect the examination round. 22. Although the use of unauthorized materials such as prepared answers and notes as well as collusion among candidates via mobile phones, passing of papers, or equivalent did take place, the credibility of the examination was not compromised at all because of these inappropriate behaviors. 23. All or almost all stakeholder groups perceive the examination results as credible. 24. Students’ Centralized Testing results were kept confidential, in line with official policy. 25. Examination results were not officially recognized by educational institutions or employers in other countries. 26. Sufficient options are available to students after they take the examination and receive their results. Students can apply to tertiary education institutions, secondary specialized institutions, or vocational education institutions. Individuals may also retake the examination in a future year. 27. There was documentation on the methods and procedures used during the examination. This documentation was sufficient and for the most part available to the general public. Documented procedures include: 42 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 - Test specifications (publicly available) - Construction of questions/items/tasks (publicly available) - Test assembly (publicly available) - Scoring of examination questions/items/tasks (publicly available) - Reliability (not publicly available) - Scaling (not publicly available) The minimum scores for the Centralized Testing (which allow for individuals to apply to intuitions of higher education) are provided on an annual basis by the Ministry of Education. The following aspects of the examination are not documented: pilot testing of questions/items/tasks; analysis of piloted questions/items/tasks; marking/scoring of open-ended questions/items/tasks. 28. To monitor the impact of the Centralized Testing, RIKC produces an Annual Report for all examination rounds. Additionally, the Order of the President of the Republic of Belarus provides for the creation of a commission on the control of the preparation and implementation of entrance examinations into institutions of higher and secondary specialized education. The commission oversees the preparation and implementation of Centralized Testing every year. 29. This indicator does not apply to this rubric. 43 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 BELARUS National (or System-Level) Large-Scale Assessment (NLSA) 44 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 LATENT EMERGING ESTABLISHED ADVANCED Indicator Absence of, or deviation On way to meeting minimum Acceptable minimum Best practice from, the attribute standard standard Program Stability No NLSA program existed at the An NLSA program existed at the A stable NLSA program had been A stable NLSA program had been system level. system level, but it was not in place for several years. in place for 10 years or more.1 sufficiently stable. Clarity of Purpose There were no policy-mandated The NLSA had clear policy- The NLSA had clear policy- This option does not apply to this purposes of the NLSA. mandated purposes, but these mandated purposes that included indicator. did not include informing policy informing policy or pedagogy.2 or pedagogy. Policy Document No policy document authorized An informal/draft policy A formal/official policy document A formal/official policy document the NLSA program. document authorized the NLSA authorized the NLSA program, authorized the NLSA program and program. but the document was not was available to the general available to the general public. public.3 Program Guidelines No official document provided An official document provided An official document provided This option does not apply to this guidelines for the NLSA program. guidelines for the NLSA program, key guidelines for the NLSA indicator. but it was missing some key program.4 guidelines. Stability of There was no unit with primary There was a unit(s) with primary There was a permanent unit(s) This option does not apply to this Organization responsibility for running the responsibility for running the with primary responsibility for indicator. NLSA program. NLSA program, but the unit(s) running the NLSA program that was temporary or had been in had been in place for 5 or more place for less than 5 years. years.5 45 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 LATENT EMERGING ESTABLISHED ADVANCED Indicator Absence of, or deviation On way to meeting minimum Acceptable minimum Best practice from, the attribute standard standard Accountability of There was no unit with primary The unit(s) with primary The unit(s) with primary The unit(s) with primary Organization responsibility for running the responsibility for running the responsibility for running the responsibility for running the NLSA program, or else the unit NLSA program was accountable NLSA program was accountable NLSA program was accountable responsible was not accountable to a clearly recognized body to a clearly recognized body to a clearly recognized external to a clearly recognized body. within the NLSA unit. within the same institution as the body. NLSA unit.6 Source of Funding There was no funding available The source of funding for the The source of funding for the This option does not apply to this for NLSA activities. majority of NLSA activities was majority of NLSA activities was indicator. loans, credits, grants, or the government’s internal equivalent. funding sources.7 Activities Funded There was no funding available Funding was not sufficient to Funding was sufficient to cover all This option does not apply to this for NLSA activities. cover all core NLSA activities. core NLSA activities.8 indicator. Organization The NLSA unit did not have the The NLSA unit had some of the The NLSA unit had most of the The NLSA unit had all of the Resources appropriate resources. appropriate resources. appropriate resources.9 appropriate resources. 46 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 Qualifications of There were no individuals Some of the individuals Most of the individuals All or almost all of the individuals Staff responsible for completing key responsible for completing key responsible for completing key responsible for completing key NLSA activities. NLSA activities had the relevant NLSA activities had the relevant NLSA activities had the relevant qualifications. qualifications. qualifications.10 47 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 LATENT EMERGING ESTABLISHED ADVANCED Indicator Absence of, or deviation On way to meeting minimum Acceptable minimum Best practice from, the attribute standard standard Effectiveness of Staff There were no individuals The responsible individuals The responsible individuals The responsible individuals responsible for completing key completed key NLSA activities, completed key NLSA activities, completed key NLSA activities, NLSA activities. but there were significant issues with only some issues in how and there were no issues in how in how these activities were these activities were completed. these activities were completed. completed.11 Staff/Teacher There were no opportunities to Opportunities to learn about the There were sufficient high-quality Opportunities to learn about the Opportunity to Learn learn about the NLSA. NLSA were minimal, or not of opportunities to learn about the NLSA were extensive, of high high quality, or did not benefit all NLSA that were available to key quality, and benefited key key stakeholder groups. stakeholder groups.12 stakeholder groups. Measuring What Is It was not clear what the NLSA There was weak alignment The NLSA measured official The NLSA measured official Intended was intended to measure. between the NLSA and what it learning standards or curriculum, learning standards or curriculum was meant to measure, or there and officially mandated reviews and officially mandated reviews was no regular review process in to verify this alignment took place to verify this alignment took place place to verify that alignment during most NLSA rounds. during all or almost all NLSA existed. rounds.13 Alignment with The NLSA was poorly aligned with The NLSA was somewhat aligned The NLSA was very aligned with This option does not apply to this Other Assessments other types of assessment with other types of assessment other types of assessment indicator. activities in the system. activities in the system. activities in the system.14 Opportunities for Students did not have Students had limited Students had sufficient Students had many opportunities Students to Be opportunities to be exposed to opportunities to be exposed to opportunities to be exposed to to be exposed to the content and Exposed to Content the content and skills measured the content and skills measured the content and skills measured skills measured by the NLSA.15 and Skills by the NLSA. by the NLSA. by the NLSA. (CONTINUED) 48 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 LATENT EMERGING ESTABLISHED ADVANCED Indicator Absence of, or deviation On way to meeting minimum Acceptable minimum Best practice from, the attribute standard standard Preparatory Official information on the NLSA A minimal amount of official A sufficient amount of official An extensive amount of official Information for was not made available to schools information on the NLSA was information on the NLSA was information on the NLSA was Schools in the system. made available to schools in the made available to most or almost made available to all or almost all system, although not necessarily all schools in the system. schools in the system. all schools.16 Quality Assurance No formal procedures were in Formal procedures to ensure the Formal procedures to ensure the Formal procedures to ensure the place to ensure the quality of the quality of the NLSA were minimal quality of the NLSA were quality of the NLSA were NLSA. in nature or not required. sufficient in nature and extensive in nature and required. required.17 Standardization The NLSA was not standardized at The NLSA was partially The NLSA was fully standardized The NLSA was fully standardized the system level. standardized at the system level, at the system level, and sufficient at the system level, and extensive or minimal or no procedures procedures were in place to procedures were in place to were in place to ensure ensure standardization. ensure standardization.18 standardization. Representativeness A nonrandom sample or a A random sample of students All students in public and private All students in public schools, or a convenience sample of students that was not representative at representative sample of schools, or a representative participated in the NLSA. the country level participated in students in public schools, sample of students in public and the NLSA. participated in the NLSA.19 private schools, participated in the NLSA. Reasons for Not All or almost all individuals could Most or some individuals could There were no non-assessment- This option does not apply to this Taking the NLSA not take the NLSA due to one or not take the NLSA due to one or relevant reasons that prevented indicator. more non-assessment-relevant more non-assessment-relevant individuals from taking the reason(s). reason(s). NLSA.20 (CONTINUED) 49 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 LATENT EMERGING ESTABLISHED ADVANCED Indicator Absence of, or deviation On way to meeting minimum Acceptable minimum Best practice from, the attribute standard standard Quality Processes Many errors or delays in activities Errors or delays in activities Any errors or delays in activities Errors or delays in activities did took place that affected the NLSA affected the NLSA to a significant had only a minimal effect on the not affect the NLSA.21 to a great extent. level. NLSA. Inappropriate Inappropriate behavior Inappropriate behavior took Inappropriate behavior was low Inappropriate behavior, if any, Behavior compromised the credibility of place and compromised the and did not compromise the was marginal and did not the NLSA to a great extent. credibility of the NLSA credibility of the NLSA. compromise the credibility of the somewhat. NLSA.22 Methods and There was no documentation on There was minimal There was sufficient and public There was extensive and public Procedures the methods and procedures documentation on the methods documentation on the methods documentation on the methods Documentation used during the NLSA. and procedures used during the and procedures used during the and procedures used during the NLSA, or the documentation that NLSA. NLSA. existed was not public.23 Publication of NLSA results were not published. Limited information on the NLSA Sufficient information on the Comprehensive information on Results results was published, or the NLSA results was published using the NLSA results was published results were published using a an array of dissemination using an array of dissemination minimum number of mechanisms.24 mechanisms. dissemination mechanisms. Credibility of Results The results of the NLSA were The results of the NLSA were The results of the NLSA were The results of the NLSA were perceived as credible by very few perceived as credible by some perceived as credible by most perceived as credible by all or stakeholder groups. stakeholder groups. stakeholder groups.25 almost all stakeholder groups. (CONTINUED) 50 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 LATENT EMERGING ESTABLISHED ADVANCED Indicator Absence of, or deviation On way to meeting minimum Acceptable minimum Best practice from, the attribute standard standard Impact Monitoring No mechanisms were in place to Minimal mechanisms were in Sufficient mechanisms were in Extensive mechanisms were in monitor the impact of the NLSA. place to monitor the place to monitor the impact of place to monitor the impact of consequences of the NLSA, or the the NLSA, and the mechanisms the NLSA, and the mechanisms mechanisms took place in only took place in all or almost all NLSA took place in all or almost all NLSA some or a few NLSA rounds.26 rounds. rounds. Readiness to Start an The system was weakly prepared The system was somewhat The system was well prepared to This option does not apply to this NLSA Program to start an NLSA program in the prepared to start an NLSA start an NLSA program in the indicator. 27 future. program in the future. future. 51 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 National (of System-Level) Large Scale Assessment (NLSA): Development-level rating justifications 1. The Republican Monitoring of students’ learning achievement in school subjects was first implemented in 2003, and until 2011 it covered grades 4, 9, and 11. In the 2012–13 and 2013–14 school years, it covered grades 3, 8, and 10. Between 2003 and 2014, students’ knowledge and skills were assessed in the subjects of mathematics and Belarusian and Russian languages. The assessment also covered biology, chemistry and physics in grades 9 and 11 (in 2003–11) and in grades 8 and 10 (in 2012–14). In 2012 and 2013, knowledge and skills in social studies were assessed in grades 9 and 10. In the 2014–15 school year, students’ knowledge and skills were assessed in the following subjects and grades: grade 6: mathematics, world history; grade 7: physics, biology; grades 5 and 8: reading. From 2003 to 2006, Republican Monitoring was regulated on an annual basis by the Minister of Education of the Republic of Belarus, and since 2007, it has been regulated by the Program on the Development of General Secondary Education for 2007–16. 2. The policy-mandated purposes of this assessment include informing policy and pedagogy, as well as monitoring education quality and evaluating interventions aimed at improving student learning. This assessment is also used to evaluate the extent to which student learning is aligned with the national education standards and syllabi requirements. 3. The Belarus secondary education development program for 2007–16 (Council of Ministers of the Republic of Belarus, 2007) is the formal/official publicly available policy document that authorizes the Republican Monitoring. 4. The decree “On Organization and Carrying Out Monitoring in Secondary Educational Institutions” (Ministry of Education of the Republic of Belarus) provided guidelines on the Republican Monitoring, including on its governance, frequency of administration, who should be assessed, what should be assessed, how the results should be used, and how stakeholders should be informed of assessment results (including through publication of assessment materials in scientific- methodological subject journals, in the magazine Vesnik Adukatsii, and on the national education portal). Guidelines on the confidentiality of results are not provided by official documents. 5. The Department of Education Quality Monitoring was established at the National Institute of Education as a permanent unit with the responsibility for scientific-methodological and organizational support of the Republican Monitoring of Education Quality. Five assessment programs fall under the Republican Monitoring, including the assessment of: a. Students’ learning achievement in school subjects (the assessment program that is the subject of this report) b. Student fatigue and efficiency (study of student functional status as well as factors such as the motivation to learn, the level of general learning abilities, academic load, compliance with sanitary requirements as to teaching arrangements, etc.) c. Student personal development and “educatedness” (study of student perceptions of moral and legal norms in social life; student value system, behavior attitudes, level of student professional identity, student social adjustment degree, their awareness of help lines, etc.) d. Quality of education services (study of organization efficiency of optional classes, industry-oriented teaching, extra school day, family-school interaction, etc.) 52 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 e. Teachers’ professional competency (study of teachers’ proficiency, their educational orientation) is conducted by the Department of Education Sociology and Continuing Education of the Academy of Postdiploma Education. The Department of Education Quality Monitoring of the National Institute of Education has had the primary responsibility of overseeing key national large-scale assessment scientific-methodological and organizational activities as related to the Republican Monitoring since 2003. The unit is responsible for assessment design, administration, data processing, and data reporting, including the preparation of data analysis and reporting of assessment results as well as for providing recommendations on the basis of the assessment results. The Ministry of Education oversees the project management activities of the assessment and ensures the quality of the assessment materials. In accordance with the Order of the Ministry of Education of the Republic of Belarus, committees that are responsible for education management in an oblast (1) select the coordinator for conducting the Republican Monitoring in institutions of general secondary education; (2) organize the participation of external observers (specialists from the rayon/city educational and methodological departments, specialists from the institutes of educational development) and the carrying out of the Republican Monitoring in regional education institutions; (3) organize technical support for the implementation of the Republican Monitoring in education institutions in the region (copying of materials, scanning of materials, carrying out a survey on the computer); and (4) provide verification and assessment of student responses in educational institutions and in the institutes of education quality. School coordinators, who are appointed by the director of an educational institution, are responsible for the implementation of assessment activities in the educational institutions. Temporary subject commissions, comprising subject teachers with the highest qualifications and teacher-methodologists (at the rayon level and in the institutes of educational development), are formed for the purposes of verification and evaluation of students’ work. Various departments and regional authorities, including those overseeing education at the oblast level and city (for example, the committee on education of the city of Minsk) and departments of education, sport, and tourism, conduct an external review of carrying out the assessment in the education institutions. 6. The Department of Education Quality Monitoring is part of the National Institute of Education, which is accountable to the Ministry of Education. The Ministry of Education is accountable to the Council of Ministers of the Republic of Belarus, and the Department of Education with oblast (rayon) executive committees is accountable to the oblast (rayon) executive committee. No unit with assessment responsibility is accountable to an autonomous board or committee that is institutionally separate from units in charge of the NLSA. 7. The budget for Republican Monitoring was provided by national and local government funding sources and covered assessment design, administration, data processing, data reporting, and project management. Research and development activities on the topic of assessment of the quality of education are also financed through other government sources. 53 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 8. Funding was sufficient to cover all core assessment activities. 9. The NLSA unit has most of the appropriate resources. Appropriate storage facilities and communication tools were the most readily available resources, while computers for all technical staff, software, and servers as well as building security were generally widely available. 10. Individuals responsible for completing key assessment activities included the permanent staff from the Department of Education Quality Monitoring of the National Institute of Education, permanent staff from the Research and Development Centre of the National Institute of Education (who carry out research on assessment issues, perform external review of diagnostic materials for monitoring, and develop educational standards and syllabi), experts from local educational authorities or local institutions of education development are involved as external observers, and teachers (teacher-methodologists of the highest qualification who work on subject commissions). All of the individuals responsible for completing key assessment activities had relevant qualifications for their roles. 11. Individuals responsible for completing key assessment activities included permanent staff from the Department of Education Quality Monitoring of the National Institute of Education, permanent staff from the Research and Development Centre of the National Institute of Education, and education specialists hired to support the assessment cycle. Generally no issues were found with the effectiveness of these individuals. Subject-matter teachers of the highest qualification are also invited to work on subject commissions. Experts from the Department of Education Quality Monitoring developed scoring guides to facilitate teachers’ review and evaluation of students’ tests. The subject committees include only top-ranked teachers and supervising teachers. Some issues were found with the effectiveness of these teachers (for example, some teachers overlooked errors, which was noted by the Department of Education Quality Monitoring of the National Institute of Education) specialists; however, the quality of the assessment overall was not compromised as a result. 12. There were sufficient opportunities to learn about the NLSA. Opportunities included: a. University graduate programs (master’s or doctorate level) on student assessment that include topics relevant to the NLSA (e.g., test design, reporting). “Monitoring of the quality of knowledge on the subject” courses are organized in higher pedagogical educational institutions. b. University courses/workshops on the content and skills measured by the NLSA (e.g., courses on curriculum). In higher pedagogical educational institutions, as part of pedagogy and methods of teaching courses, topics include subject syllabus and learning outcomes to be measured in the course of education (as well as monitoring). For example, “Monitoring of the Quality of Knowledge on the Subject” courses are made available. c. Nonuniversity courses/workshops on the content and skills measured by the Republican Monitoring (e.g., courses on curriculum). In continuing education institutions (Academy of Postdiploma Education, regional education development institutions) advanced training course syllabi cover topics such as learning outcomes and measurement approaches. For example, the National Institute of Education carries out live and distance-learning courses and methodological workshops for subject teachers on the topic of “Teacher and Student Control and Assessment.” 54 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 d. Nonuniversity courses/workshops on Republican Monitoring topics other than the content and skills measured by the assessment (e.g., test design, reporting). The Republican Institute of High School (RIHS) runs advanced training courses on the “Basic Theory and Methods of pedagogical measurements.” e. Funding for attending international programs, courses, and workshops on student assessment that cover topics relevant to the Republican Monitoring Education. Quality assessment system experts participate in education quality assessment training courses (including carrying out monitoring research) in the Russian Training Centre of the Institute of Education of NRU “Higher School of Economics” and in the Eurasian Association of Educational Assessment (EAOKO) and participate in international expert consultations on student academic achievement assessment. f. Opportunities to participate in webinars, distant learning courses, and online student academic achievement conferences. Specifically, education experts participate in online student academic achievement conferences, in webinars of the Russian Training Centre of the Institute of Education of NRO “Higher School of Economics,” and benefit from the study materials published on the Eurasian Association of Educational Assessment (EAOKO) website. Opportunity (a) is available only to those studying in higher education institutions. Opportunities (b), (c), and (e) are available only to existing full-time staff at the NLSA unit, university professors, and primary and secondary school teachers and educators. Opportunities (d) and (f) are available only to existing full-time staff of the NLSA unit, other specialists of the National Institute of Education, and specialists of RIKC. University courses/workshops on NLSA topics other than the content and skills measured by the Republican Monitoring (e.g., test design, reporting), internships/short-term employment in the unit running the Republican Monitoring, or presentations about the Republican Monitoring (e.g., test design, administration) are not made available. 13. The Republican Monitoring is intended to measure official learning standards and the official curriculum, with which it is closely aligned. Officially mandated reviews ensure that the Republican Monitoring measures what it is intended to measure as part of the Republican Monitoring, including regular internal and external reviews that take place during all or almost all Republican Monitoring rounds. Specifically, diagnostic tools for the Republican Monitoring are subject to: a. Internal review in the Department of Education Quality Monitoring of the National Institute of Education b. Informal external review in laboratories of the Research and Development Centre of the National Institute of Education c. Formal external review in the Ministry of Education of the Republic of Belarus (in accordance with the decrees of the Minister of Education, all diagnostic and instruction materials for carrying out the Republican Monitoring are to be coordinated in the Departments of the Ministry of Education). 14. Although the format of the questions across assessment types may different, classroom assessment and examinations are closely aligned with the Republican Monitoring. 55 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 15. Public school educators provide students with the opportunity to be exposed to the content and skills measured by the Republican Monitoring. School educators provide all or almost all students with the opportunity to be exposed to the content and skills measured by the Republican Monitoring through regular course instruction at school. Students can also access samples of previously administered Republican Monitoring assessments. Students in private institutions of general secondary education comprise 0.07 percent of students in institutions of general secondary education in the Republic of Belarus and are therefore not included in the Republican Monitoring sample. Thus, although public schools are covered in this review, private schools are not. 16. During meetings with coordinators of the assessment in institutions of general secondary education in the regions, which take place on an annual basis at the National Institute of Education, information on the main areas of the assessment, the objectives and content of the assessment in the given year, and the types of questions that will be on the assessment are discussed. Similar meetings are held immediately before the implementation of the Republican Monitoring in every region for coordinators in those institutions where the Republican Monitoring will take place (those institutions that are include in the sample). 17. Sufficient formal quality assurance procedures were in place to ensure the quality of the Republican Monitoring. Procedures that were required to take place every assessment round include: a. All proctors or administrators were trained according to a protocol b. There was a standardized manual for NLSA administrators c. All answer sheets were numbered d. There were external observers (e.g., specialists from the rayon or city teaching offices of institutes of educational development) e. There were internal observers (e.g., NLSA staff observing at administration sites) f. Double data scoring (conducted by rayon subject commissions and subject commissions formed on the basis of the oblast institute of educational development; in cases of discrepancies in scoring, National Institute of Education specialists conduct a review as well). Procedures that were not required to take place every assessment round include: a. Piloting of questions/items/tasks before the official NLSA administration b. Training of scorers to ensure high interrater reliability c. Double processing of data. 18. The Republican Monitoring is fully standardized at the system level. Assessment design, administration, scoring, and reporting are the same (or equivalent) for all students in the same assessment round. Specifically: a. Republican Monitoring papers and questions/items/tasks were the same or equivalent for all students 56 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 b. Republican Monitoring administrators were trained to ensure that all students took the NLSA under the same conditions c. Quality control monitors/observers were used to ensure the same administration conditions in all locations where the NLSA was administered d. The same scoring criteria were used to correct the Republican Monitoring questions/items/tasks and e. Republican Monitoring results were computed using the same procedures for all students. Republican Monitoring results are not reported to all students in the same way. 19. A representative random sample of public school students in the target grades is selected to participate in the Republican Monitoring. Students in private institutions of general secondary education comprise 0.07 percent of students in institutions of general secondary education in the Republic of Belarus and are therefore not included in the Republican Monitoring sample. 20. The Republican Monitoring is administered to students studying in secondary schools under the secondary school curriculum. All eligible students can take part in the Republican Monitoring. Starting in 2014, students from all rural schools have an equal opportunity to be included into the Republican Monitoring sample. This required increasing the number of external observers by employing specialists from the rayon and city teaching offices of the institutes of education development. 21. Errors in item/question/task development and errors in scoring student responses did occur; however, these did not affect the Republican Monitoring. To avoid incorrect instructional wording in diagnostic materials used on the Republican Monitoring, a decision was made to carry out pilot research to test the diagnostic materials before the administration of the assessment. Additionally, to avoid scoring errors, a decision was made to brief the subject committee members in charge of checking students’ tests to provide additional guidance on the scoring procedures. The following issues did not occur: errors in test design; delays in test design; delays in item/question/task development; errors in administering the NLSA; delays in administering the NLSA; poor training of NLSA administrators; delays in scoring student responses; errors in data processing; delays in data processing; errors in reporting results; delays in reporting results; results not being reported. 22. The only inappropriate behavior that took place was students copying from other students. This inappropriate behavior did not affect the credibility of the Republican Monitoring, and a decision has been made to increase the number of different versions of the assessment from two to four to address this issue. The following inappropriate behaviors did not occur: leakage of the content of the NLSA paper or part of a paper before the NLSA administration; students being trained to answer specific questions on the NLSA; students being excluded from taking the NLSA; providing student assistance via teachers, supervisors, mobile phone, or equivalent during the administration of the NLSA; changing students’ responses after they have submitted their NLSA papers; changing students’ results (e.g., score points) after their papers have been scored. 57 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 23. Information on the evaluation and assessment of all questions is provided in the “Recommendations for Evaluation and Assessment of Students’ Work” document that is made available on the website of the management of the monitoring of the quality of education at the National Institute of Education on the day that the Republic Monitoring takes place. This information is made available to assessment coordinators in those educational institutions that took part in the monitoring. The following methods and procedures are not documented: construction of questions/items/tasks; pilot testing of questions/items/tasks; analysis of piloted questions/items/tasks; test assembly; reliability; scaling; setting cutoff scores. 24. An official report on the results of the Republican Monitoring is submitted to the Ministry of Education of the Republic of Belarus (in accordance with the decree of the Minister of Education). Based on the report, recommendations on enhancing the quality of secondary school education are prepared and published on the national educational portal and in research and methodological journals. The report provides information about the number of students who acquired the content of the taught (and assessed) subject at a high, sufficient, average, acceptable, and low level. Official results of the Republican Monitoring are made available at the Ministry of Education and in education-specific scientific-methodological journals, as well as on the national education portal, which publishes a summarized version with recommendations on how to improve the management of education quality. Results are communicated at the collegium of the Ministry of Education, at meetings of specialists of the education system, and as part of seminars for increasing teacher qualifications. National results at the question/item level are also available through the newspaper Nastaunitskaya Gazeta. Teachers of general secondary education, parents, and the local authorities have access to this information. 25. Most stakeholder groups perceive the Republican Monitoring results as credible. 26. The impact of the Republican Monitoring on the education system is studied by experts of the Department of Education Quality Monitoring of the National Education Institute through: a. Focus-group studies involving secondary school teachers at the premises of local education development institutes (at advanced education courses, methodological workshops, and meetings) (which occurs during most NLSA rounds) as well as individual and group discussions with secondary school teachers at the later monitoring rounds (which occurs during most NLSA rounds) b. Benchmarking of student academic achievement level for school subjects of various target groups (based on the results of various monitoring rounds) (which occurs during some NLSA rounds) c. Content analysis of republished syllabi and textbooks (which occurs during some NLSA rounds). The following mechanisms are not in place to monitor the impact of the NLSA: oversight committee; expert review groups; funding for independent research on the NLSA; studies (e.g., effect on school practices) on the NLSA. Over the past six assessment rounds, Republican Monitoring results have informed: a. Syllabi adjustment for school subject upon republishing in 2009 and 2012 58 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 b. Textbook editing by separate school subjects upon republishing, development of various teaching materials for separate subjects (e.g., biology) c. Decision making on introduction of prespecialized training at the stage II of secondary education (in the 2015–16 academic year) d. Adjustment of teachers’ advanced training syllabi e. Annual development of work plans of subject teachers’ methodological association. 27. This indicator does not apply to this rubric. 59 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 BELARUS International Large-Scale Assessment (ILSA) 60 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 LATENT EMERGING ESTABLISHED ADVANCED Indicator Absence of, or deviation from, On way to meeting minimum Acceptable minimum standard Best practice the attribute standard Stability of The system did not participate in an The system participated in an ILSA The system completed one ILSA The system completed two or more Participation ILSA round in the last 10 years.1 round in the last 10 years, but did round in the last 10 years. ILSA rounds in the last 10 years. not complete it. Policy No policy document authorized the An informal/draft policy document A formal/official policy document A formal/official policy document Document ILSA program. authorized the ILSA program. authorized the ILSA program, but authorized the ILSA program and the document was not available to was available to the general public. the general public. Stability of There was no unit with primary There was a unit(s) with primary There was a permanent unit(s) with This option does not apply to this Organization responsibility for running the ILSA responsibility for running the ILSA primary responsibility for running indicator. program. program, but the unit(s) was the ILSA program that had been in temporary or had been in place for place for 5 or more years. less than 5 years. Accountability There was no unit with primary The unit(s) with primary The unit(s) with primary The unit(s) with primary of Organization responsibility for running the ILSA responsibility for running the ILSA responsibility for running the ILSA responsibility for running the ILSA program, or else the unit program was accountable to a program was accountable to a program was accountable to a responsible was not accountable to clearly recognized body within the clearly recognized body within the clearly recognized external body. a clearly recognized body. ILSA unit. same institution as the ILSA unit. Source of There was no funding available for The source of funding for the The source of funding for the This option does not apply to this Funding ILSA activities. majority of ILSA activities was majority of ILSA activities was the indicator. loans, credits, grants, or government’s internal funding equivalent. sources. (CONTINUED) 61 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 LATENT EMERGING ESTABLISHED ADVANCED Indicator Absence of, or deviation from, On way to meeting minimum Acceptable minimum standard Best practice the attribute standard Activities There was no funding available for Funding was not sufficient to cover Funding was sufficient to cover all This option does not apply to this Funded ILSA activities. all core ILSA activities. core ILSA activities. indicator. Organization The ILSA unit did not have the The ILSA unit had some of the The ILSA unit had most of the The ILSA unit had all of the Resources appropriate resources. appropriate resources. appropriate resources. appropriate resources. Qualifications There were no individuals Some of the individuals Most of the individuals responsible All or almost all of the individuals of Staff responsible for completing key ILSA responsible for completing key for completing key ILSA activities responsible for completing key ILSA activities. ILSA activities had the relevant had the relevant qualifications. activities had the relevant qualifications. qualifications. Effectiveness of There were no individuals The responsible individuals The responsible individuals The responsible individuals Staff responsible for completing key ILSA completed key ILSA activities, but completed key ILSA activities, with completed key ILSA activities and activities. there were significant issues in only some issues in how these there were no issues in how these how these activities were activities were completed. activities were completed. completed. Staff/Teacher There were no opportunities to Opportunities to learn about the There were sufficient high-quality Opportunities to learn about the Opportunity to learn about the ILSA. ILSA were minimal, or not of high opportunities to learn about the ILSA were extensive, of high Learn quality, or did not benefit all key ILSA that were available to key quality, and benefited key stakeholder groups. stakeholder groups. stakeholder groups. (CONTINUED) 62 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 LATENT EMERGING ESTABLISHED ADVANCED Indicator Absence of, or deviation from, On way to meeting minimum Acceptable minimum standard Best practice the attribute standard Alignment with The ILSA was poorly aligned with The ILSA was somewhat aligned The ILSA was very aligned with This option does not apply to this Other other types of assessment activities with other types of assessment other types of assessment activities indicator. Assessments in the system. activities in the system. in the system. Opportunities Students did not have Students had limited opportunities Students had sufficient Students had many opportunities for Students to opportunities to be exposed to the to be exposed to the content and opportunities to be exposed to the to be exposed to the content and Be Exposed to content and skills measured by the skills measured by the ILSA. content and skills measured by the skills measured by the ILSA. Content and ILSA. ILSA. Skills Quality Many errors or delays in activities Errors or delays in activities Any errors or delays in activities Errors or delays in activities did not Processes took place that affected the ILSA to affected the ILSA to a significant had only a minimal effect on the affect the ILSA. a great extent. level. ILSA. Inappropriate Inappropriate behavior Inappropriate behavior took place Inappropriate behavior was low Inappropriate behavior, if any, was Behavior compromised the credibility of the and compromised the credibility of and did not compromise the marginal and did not compromise ILSA to a great extent. the ILSA somewhat. credibility of the ILSA. the credibility of the ILSA. Meeting ILSA results for the system did not ILSA results for the system met ILSA results for the system met all This option does not apply to this Standards for meet the standards required for sufficient standards to be of the standards required to be indicator. Publication publication in the international presented beneath the main presented in the main displays of report. displays in the international the international report. report. (CONTINUED) 63 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 LATENT EMERGING ESTABLISHED ADVANCED Indicator Absence of, or deviation from, On way to meeting minimum Acceptable minimum standard Best practice the attribute standard Publication of ILSA results were not published in Limited information on the ILSA Sufficient information on the ILSA Comprehensive information on the Results the system. results was published in the results was published in the system ILSA results was published in the system, or the results were using an array of mechanisms. system using an array of published using a minimum mechanisms. number of mechanisms. Credibility of The results of the ILSA were The results of the ILSA were The results of the ILSA were The results of the ILSA were Results perceived as credible by very few perceived as credible by some perceived as credible by most perceived as credible by all or stakeholder groups. stakeholder groups. stakeholder groups. almost all stakeholder groups. Use of Results ILSA results were not used by ILSA results were used in minimal ILSA results were used in sufficient ILSA results were used in extensive stakeholders in the system. ways by stakeholders in the ways by stakeholders in the ways by stakeholders in the system. system. system. Readiness to The system was weakly prepared to The system was somewhat The system was well prepared to This option does not apply to this Participate in an participate in an ILSA program in prepared to participate in an ILSA participate in an ILSA program in indicator. ILSA the future. program in the future. the future.2 64 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 International Large Scale Assessment (ILSA): Development-level rating justifications 1. Belarus has not participated in an international large-scale assessment exercise. 2. Belarus is planning to take part in the 2018 Program for International Student Assessment (PISA 2018). Funding support for PISA 2018–related activities in Belarus will be provided through a World Bank project loan. The Republican Institute for Knowledge Control (RIKC) will be responsible for PISA 2018 activities in the country. RIKC has prior experience carrying out assessments because its main function is to carry out the Centralized Testing in Belarus. Specialists who work at the National Institute of Education, the Academy of Post-diploma Education, institutes of educational development, and local educational authorities are also prepared to work on PISA 2018, mainly because of their previous experience working on the Republican Secondary Education Quality monitoring program. These experts have degrees in pedagogy, social studies, and psychology, and many of them have completed training courses relevant to PISA, including a “Basic Theory and Methods of Pedagogical Measurements” course. National education quality assessment system experts regularly participate in webinars conducted by the Higher School of Economics Russian Training Centre at the Institute of Education in Moscow and receive information on education quality assessment from Eurasian Association for Educational Assessment (EAOKO) news bulletins. Experts from the National Institute of Education also have experience conducting research in the area of educational assessment. In 2013 the institute, in association with the Russian National Training Foundation, conducted research to measure the information and communication competencies of grade 9 students in Belarus. The “Information and Communication Competence Assessment of Grade 9 Students in the Republic of Belarus” project was implemented within the framework of a small grants program implemented by the Russian Centre for International Cooperation in Education Development (CICED). 65 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 Acknowledgments This report was prepared by the World Bank SABER– Student Assessment Team (Marguerite Clarke, World Bank Senior Education Specialist; Julia Liberman, World Bank Operations Officer; and Rachel Cooper, World Bank Education Consultant) in collaboration with Igor Kheyfets, World Bank Senior Economist and Task Team Leader for education projects in Belarus. The Team is grateful for the feedback and support from Irina Oleinik, World Bank Operations Officer; Hanna Shvanok, World Bank Team Assistant; and Anna Sakalouskaya, World Bank Program Assistant. This report was made possible with the input and contributions of Mr. Valeriy Alexandrovich Struzhko, Director of the Minsk Secondary School no. 161, who oversaw the data collection for the SABER–Student Assessment exercise in Belarus. The Team would like to thank Ms. Valentina Vasilevna Ginchuk, Head of the Department of Education Quality Monitoring at the National Institute of Education, and Mr. Viachaslav Zairovich Suleymanov for their invaluable inputs, feedback, and collaboration. References Clarke, M. 2012. “What Matters Most for Student Assessment Systems: A Framework Paper.” READ/SABER Working Paper Series. World Bank, Washington, DC. UNESCO Institute for Statistics. Belarus Education Statistics. http://www.uis.unesco.org/Pages/default.aspx/. World Bank. 2015. Belarus Country Indicator Data. World Bank, Washington, DC: http://databank.worldbank.org/data. ———. 2015. “Belarus Country Overview.” http://www.worldbank.org/en/country/belarus/overvie w. ———. 2015. Belarus—Education Modernization Project. Project Appraisal Document. World Bank, Washington, DC. 66 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS BELARUS ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2015 www.worldbank.org/education/saber r The Systems Approach for Better Education Results (SABER) initiative produces comparative data and knowledge on education policies and institutions, with the aim of helping countries systematically strengthen their education systems. SABER evaluates the quality of education policies against evidence-based global standards, using new diagnostic tools and detailed policy data. The SABER country reports give all parties with a stake in educational results—from administrators, teachers, and parents to policy makers and business people—an accessible, objective snapshot showing how well the policies of their country’s education system are oriented toward ensuring that all children and youth learn. This report focuses specifically on policies in the area of student assessment. This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. 67 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS