Ukraine SABER Country Report EDUCATION MANAGEMENT INFORMATION SYSTEMS 2017 Key Policy Areas Status 1. Enabling Environment In 2014, the Ministry of Education and Science (MoES) established the Institute of Educational Analytics (IEA) and mandated the IEA to collect, analyze and disseminate education data. The combined legal framework of ministerial orders, decrees and IEA Statute (charter) ensures that data from all primary and secondary schools is captured and validated in the State Information System of Education (DISO) managed by the IEA. It also dictates procedures and timelines for the data collection of the annual school census forms ZNZ-1 (student data) and 83-RVK (teacher data). The current Statute also requires the IEA to define information security procedures in EMIS in relation to confidential education data. EMIS expenses are specified in the IEA budget, including maintenance, technical support and staffing expenses, but there are challenges in budget coverage and predictability, especially in relation to software or hardware acquisition. With regards to staffing, EMIS staff positions often lack (financial) competitiveness in comparison to the private sector and the IEA faces the challenge of hiring and retaining qualified personnel. 2. System Soundness Since 2016, the IEA has managed the collection of school-level data through DISO, an existing functioning EMIS. But the lack of a long term wireframe for the system architecture will negatively affect future plans for expansion, data production and system integration. Currently, data collection focuses primarily on administrative data, but does not include financial, human resource or learning outcome data. DISO is also not integrated with other education sector databases nor are there provisions for facilitating inter-ministry data sharing. In the Ukrainian EMIS two sets of data flows exist: digital and paper-based; both versions are entered into the DISO database. Data validation is supposed to occur at the local, regional and central level. The current system is limited to basic statistical analysis as advanced statistical tools have not been incorporated. The government has made plans to expand and modernize the DISO, which will likely introduce new data collection, statistical and report generation modules. In addition, data sharing and integration mechanisms should be included to link DISO with other databases collecting information relevant to education. However, compatibility challenges may occur with regards to the original EMIS software, which was first developed in 2009. 3. Quality Data The school census data is generally reliable, but data validation and timeliness remain challenging. Ideally, both the digital and paper versions are compared for accuracy at the local, regional and central level. However, data is often incomplete, missing or erroneous and there are currently insufficient mechanisms in place to correct the data. This is particularly true for EMIS data outside of the school census data collection process (forms ZNZ-1 and 83-RVK). Moreover, the IEA does not yet have sufficient capacity and authority to ensure the timeliness of reporting and thus, some delays in data collection occur. Nevertheless, a school census is conducted annually and most data is available within 6 months of data collection. Data terminology and concepts are clearly defined and follow international classification standards. The description of data fields and metadata are included in the operations manual. 4. Utilization for Decision Making Operational use of education statistics is limited. Currently, education data is generally used for resource allocation decisions. Sometimes EMIS data acts as a supporting tool in strategic decisions at the regional level such as in the decision making process on the reorganization of the school network. Donor agencies and non-governmental organizations (NGOs) and independent researchers have limited access to EMIS data for their research purposes. Parents and schools generally do not use EMIS data as it is presented to them in aggregate form without necessary explanations or training. Open data dissemination practices and effective feedback mechanisms are lacking. Many countries have realized the substantial benefits of effective data dissemination and utilization in improving student learning and achieve education goals. [Type here] UKRAINE ǀ EMIS SABER COUNTRY REPORT |2017 Table of Contents Introduction .................................................................................................................................................. 3  Approach of SABER-EMIS ...................................................................................................................... 3  Assessing Policy Intent & Implementation ........................................................................................... 4  Methodology......................................................................................................................................... 5 Country Overview ......................................................................................................................................... 6 Ukraine EMIS Results .................................................................................................................................... 8  Policy Area 1: Enabling Environment .................................................................................................... 8  Policy Area 2: System Soundness ........................................................................................................ 11  Policy Area 3: Quality Data ................................................................................................................. 15  Policy Area 4: Utilization for Decision Making .................................................................................... 17 Recommendations and Proposed Activities ............................................................................................... 20  Recommendations 1: Enabling Environment...................................................................................... 21  Recommendations 2: System Soundness ........................................................................................... 22  Recommendations 3: Quality Data ..................................................................................................... 25  Recommendations 4: Data Utilization ................................................................................................ 26 Acknowledgments....................................................................................................................................... 31 Acronyms .................................................................................................................................................... 31 References .................................................................................................................................................. 32 Appendix A: Summary of Policy Lever Benchmarking ................................................................................ 34 Appendix B: Extended Rubric, Ukraine Scores Highlighted in Red ............................................................. 35 Appendix C: Structure of the Ukrainian Education System ........................................................................ 44 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 2 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2017 Introduction The Systems Approach for Better Education Results (SABER) is a flagship program designed to support countries in systematically examining and strengthening the performance of their education systems. Part of the World Bank’s Education Sector Strategy, SABER uses diagnostic applications for examining education systems and their component policy domains against global standards and best practices in comparison with the policies and practices of countries around the world. By leveraging global knowledge, the SABER applications fill a gap in the availability of data and evidence on what matters most to improve the quality of education and achievement of better results. This report discusses the results of applying the SABER–Education Management Information Systems (EMIS) framework to analyzing the main information system currently used for data collection in Ukrainian pre-university education, the State Information System of Education (“DISO”). The objectives of this report are to examine the system according to key policy areas, identify successes and challenges in the system, and provide recommendations to support the continued advancement of EMIS in Ukraine. The information used in this analysis was collected by a local consultant during late 2016 and analyzed by World Bank staff in early 2017.  Approach of SABER-EMIS Information is a key ingredient in an effective education system. SABER–EMIS aims to help countries improve data collection, data and system management, and data use in decision making. SABER-EMIS assesses the effectiveness of a country’s EMIS with the aim of informing policy dialogue and helping countries better manage education inputs and processes to achieve better system efficiency and improved learning outcomes. A successful EMIS is credible and operational Figure 1: SABER-EMIS Policy Areas and Levers in planning and policy relevance, as well as Policy Areas Policy Levers: legal framework, organizational structure teaching and learning. It produces and Enabling and institutionalized processes, human resources, monitors education statistics within an Environment infrastructural capacity, budget, data-driven culture education system and has a multifaceted System Policy Levers: data architecture, data coverage, data structure, comprising the technological and Soundness analytics, dynamic system, serviceability institutional arrangements for collecting, Policy Levers: methodological soundness, accuracy and processing, and disseminating data (Abdul- Quality Data reliability, integrity, periodicity and timeliness Hamid 2014). It is crucial for tracking changes, Utilization for Policy Levers: openness to EMIS users, operational use, ensuring data quality and timely reporting of Decision Making accessibility, effectiveness in disseminating findings information, and facilitating the utilization of information in decision making. Source: Abdul-Hamid 2014 The SABER-EMIS assessment methodology is built on four key policy areas that are essential to EMIS and must be assessed to understand and ultimately strengthen the system. Each policy area is defined by a set of policy levers (actions that help governments reach the policy goal) and indicators (measuring the extent to which the policy levers are achieved) (figure 1). A strong enabling environment lays the foundation for an effective EMIS. Enabling environment refers to the laws, policies, structure, resources, and culture surrounding an EMIS that make data collection, management, and access possible. In essence, this policy area is the context in which an EMIS exists. This defined scope of an enabling environment builds on lessons learned from studies of education management systems. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 3 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2017 Figure 2: SABER Scoring and EMIS Development System soundness ensures key processes, 4 structures and integration capabilities in an Advanced effective EMIS. Education data are sourced 3 Comprehensive from different institutions, but all data feed Established enabling 2 Enabling environment, into and make up the EMIS. Databases within Emerging environment, processes, an EMIS are not viewed as separate databases, 1 processes, structure, data Basic enabling Latent management, but as part of the whole EMIS. Key aspects of environment, structure, data utilization, and processes, management, system soundness include what data are Limited enabling structure, data utilization in integration in environment, place, with covered in the EMIS and how they come processes, management, place with some intelligent utilization integration together in the overarching system. structure, data analytics management, utilization Quality data establishes the mechanisms Source: Abdul-Hamid 2014 required to collect, save, produce, and utilize information in an accurate, secure, and timely manner. Data quality is a multidimensional concept that encompasses more than just the underlying accuracy of the statistics produced. It means that not only are the data accurate, but that the data address specific needs in a timely fashion. Quality data lays the groundwork for utilization. An effective EMIS is utilized in decision making by all stakeholders (parents, students, teachers, principals and policy makers) across the education system. An EMIS needs to be used so that measures can be taken to improve educational quality. Accurate information on education sector performance enables the design of more informed policies and programs. It is imperative to understand where decision making occurs, if the capacity to analyze and interpret education data exists, and if specific data are available to inform decisions. Figure 3: Policy Intent, Implementation and Outcomes Cycle, Using the EMIS data collection instrument, with Examples policy levers are scored on a four-level scale  Policies  Utilization (latent, emerging, established, and advanced)  Vision & buy-in  Processes to assess the extent to which both policy intent  Standards  Institutionalization and implementation are achieved (figure 2).  Strategy  Budget documents  Human Resources  Assessing Policy Intent &  Communication Implementation Intent Implementation The EMIS assessment examines policy intent and the degree to which intended policies are effectively implemented on the ground (figure 3). Intent refers to the way in which EMIS and its overarching purpose are articulated by decision makers and documented in policies Outcomes and legislation, as well as standards and strategy documents. Assessing intent alone  Teaching & Learning only reveals part of the picture.  Management  Planning & Monitoring  Transparency As such, this EMIS assessment also evaluates  Governance policy execution. Implementation refers to the degree to which policy intentions are Source: Authors SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 4 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2017 translated into the day-to-day activities of stakeholders (e.g., policy makers, local administrators, principals, teachers, students, etc.). Implementation can be observed through the utilization of EMIS by stakeholders, budget allocation, distribution of human resources, availability of professional development activities, communication and dissemination of information, as well as the extent of institutionalization across the system. Once policy intent and implementation are analyzed, the EMIS assessment explores the results of these two key components, with a focus on system effectiveness and efficiency, in addition to teaching and learning, and management and planning. Strong education systems will ultimately use these outcomes to inform the effectiveness of policies and education strategies and make adjustments as necessary, creating the cyclical process illustrated in figure 3. In Ukraine, EMIS intent and implementation were assessed through desk research, analysis of system structures and utilization, as well as interviews with a variety of stakeholders (table 1). Table 1: Measuring Policy Intent and Implementation in Ukraine Policy Intent Processes Policy Implementation  Meetings with  Analysis of data  Interviews at MoES quality and national,  Review of comprehensiveness district and relevant policies,  Examination of school levels national professional including strategies, development decision standards, and activities makers and planning education documents stakeholders Source: Authors  Methodology The EMIS assessment methodology consists of a review of written policies and technical documents, as well as interviews with key stakeholders across the education system to understand policy implementation. Primary data collection for the Ukraine SABER-EMIS assessment took place in November 2016. The authors conducted a comprehensive review of policies, as well as technical documents and other background materials. To further examine policy intent and implementation, a series of interviews and/or meetings took place with the following entities: 1. Ministry of Education and Science 2. Institute of Educational Analytics 3. Parent organizations 4. Local education departments SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 5 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2017 Country Overview Ukraine is a lower middle income country with a population of approximately 45 million and per capita GDP of US$2,174. Traditionally a centralized state, Ukraine initiated a decentralization reform in 2014 that began devolving additional authority to the regions (including in the sphere of education). The Ukrainian economy has suffered in recent years as a result of political instability and conflict, with GDP shrinking by 6.6 percent in 2014 and 9.8 percent in 2015, followed by a return to modest growth of 2.3 percent in 2016. The percentage of the population living below the poverty line increased significantly in as a result of the crisis before moderating slightly in 2016 (World Bank 2016a). The pre-university education system in Ukraine consists of pre-primary education and 11 years of compulsory schooling at the primary and secondary levels. Primary education lasts for 4 years and basic (lower) general secondary education continues for 5 years. After basic general education, students can enter vocational training of up to 4 years or continue with complete (upper) general secondary education for 2 years (Appendix C). Table 2 illustrates the decline in student enrollments in recent years due to demographic changes. Currently, detailed data publicly available about Ukraine’s education system is limited and does not include important indicators such as dropout rates. Table 2: Education Indicators at a Glance 2012 2013 2014 2015 2016 Gross Enrollment Pre-primary 1,354,394 1,428,390 1,470,817 1,294,891 1,291,207 (number of Primary 1,584,382 1,638,497 1,685,030 1,536,578 1,599,250 students)* Secondary 2,900,544 2,803,725 2,711,465 2,370,298 2,325,864 Tertiary 2,347,380 2,205,595 2,085,991 1,719,257 1,635,578 Net Enrollment Rate Primary 98.6 98.6 96.2 - - (percent)** Secondary 85.1 86.6 88.6 - - Government Percent of GDP 7.0 7.3 6.4 5.8 6.6 Expenditure on Percent of Total 20.6 21.2 19.1 16.8 15.6 Education* Government Expenditure Source: *State Statistical Service 2017; **EdStats 2016. The education sector is in need of structural reform and modernization. Over the past two decades, the number of school-aged children has declined, while little reduction in education institutions has taken place. Despite high enrollment rates in secondary and tertiary education, there is often a considerable gap between the skills of recent graduates and the needs of the labor market. To increase accountability and autonomy at the regional and local level, ambitious reforms have been introduced since 2014. One of the objectives has been to introduce evidence-based policy making and bring the Ukrainian education system in line with international standards (World Bank 2016b). A wide-reaching reform program for pre- university education envisions the creation of the New Ukrainian School, extending compulsory education to 12 years and adopting a curriculum rooted in the acquisition of key life competencies (MoES 2017). SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 6 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2017 Little is known about the quality of education in Figure 4: TIMSS Scores Grade 8, Ukraine Ukraine compared to other countries. A system of 510 national assessments—“external independent evaluations” administered by the Ukrainian Center for 490 Educational Quality Assessment (UCEQA)—is used to 470 assess students’ knowledge for admission to higher 450 education. However, Ukraine has had limited participation in international student assessments to 430 2007 2011 date, although two rounds of participation in TIMSS Math Science (2007 and 2011) showed moderate improvements in Source: EdStats 2016. Grade 8 math and science results (figure 4). Recently the government committed to Ukraine’s participation in PISA 2018, which will allow policy makers to make evidence-based conclusions about the quality of schooling within their system and in comparison to other education systems around the world. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 7 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2017 Ukraine EMIS Results This section presents the main results of the EMIS analysis described in the previous sections. Results and scores for each policy goal are presented along with supporting evidence.  Policy Area 1: Enabling Environment Established  Ukraine’s enabling environment was assessed in the following areas: (1) Legal Framework; (2) Organizational Structure and Institutionalized Processes; (3) Human Resources; (4) Infrastructural Capacity; (5) Budget; and (6) Data-driven Culture. Policy Lever: Legal Framework The Institute of Educational Analytics (IEA) is responsible for education data collection, storing, maintenance, processing and dissemination. The IEA was founded in 2014, based on Order Number 687 issued by the Cabinet of Ministers of Ukraine, which was updated by Order Number 444 in 2015. The IEA has been institutionalized as a state research organization subordinated to the Ministry of Education and Science of Ukraine and its Statute (charter) dictates data collection for the entire education system, creation of education databases and the development of information and communication technology (ICT) to support the data production process. The IEA’s aim is to ensure a unified methodology in the education data production and dissemination process. Table 3 provides an overview of the official aim and objectives as stipulated in Order Number 444. Table 3: Main Aim and Objectives of the Institute of Educational Analytics (IEA) Aim The main aim of the Institute is to develop and implement educational system analysis in Ukraine. Objective I:  Analysis of qualitative and quantitative indicators in education Monitoring and  Legal, organizational, scientific and methodological and informational support of analysis educational process  Innovative education analysis activity  Gathering statistics on the state of the education system  Creating databases and analytical guides in education  Applied education research  Monitoring of qualitative and quantitative indicators of status and educational processes Objective II:  Implementation of applied research in education Monitoring and  Monitoring of qualitative and quantitative indicators of education and implementation implementation of their results in the educational process  Development and implementation of information and communication technologies for the collection, transmission, processing, storage and dissemination of industry information based on a common methodology  Studying the experience of the world's educational systems analysis and comparison of the national education policy of foreign countries  Implementation of analysis, interpretation and visualization of data Source: IEA 2015. In 2016, the Ministry of Education and Science (MoES) issued a Decree Number 1054 requiring all education institutions to record and provide education data. It stipulates clear data reporting responsibilities and timelines. The local education departments oversee the process of data entry and check data for accuracy. The IEA coordinates and supervises this process and provides support with data SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 8 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2017 entry, processing and analysis. The MoES made local authorities aware of the data collection and reporting responsibilities by sending official letters about the Decree to local education departments, which later forwarded the letters to the schools in their districts. Private institutions are included Table 4: Enrollment by School Type in the Ministerial Decree 2012 2013 2014 2015 Number 1054. This ensures that Pre-primary education all private schools provide Public 1,341,677 1,187,221 - 1,084,719 relevant education data. Even Private 12,717 11,351 - 9,224 though current enrollment in % private 0.9% 0.9% - 0.8% private schools is limited to less Primary education than 1 percent of total Public 1,576,035 1,629,955 1,677,268 1,527,720 enrollment (table 4), it is Private 8,347 8,542 7,762 8,858 important to ensure their % private 0.5% 0.5% 0.5% 0.6% inclusion in the data collection Secondary education process to fully capture all Public 2,888,214 2,792,779 2,703,266 2,361,311 aspects of the education system Private 10,902 10,585 10,380 8,987 in official statistics. % private 0.4% 0.4% 0.4% 0.4% Source: UIS 2017. The current IEA Statute requires the IEA to define conditions and mechanisms of information security in the EMIS, including personal and confidential data security. The Statute ensures the existence of data protection in the form of a password, which denies unauthorized access to the system. The processes of registration and creation of a login and password for authorized users are clearly outlined in the EMIS user manual. The focus in the manual is on confidentiality. In fact, Ukrainian statistics law protects education data and it is deemed confidential, even if there is no personal data in the specific data set. The MoES is currently drafting an updated IEA Statute, which will facilitate EMIS institutionalization and expand its capacities. The current IEA Statute focuses on data collection and processing as one of the objectives of EMIS. The forthcoming IEA Statute, which is not yet approved, will go beyond these aspects and focus on a data-driven education policy. It will strengthen the legal framework for EMIS and will be complementary to the Decree Number 1054, which was issued in August 2016 and ensures data collection from all education institutions. It is likely to increase IEA capacities to facilitate its role as the main EMIS agency. Policy Lever: Organizational Structure and Institutionalized Processes The legal framework requires that all local education departments check for accuracy and completeness of the data received from the schools. The aim is to ensure quality data through legal institutionalization. However, at this point these are the only requirements concerning data quality and the data validation requirements are not necessarily followed. EMIS regulations authorize the administrators to restrict users’ access to the DISO system for breaching its operating procedures. However, due to limited infrastructural capacities the IEA does not have the means to effectively discipline district and municipal education departments. In fact, there are no sanctions for delays in submitting EMIS forms or for inaccurate data entry. As part of the annual school census, local governments have to account for all school-aged children in their territory. According to Ukrainian law, local authorities are required to account for all children between the age of 6 and 18, including out-of-school children. In fact, local teachers often visit all households and record the presence of children. At the beginning of the academic year, every school SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 9 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2017 compares the list of its students with the list of all children who live in the territory, assigned to this school to determine the rate of out-of-school children. However, this data is not yet recorded in the EMIS. The legal framework allows for data collection and data access by other government agencies and third parties both before and after the introduction of EMIS. The current education data regulations allow third parties and other government agencies, meaning other institutions than IEA, to process and disseminate education data. For instance, prior to the launch of the current EMIS all education statistics were collected and aggregated by local education departments, which transferred the data to the MoES. The MoES provided the State Statistical Service with the aggregated data, which then analyzed and published the statistical yearbooks for all levels of education. Currently the State Statistical Service of Ukraine collects the necessary educational statistics data on its own. After the implementation of the EMIS full-scale operational mode, the state statistical data formation procedure is planned to be amended. Starting in 2019 the state educational statistics data will be summarized by the Ministry of Education and Science of Ukraine and submitted to the State Statistical Service of Ukraine. Policy Lever: Human Resources Recruitment and retention of qualified IT and data specialists remains challenging. Private sector positions tend to be more (financially) competitive than IEA equivalents requiring a similar skill set. IEA workers, being the employees of a budget-funded institution, have to follow a career path defined by the Labor Code, which includes salary rate schedules as approved by a special government decree. Therefore, IEA finds it difficult to recruit and maintain qualified personnel. In fact, many IEA employees do not have enough experience and qualification with database work. Due to the lack of IT and statistical specialists, there data quality assurance is challenging. In general, the job of quality assurance partly falls on IEA director, who also formally acts as EMIS manager, and his deputies. Staff have some access to training. EMIS staff at IEA receives training on SPSS software, which is aimed to enhance their data processing and analysis skills. In addition, staff receive access to open access resources from the internet, which may be used to facilitate and enhance data production processes. However, at the school level there is a lack of technical skill and training on how to correctly enter data. This potentially harms data quality as the data entered at the school level forms the base for any further data processing. Policy Lever: Budget The IEA budget covers all EMIS-related activities. There is no designated “EMIS budget”, but rather the MoES includes a specific IEA budget in its annual budget request. The respective job descriptions at the IEA include EMIS responsibilities, which in turn make expenses by the IEA also EMIS expenses. The IEA budget also covers expenses on personnel, maintenance, technical support and equipment. However, there is no designated budget for software or hardware specifications within the IEA budget. In recent years, the Swedish-Ukrainian project supporting decentralization efforts has been instrumental in the development of the current EMIS. Policy Lever: Data-driven Culture Data-based decision making has traditionally focused on the allocation of resources. Even before the official launch of an EMIS, education data had been collected and utilized for funding allocations. However, recently a new data-driven culture has been on the rise as the government has acknowledged the importance of data and analysis in the decision making process. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 10 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2017  Policy Area 2: System Soundness Emerging  Ukraine’s EMIS soundness was assessed in five critical areas: (1) Data Architecture; (2) Data Coverage; (3) Data Analytics; (4) Dynamic System; and (5) Serviceability. Policy Lever: Data Architecture The Ukrainian EMIS was privately developed and did not follow a specified wireframe for the system architecture, which could negatively affect future plans on system expansion. However, some specifications guiding the structure of the EMIS and its databases did exist. The system was initially designed by a private company in 2009 and selected by the MoES in 2011 for EMIS purposes through a single tender process. The system was then transferred to the control of the IEA in 2016. There are two flows of data from schools: electronic and paper-based. According to Ministerial Decree 1054, each school uploads electronic education data reports directly to the DISO database through a web- based application. In parallel, the school also sends a paper version of the report to the local education department. At the beginning of the year, schools have to submit the ZNZ-1 (student data) form no later than September 12th and the 83-RVK (teacher data) form no later than October 10th. The local education department has access to the electronic as well as the paper reports. It aggregates the data and compares both versions to check for potential errors and inconsistencies. Next, the local education departments send aggregate data from the paper reports to the regional (oblast) education departments, which also verify the paper and electronic versions of each report. In theory, district and municipal education departments are tasked with overseeing data entry at the school level, while the regional education departments supervise the data collection process by the district and municipal departments. However, the mechanisms to correct missing, invalid or incomplete data are rather limited. This could be minimized or avoided by implementing data checking techniques to question unrealistic or missing data at entry or aggregation phases which then be verified from schools and corrected. The regional departments compile the aggregated education reports for the respective region and send their reports to the IEA. The failure to submit a form or delay in doing so is a violation of the Ministerial Decree 1054. At the end of the data collection cycle, the IEA receives the aggregated paper reports in addition to the electronic data through the DISO system. Before December 5th of the current school year, the IEA must submit aggregate statistical education reports to the State Statistical Service (figure 5). SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 11 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2017 Figure 5: EMIS Information Flow Chart Ministry of Regional (Oblast) Analytical Education and Department of Reports Science (MoES) Education Supervise Orders Decree 1054 Institute of District and Educational Ensure that school data is verified Municipal Analytics (IEA) Department of Education Compare digital and physical Data flow school questionnaires State Information Schools System of Data flow Education (DISO) Hard copy of questionnaires Source: Authors. The EMIS data is hosted on servers in the building of the IEA. There is currently one database in place, namely the State Information System of Education (DISO). The plan is to archive education data indefinitely. Data backup plans should be included in the EMIS to ensure that the data will not be lost in case of a software or hardware malfunction. In general, the database is fed with data from schools and the few schools that are not connected to the internet bring external media in addition to paper copies to the local education departments. The capabilities of the current DISO are limited, which is why plans have been made to introduce a new DISO database with expanded capabilities. Plans have been made to introduce a new DISO database, which would extend data collection scope, maintenance and analytical capabilities. The forthcoming DISO database will likely expand data collection, entry, storage and analysis capabilities of the current EMIS. The new DISO is expected to expand data collection by introducing a registry of secondary education facilities (“school registry”), collecting human resource, individual student and financial data (figure 6). An accurate school registry – of private and public institutions at every level of the education system – will facilitate data collection and will be an instrumental part of the new DISO (Herczyński 2016). Such a school registry should be complemented by unique school identifiers to track school progress. Furthermore, a robust data backup system should be added to the new DISO. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 12 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2017 Figure 6: Potential Data Collection with Updated DISO System Admin. Data Admin. Learning Data Data Current Data New Data Potential Transition Warehouse Warehouse (current DISO) (new DISO) Finance H.R. Data Data Source: Adapted from Abdul-Hamid 2014. Policy Lever: Data Coverage Data collection in the EMIS could benefit from more detailed student and teacher data such as information on student learning outcomes. For now, the data coverage in the EMIS is defined by the two state statistical forms sent to schools, namely the ZNZ-1 (student data) and 83-RVK (teacher data) forms. However, in general electronic EMIS data is quite limited. In particular, there is a lack of coverage on student learning outcomes, human resources, and financial information (table 5). The EMIS captures aggregate data, such as enrollment numbers by gender and absolute enrollment numbers of students from low-income families, and also gathers some basic information at the level of the school, but contains no student-level data. There is no effective system of tracking schools, teachers or students over time. Even though, in theory, a longitudinal data system has existed in Ukraine since 1991, it has been ineffective for longitudinal tracking. In 2003, an attempt to rectify this was made by establishing the Ukrainian Longitudinal Monitoring Survey (ULMS), which also included a section on the individual’s educational attainment and skills (Lehmann, Muravyev and Zimmermann 2012). However, the ULMS has not been designed for effective educational tracking and no individual data is included in the EMIS. Importantly, there are no effective universal unique identifiers assigned to students, which makes it challenging to track students across schools and levels of education. Additionally, there is no universal school registry with unique school identifiers, which hinders the government’s ability to track the progress of individual schools. Similarly, no unique teacher identifiers exist and human resources information is available only in aggregate form (such as the number of teachers by age group or level of experience). Therefore, some indicators at the level of the school, such as student-teacher ratios are not present in the EMIS, but can be inferred from the reported number of students and number of teachers. No information on teaching evaluations or professional development data is included in EMIS. Financial data is also absent from the EMIS. In fact, all financial records are collected and maintained by the Ministry of Finance (MoF). However, even the MoF does not capture financial data on each school and only reports district-level data. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 13 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2017 Table 5: Data Coverage Fully Integrated Expanding Beyond the School Census: Best Practice, Ukraine Data Type Best Practice Ukraine Administrative data  Demographic  Demographic  Health records  Attendance (enrollment, repetition)  Attendance  School level data (student-teacher ratio, student-  School level data classroom ratio, student-school ratio, number of classrooms, students per classroom, etc.) Financial data  Budget and revenues  Not included in EMIS, but possible to estimate unit  Spending costs per student at district level from MoF data  Cash transfers and subsidies (financial data is collected by the MoF and not fed  Unit cost per student into EMIS) Human resource data  General demographics  Some basic demographic data  Salaries  Performance evaluations  Professional development Learning outcome data  Classroom assessments  Not included in EMIS  National assessments  International assessments Source: Adapted from Abdul-Hamid 2014. The Ukrainian EMIS is not yet fully integrated with other MoES, other ministries or government agencies’ databases. The EMIS is a stand-alone system and, for instance, has no link with the financial databases of the MoF. The Ministry of Finance is in charge of data collection on education expenditures (at district level) and this data is not shared with the IEA. Information sharing and data integration can lead to insightful and effective policy conclusions. This starts with establishing inter-ministry data sharing arrangements, which include the integration of all education databases such as, for instance, financial data and learning outcomes data. Additionally, linking the DISO database with other data sources (for instance, those maintained by the Ministry of Health or the Ministry of Social Policy) can bring many benefits. Integrating learning assessment and labor market data into the EMIS can facilitate a skills-gap analysis between employer needs and graduate skills. Policy Lever: Data Analytics There are limited statistical capabilities integrated into the Ukrainian EMIS. The system is not yet capable of performing basic statistical analysis and is unable to perform more advanced evaluations such as population projections, scenario analysis, and predictive models. All of these features would be beneficial to decision makers. Plans have been made to introduce basic statistical capabilities and automated report generation into the new DISO. However, until the new DISO is designed and operational it is unclear what the new features will be. So far, it is unclear if advanced statistical capabilities will also be introduced into the new DISO. Policy Lever: Dynamic System The EMIS should be able to adapt to new technologies and requirements. It is not yet certain if the current system can be adapted to fully integrate new technologies. The EMIS software was first developed in 2009 and so far some compatibility issues with the data collection system in Kyiv have occurred. The Kyiv city administration uses its own statistical reporting system for schools, which was developed by a separate company and it is yet not compatible with the EMIS. Compatibility could become one of the main challenges for the development of the new DISO. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 14 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2017 To adapt EMIS to arising data demands and new education needs, IEA personnel regularly conduct meetings with stakeholders to discuss enhancements in the EMIS. These meetings are particularly important as the IEA is preparing the design and architecture for the new DISO. In fact, the IEA has plans for including new data requirements and capabilities. However, system adaptability remains a challenge. Currently, the EMIS is able to include new variables at the school level, but cannot yet integrate data on individual teachers and students. Policy Lever: Serviceability The lack of individual-level data reduces data security concerns, but it is also a missed opportunity. There have not yet been any cyber-attacks on EMIS and these remain unlikely. However, there are also some security measures in place to prevent potential future attacks. Including individual-level data on students and teachers will allow EMIS to paint a more accurate picture of the state of the education system. As soon as individual data is included as one of the next steps in the evolution of the Ukrainian EMIS, data security will likely need to be enhanced and supported by the right policies on data exchanges and access Dissemination occurs through the newly launched IEA website. Recently, the IEA launched its own website (http://iea.gov.ua/) where education information can be published and stakeholders can reach IEA staff directly to improve data utilization. In addition, the annual statistical reports on secondary education are prepared by the State Statistical Service and are published on its website. The data for the annual statistical reports is based on the aggregated data from the regions as provided to the IEA. In addition, the MoES had always compiled a separate statistical report, which it published on its website. Starting from this year, the IEA will take over this task and will publish the reports on its own website.  Policy Area 3: Quality Data Emerging  The quality of data captured by Ukraine’s EMIS was assessed in four areas: (1) Methodological Soundness; (2) Accuracy and Reliability; (3) Integrity; and (4) Periodicity and Timeliness. Policy Lever: Methodological Soundness The MoES has ensured that the school census data collection forms have been designed to be easily comprehendible. A template provided to data collectors is appropriate for computer processing. The questionnaires represent the base for the annual publication of education statistics and the indicators are compiled utilizing sound methodological practices. The data for the annual statistical report is generally reliable data, but data validation and timeliness challenges exist. Consequently, due to limited data validation and correction capacities at the IEA the data may sometimes be erroneous or incomplete. Subsequently, the drawbacks regarding the limited data validation and correction capacities in the EMIS are expected to be eliminated by improving the methodology of the statistical data processing and implementation of the contemporary information technologies. The EMIS processes are reviewed internally, but the quality checks do not yet follow standardized, well- defined and documented procedures or internationally recognized standards. EMIS data is sourced at the school level with data validation points at the local and regional education departments in addition to the central level in the IEA. Data quality assurances are restricted to internal checks only and need to be brought closer in line with international standards. The quality checks occur on a monthly basis, but due SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 15 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2017 to the lack of defined procedures and international standards, the effectiveness of those quality assurances is insufficient. Policy Lever: Accuracy and Reliability Currently, the IEA has limited capabilities to correct erroneous data. Due to the limited statistical skills among EMIS staff, there are no imputation or estimation mechanisms in place to correct for missing, invalid or inconsistent data. There are also no descriptions available in the EMIS that explain missing/unfilled data. In order to enhance data quality, regular revisions and upgrades to statistical capabilities and processes are essential. This can be problematic as policy makers generally rely on the aggregate information presented to them in the annual statistical yearbook to make policy decisions. In order to achieve comprehensive data utilization, the EMIS has to produce high quality detailed data, which will then be disseminated in a timely manner and use-friendly format to encourage utilization. Terminology and concepts are defined in the operations manual of the Ukrainian EMIS. The concepts are in accordance with national datasets and follow national data concepts. In fact, the data included in DISO as approved by the MoES does contain a provision to link the EMIS definitions to other education laws in Ukraine. Description of data fields and metadata indicators are included in the operations manual. However, the definitions do not follow international guidelines nor are data collectors and engineers aware of international differences in the concepts, definitions and terminology. Education system classifications are in line with international standards . The documented education system classifications are based on technical guidelines and manuals as prescribed by the UNESCO Institute for Statistics (UIS). The State Statistical Service ensures the consistency of the classifications and definition as it is also responsible for reporting education data to UIS. In addition, the IEA has also developed a system of quality secondary education indicators that are based on OECD definitions. The definitions set forth by the State Statistical Service and the IEA ensure that the reported data is compatible with nationally and internationally reported data (upon approval from the State Statistical Service of Ukraine). Policy Lever: Integrity EMIS data is subject to internal reviews and staff are encouraged to participate in conferences and meetings with other professionals to gain insights into best practices in education statistics. To ensure data quality, internal reviews and exchanges with other professionals in the same field are essential. Internal reviews and data validations occur by comparing digital and paper copies of education data. Both internal data reviews and professional exchanges maintain the professional reputation of the IEA and may also introduce new ideas and statistical methods to ensure data quality. Effective internal reviews and exchanges with other professionals at conferences and other fora are linked to enhanced data quality and are important components of professional development for EMIS staff. The IEA does not prescribe codes of professional conduct for its staff, potentially reducing data quality and accurate data interpretation. Currently, there is no code of professional ethics that dictates how to process education data and ensure EMIS staff integrity. As there is no confidential and personal data in the EMIS, data access is not restricted by passwords and all staff within the IEA can view the data. This wide accessibility of data heightens the importance for data explanations. However, the code of professionalism for Ukrainian EMIS staff does not yet prescribe the publication of explanatory materials or briefings to the public to prevent potential misinterpretation of data. Misinterpretation of data can directly lead to ill-informed decision making that can negatively impact the learning outcomes of students. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 16 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2017 Policy Lever: Periodicity and Timeliness Delays in data collection are possible. Schools and local education departments are responsible for providing education data and also for the relevance, consistency and timeliness of such data. In theory, schools have to provide data immediately after the start of the new school year according to the MoES Decree 1054, but in practice some delays occur. The IEA does not have the capacity or relevant authority to ensure the timeliness of data and there are no processes in place to sanction schools for violating the rule. For instance, in 2016 approximately 5 percent of schools had not yet submitted the ZNZ-1 forms and 15 percent of schools had not submitted the 83-RVK forms approximately three to four weeks after the deadline. However, in general most data will be available for the public within 6 months.  Policy Area 4: Utilization for Decision Making Latent  The utilization of Ukraine’s EMIS was assessed by examining four areas: (1) Openness; (2) Operational Use; (3) Accessibility; and (4) Effectiveness in Disseminating Findings. Policy Lever: Operational Use Operational use of EMIS data is limited. In general, almost all data utilization is reserved for the central and regional level. The MoES, regional education departments and other government agencies use education data in their planning. However, the utilization possibilities are limited to the most basic statistics in aggregate form, such as the number of schools, students, teachers. Mostly education statistics are used for the allocation of resources at the national and regional levels. This includes the determination of performance ratios such as student-teacher, student-space and student-staff ratios. Table 6 provides an overview of best practice uses of EMIS data and how education data is currently utilized in Ukraine. Table 6: Operational Use: Best Practice, Ukraine Data Type Best Practice Ukraine In evaluation  School performance  Overall education performance  Student performance  Growth reports  Diagnostic reports  Graduation rates  Transition rates  Teacher performance  Overall education performance In governance  Policy decisions  Policy decisions  Accountability  Planning (national and sub-national  Planning allocation of funds)  Management By schools  Academic performance  Not included  Teacher performance  Management  Comparison with other schools By clients  Parents’ access to EMIS  Not included  Communities’ access to EMIS  Use data to make decisions SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 17 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2017  Use data to demand quality By government  Performance ratios  Performance ratios  Infrastructure capacity  Infrastructure capacity  Quality/outcome indicators  Quality/outcome indicators  Spending efficiency  Gender equality  Teacher salaries  Equality indicators Source: Adapted from Abdul-Hamid 2014. The Institute of Education Content Modernization (IECM), a body which reports to the MoES, is tasked with applying education data and research for the purpose of education modernization. In 2014, the Cabinet of Ministers issued Decree Number 687, which established the IECM as well as the IEA. According to Decree 687, “the purpose of the Institute [IECM] is the scientific and methodological support modernization of educational content, the process of education, development and socialization through basic and applied research and implementation of their results” (IECM 2014). This means that, in theory, education research is directly linked to improving education content such as curricula and textbooks to improve student learning. Table 7 provides an overview of selected tasks that the IECM carries out. The focus lies on education research for informed policy decisions and implementation regarding education content. Table 7: Selected Activities of IECM Relevant to Education Data and EMIS Research  Realization of fundamental and applied research on issues of education, educational, activities psychological and related sciences  Development and implementation of educational information including analysis of the current state of the education system as a whole and its individual sectors and forecasting development trends  Development education policy guidelines based on the results of scientific research (monographs, textbooks, handbooks, manuals, reports, etc.) in addition to guiding documents aimed at the implementation of research and development in the practice of educational institutions Scientific-  Provide scientific and methodological support for the modernization of education content, organizational education, development and socialization and  Drafting of regulations on education educational  Providing organizational and educational assistance to individuals and legal entities that are activities engaged in informal education Innovative  Design, pilot testing and implementation of the educational process innovation and ICT activities training and education Editing and  Realization of publishing, manufacturing and distribution of publishing products publishing International  The Institute takes part in organizing and conducting international cultural and educational cooperation events (conferences, seminars, webinars, exhibitions, presentations, etc.) by cooperating with international organizations in Ukraine and abroad Source: Adapted from IECM 2014. Policy Lever: Openness and Accessibility Local governments do not have the ability to produce sophisticated education analysis and thus, data is barely used in their decision-making processes. In fact, at the local level education data is only used at the initiative of official representatives, if at all. They may utilize EMIS data to make informed decisions on how best to organize their school networks (e.g., which school to close or reorganize and where to place the students). However, education statistics act as a supporting factor and not a decisive contributor SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 18 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2017 in the decision-making process. The exception are big cities such as Kyiv where the Department of Education, Science and Youth under the Kyiv City Council analyses education data and produces statistical reports, which provide a basis for informed policy decisions. Generally, parents do not use EMIS data for decision-making purposes. The usage of education data by parents is not actively encouraged by the government nor facilitated by how education data is presented. No “open data” portal currently exists and extrapolating information from aggregate education statistics for informed decisions is difficult. Unless the format and availability of comprehensive education statistics is enhanced, parents will not be able to make effective use of education data. In fact, only a very small number of parents use school data in combination with data on external assessment results to select schools. Non-governmental organizations (NGOs), research and multilateral organizations are able to make limited use of EMIS data. For research and education evaluation purposes these organizations utilize EMIS data. However, the aggregate presentation of the data and the limited number of indicators make sophisticated analysis and data manipulation for research purposes challenging. In addition, primary data from EMIS is not in the public domain and access has to be requested on a case-by-case basis. A lot of analysis for research purposes is based on the annual statistics report published by the State Statistics Service, which presents education data in aggregate form. Policy Lever: Effectiveness in Disseminating Findings There have been some efforts to make education stakeholders aware of EMIS data. With the launch of the DISO system, the MoES had issued press releases announcing the new data management systems to the local departments of education. However, no leaflets, regular announcements via digital means such as e-mail or the website have taken place. Moreover, the IEA’s newly launched website does not yet have the capability to disseminate education data in user-friendly disaggregated form. There are no trainings on data utilization for education stakeholders. The annual report includes tables and charts to facilitate data analysis, but some education stakeholders may require additional training on their interpretation. Currently, there are no available trainings on how to interpret, analyze, manipulate or utilize education data produced by the EMIS. NGOs, experienced research and international organizations are likely to have or develop the necessary skill set to perform these processes, but other stakeholders would likely benefit from additional trainings to make data utilization possible. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 19 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2017 Recommendations and Proposed Activities This section presents a set of recommendations and proposed activities based on the assessment of EMIS in Ukraine (table 8). Recommendations and activities aim to Table 8: Ukraine EMIS Rankings improve the overall EMIS functionality in a Established 1. Enabling Environment sustainable and effective manner, to ensure  better access and use of information for Emerging 2. System Soundness decision making, planning, and student  learning. Future activities to improve the EMIS Emerging should be strategically designed such that they 3. Quality Data  incrementally boost dimensions of the EMIS to Latent a more advanced level, ultimately improving 4. Utilization for Decision Making  overall EMIS functionality in a sustainable and effective manner. The Strengths Weaknesses Opportunities Threats (SWOT) identification (Table 9) summarizes key points from the needs assessment and informs recommendations. Table 9: Ukraine EMIS SWOT Profile Strengths Weaknesses  Political will to establish an operational EMIS  Limited data utilization  Establishment of a sound legal framework  Limited scope of collected data  Clear allocation of responsibilities to the IEA  No capabilities to conduct advanced statistical  EMIS expenses are included in the IEA budget analysis  Indicator definitions and classifications follow  No trainings on data interpretation for education international standards stakeholders (training is generally reserved for EMIS  EMIS includes all schools (private, state, and staff) community-owned, with and without internet  Lack of financial capacity of the IEA to hire and connection) retain highly qualified staff in comparison with the  Ability to conduct an annual school census that private sector generally produces reliable data  Potential compatibility issues with other data systems  No national school registry with unique school IDs currently exists  Insufficient mechanisms exist to correct missing, invalid or incomplete data at the central level Opportunities Threats  Support from international organizations  Limited stakeholder buy-in due to limited utilization  Data access is granted to the public, capabilities researchers, NGOs and international  Insufficient data awareness by education organizations stakeholders  Scope and political will exist to establish an  Lack of clear blueprint for EMIS expansion Open Data Portal and integrate EMIS with other  No sanction processes in case of non-compliance data systems with prescribed deadlines, data protection and  Data validation mechanisms are in place at the confidentiality measures local level and could be expanded centrally  Only basic EMIS and statistical training in data collection, analysis and statistical comprehension is available for EMIS staff SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 20 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2017  Recommendations 1: Enabling Environment Key Recommendations:  Fully institutionalize the EMIS following a specified blueprint, which covers the legal framework, organizational processes, institutional capacities, human resources and budget.  Update the IEA Statute to expand relevant capacities and strengthen the organizational EMIS infrastructure.  Ensure that a designated EMIS budget is embedded in the IEA budget to account for EMIS- specific expenses such as personnel training, hardware or software requirements. This is important to avoid overreliance on donor funding of EMIS activities. The EMIS needs to be fully institutionalized so that operations are sustainable in the long term. A well- defined EMIS policy should have clearly stated procedures and regulations for (i.) central, local, and school reporting requirements, (ii.) defined responsibility and ownership of data, (iii.) allocation of the EMIS budget, (iv.) data collection processes, (v.) data submission requirements, (vi.) procurement guidelines (if purchasing the software/hardware from an external vendor), (vii.) technical specifications of the EMIS, (viii.) type of data collected, (ix.) data validation mechanisms (internal and external), (x.) confidentiality policies, (xi.) dissemination strategy, (xii.) extent to which data should be utilized by stakeholders, (xiii.) code of conduct for staff, and (xiv.) professional development activities. In order to fully institutionalize the EMIS in the long term, a blueprint for its development can be helpful. Currently, the EMIS architecture development follow specific guidelines, but does not yet have a clear blueprint for future development. A clearly articulated and publicly available design and implementation plan for EMIS raises transparency and accountability and provides policy makers with a roadmap on how to establish an operational EMIS. This may be particularly important in times of political instability. The new IEA Statute should aim to enhance IEA data capacities so that reliable, accurate and timely data can be produced, disseminated and utilized. Designated EMIS budget provisions should be made to ensure EMIS sustainability in the long term. Reliance on foreign project funds to finance EMIS operations and development decreases sustainability of the EMIS. Currently, international donors contribute funds and technical assistance to the establishment and implementation of the Ukrainian EMIS. However, overreliance on donor project funds is not sustainable in the long-term. There is a need to develop stable and long-term budget lines for EMIS that are separate from the current IEA operational budget provisions. Among others, there should be a designated EMIS budget for data specialists, statisticians, IT personnel, software and hardware specifications to cement a well-functioning EMIS. Increasing appeal and competitiveness of EMIS staff positions can help the government ensure that specialized staff can be hired and retained. Qualified personnel are essential in any EMIS. In order to hire and retain qualified personnel the MoES needs to compete with the private sector. In general, the lack of appeal and (financial) competitiveness with the private sector is a factor deterring highly qualified staff from accepting employment within the MoES and the IEA. This can pose a risk to continuity in the management and development of the EMIS. The expansion of statistical capabilities in EMIS also includes the provision of appropriate trainings to EMIS staff and other stakeholder groups. Professional development plans and trainings will enable EMIS staff and education stakeholders to strengthen their analytical skills. To ensure that trainings and SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 21 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2017 professional development takes place, the EMIS budget should account for them. Annual evaluations of EMIS staff can complement professional development plans to ensure that all staff truly benefit from the provided trainings.  Recommendations 2: System Soundness Key Recommendations:  Expand data coverage to include human resources, financial, and learning outcomes data.  Upgrade statistical and analytical capacities and linkages of DISO with other data systems.  Strengthen data validation and quality assurance standards.  Include data backup plans to ensure that the data will not be lost in case of a software or hardware malfunctions. Under the new Statute, data collection should be expanded to include human resource, financial, and learning outcomes data. The Ukrainian EMIS already covers an array of general administrative indicators, but in order to paint a comprehensive picture of the entire education system, more indicators should be captured. Human resource data on professional qualifications, development, training and experience of staff should be included. This could be linked to unique staff identifiers, which track teachers and other staff throughout their career. The longitudinal tracking may facilitate important policy conclusions. Financial data is currently only collected by the Ministry of Finance, but it is essential to fully integrate it into EMIS (possibly through data sharing arrangements between the MOES and the MoF). By tracking how and where funds are spent and how spending is linked to learning outcomes allows for important policy decisions in the allocation of funds. Education databases should be integrated with each other and linked to other databases, which may provide information relevant to education. Database integration expands beyond education data (early childhood, primary, secondary and tertiary education), but should also include other relevant data. For instance, students’ health data could provide an overview of the overall status of youth health in Ukraine. Linking enrollment data to national ID and civil registration databases may offer more efficient processes on determining the number of out-of-school children than sending teachers to collect data by going from house to house. Financial data, labor market data, university admissions data, and so forth can greatly expand the range of analytical information available to education sector policy makers. Statistical capacity of EMIS should be expanded. In order to build a strong base for informed decision- making, the EMIS should be able to perform advanced statistical analysis. Currently, the EMIS is able to perform basic statistical analysis such as calculation of transition rates, enrollment number, etc., but for many policy decisions additional data analysis is required. The EMIS could benefit from expanding its capabilities to include scenario (“what-if”) analyses, predictive modeling, efficiency analyses, and population projections. The modernization of the State Information System of Education (DISO) will be an integral part in expanding the scope of data collection and statistical capacity. As the planning and design of the new DISO is currently underway, it is important to include many essential features to strengthen EMIS capacity in the long term. The new modules should include a school registry, human resource, student and financial data. In addition, data entry, export and report generation should be included in the new DISO design. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 22 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2017 Table 10 provides an overview of some suggested modules. Data sharing and linkages across different ministry databases should be taken into consideration for the new DISO design. Table 10: Potential Features of the New DISO Registry of The school registry should include all education institutions starting at the pre-school level up to secondary the end of general secondary school (and integration with higher education should be made education possible). It should also include evening schools, special schools, boarding schools and facilities professional schools. DISO should perform the following tasks: (“School  Collection and maintenance of statistical data about students and teachers; Registry”)  Student data about professional schools should be imported from EDEBO using an agreed- upon format;  Preschool data should be included over time. The MoES should regulate the introduction of the new DISO through the issuance of policy documents to cement its institutionalization. Human  DISO can collect and maintain data on each individual teacher and school administrator, who resource will be identified by the staff member’s unique tax-payer identification number; data  DISO should be able to automatically generate RVK-83 (“teacher”) reports both on the number of physical persons and full-time equivalent staff (“stavkas”);  Plans should be made to include data collection on human resources by 2019. Student data  DISO should become the main data collection center for all student data, which also requires the capabilities to add new statistical item lines on students;  Eventually DISO should be able to capture individual student data including date of birth, attendance, enrollment, and progression through the education system;  DISO should be flexible and adaptable to the future education needs. Financial DISO should be able to collect and maintain all education expenditure data from all institutions, data private and public. It should be obtained from a variety of sources, including the Ministry of Finance. It should include all education expenditure information, such as:  Staff wages and salaries, as well as other non-salary remuneration for all staff categories;  Investment in school infrastructure;  Spending on teaching and learning materials (e.g., textbooks) and other operating costs;  Expenses associated with staff training and professional development. Data entry Data entry into DISO can occur in different ways:  Digitally through a web-based interface;  Collected through third-party software (e.g. ISUO, AS Shkola);  Through inter-agency and inter-ministry data sharing arrangements (e.g., linkages with other education data systems, such as EDEBO, and those of other ministries, such as the MoF). Data export The new DISO should include automatic report generation: and report  DISO should be able to easily generate the mandatory statistical reports on students and generation teachers;  Some reports and custom data queries should be made possible through a publicly available Open Data Portal;  All statistical report automation should be designed to include a level of flexibility so that the reports can be adjusted in the future;  A custom report generator should allow users to export data in various formats (Microsoft Word, Excel, PDF). Source: Adapted from Herczyński 2016. Data backup plans should be included in the activities to ensure that the data will not be lost in case of a software or hardware malfunction. Different options are available at reasonable price as hard disk space prices are getting cheaper and cheaper. Cloud storage options could also be adopted in a secure and inexpensive manner. Other countries use standalone backup servers as a major component in the SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 23 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2017 data warehouse architecture. This will keep the system going in case one server went down a backup will take over. Hence a replication mechanism need to be implemented based on a schedule to keep the information current at all servers. The EMIS should exhibit built-in quality assurance standards to ensure continuous improvements of the data management system. Figure 7 illustrates what potential quality assurance processes could look like. Here the use of International Organization for Standardization (ISO) 9000 quality processes has been adapted to assess the soundness of any EMIS. The aim is to ensure continuous improvements of EMIS and create a system that is open, compatible and adaptable. In general, education data systems and databases should be compatible with each other and able to adapt to new technology. The Ukrainian system was originally designed in 2009 and it is unclear how compatible and adaptable it is with modern technology. In fact, the Kyiv City education data system is currently not compatible with the IEA systems. In order to ensure that all education data seamlessly feeds into one single EMIS, compatibility is key. In addition, the system needs to be open and adaptable to new technology and education needs. Figure 7: Use of ISO 9000 Quality Processes to Assess the Soundness of an EMIS Source: Adapted from ISO 2016. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 24 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2017  Recommendations 3: Quality Data Key Recommendations:  Ensure accurate, reliable and useful data to improve learning outcomes through informed decision making.  Improve data validation capacities within the IEA and through a system of external inspectors and feedback providers.  Equip the IEA with sufficient resources, authorizing environment, and internal capacities to ensure timely data collection. One of the objectives of any functioning EMIS is to provide accurate, reliable and useful data, so that improved learning outcomes for all students can be achieved. Therefore, it is highly important to capture as much learning data as possible in the EMIS. Currently, the Ukrainian EMIS does not include learning outcome indicators and external assessment scores are not linked to the EMIS. In order to track learning developments over time and identify areas for improvement, IEA systems should be integrated with the learning outcomes data currently being collected by the Ukrainian Center for Educational Quality Assessment (UCEQA). It is important to collect data in disaggregated terms (at the level of schools or individual students) and present it in breakdowns by rural/urban, private/state/community-owned, gender, age, socio-economic background, ethnicity and region. Data validation capacities are very important to ensure data quality. IEA capacities and authority should be enhanced so that data correction processes can easily take place if incomplete, erroneous or missing data is uncovered. Quality data represents the base for informed decision making and should be considered a prerequisite for designing effective and informed education policies. A system of external inspectors can improve data validation and verification mechanisms. The current system of school inspections in Ukraine does not verify the quality or reliability of data submitted to the EMIS. However, such a system can improve data quality by correcting for missing, invalid or incomplete data. Parents and other stakeholders can also serve as external feedback providers to improve the coverage and quality of the data. In many countries inspectors visit schools to verify numbers and compare against system data. This function can be performed by current school inspectors or as part of a new quality assurance system of education envisioned under the New Ukrainian School. Other countries add the data validation to district IT or education inspectors’ functions. Romania, for example, has a role of data inspectors that could be adapted to the Ukrainian context. Timely data collection is important. The IEA should be equipped with the necessary institutional capacity and authority to ensure timely and quality data reporting from schools. In theory, penalties can be applied in case of non-compliance of deadlines. However, the IEA does not yet have the capacities to enforce timely data collection. The EMIS governing documents should include clear guidelines of ethical standards for staff to ensure data integrity. Currently, there is a lack of written ethical standards and conflict of interest guidelines, which may limit data integrity. There are no clear processes in place to enforce penalties in case of misuse or wrongful manipulation of data. This calls for an established set of ethical guidelines and processes in case of non-compliance, which is needed for the professionalization of staff. The IEA should also be provided with the necessary capabilities and authority to enforce these processes, if necessary. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 25 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2017  Recommendations 4: Data Utilization Key Recommendations:  Empower and enable all stakeholders to utilize education data.  Ensure timely access to relevant education data.  Improve data dissemination strategies and tailor data presentation to the needs of the stakeholder groups. Establishing a culture of evidence-based policy making is crucial and a particular focus should be paid to data utilization. A data-driven culture ensures that information is collected, assessed and utilized to improve students’ learning. The information provided by the EMIS and its utilization should be at the core of every education system architecture. The need for data is omnipresent and information should be shared widely. This means that policy makers should encourage data utilization and data awareness at every level. As the agency responsible for the EMIS, the IEA should build its capacity so that it can act as one of the main players in establishing a data-driven culture. Quality data is the underlying force for effective data utilization. Quality data could be considered a pre- requisite for effective decision making and education improvements. In order to effectively use data, education stakeholders should be able to rely on the data to provide them with sound and accurate information. Then, policy makers and other stakeholders can make important decisions that are based on quality information. Otherwise, policy makers may draw the wrong conclusions from inaccurate data and take decisions that harm learning outcomes. Data can be utilized beyond the allocation of resources. Any EMIS is only as good as its utilization. An EMIS wireframe should also include data utilization strategies. Currently, EMIS data is mostly used to inform the allocation of funds and sometimes as supporting data in decisions on school network organization. However, the potential operational uses for EMIS data expand far beyond the current usage. EMIS data should be utilized at all levels: central, regional and local. Ideally, data should be utilized by all education stakeholders at every level of the education system and incorporated in all decision-making processes. Data utilization mechanisms will benefit from institutionalization. To build a data-driven culture and ensure that data is used at every step of the decision making process, it should be cemented in a strong legal framework. The new Statute should clearly specify how data should be utilized and establish data utilization processes. The aim should be to institutionalize education data utilization by all stakeholders, not just the central and local government. Thus, it is necessary to amend the legislative framework, in particular to develop the new acts of the Ministry of Education and Science, the Cabinet of Ministers, and school regulations to enable schools, parents and the broader community to better use education data. This may be done by effectively disseminating education data in a way that empowers and enables all stakeholders to utilize it. Data dissemination and utilization can benefit from a focused strategy. An effective dissemination and utilization strategy should go beyond the publication of annual education statistics, but include all channels of communication. These may include online publications, newsletters, pamphlets, radio and TV, newspapers, social media, as well as the establishment of a dedicated Open Data Portal or electronic school report cards. Each channel may serve a different purpose and reach different education stakeholders. Therefore, some countries (like Chile) have designed comprehensive data dissemination SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 26 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2017 strategies in order to reach as many different stakeholders as possible (table 11). Such a comprehensive strategy would ensure that stakeholders such as parents, teachers and the broader community could be reached and empower them to utilize education data. Table 11: Examples of Data Dissemination Strategies in Chile Channel Purpose Assessment Provide pedagogical support to school principals, pedagogical coordinators and teachers guidelines School report Provide pedagogical support to school principals, pedagogical coordinators and teachers National report Inform decision makers and the general public on student performance across grades and regions Newspaper Hold schools accountable by publishing data on school performance by subject across supplement regions Parent report Hold schools accountable and involve parents in school activities Online item bank Provide pedagogical support to teachers through releasing test questions from all subject areas and target grades Press kit Inform policy makers by providing materials to journalists and regional education offices Data files Inform policy makers, provide pedagogical support and hold schools accountable to researchers with regards to their respective education topic Data analysis tool Inform policy makers, provide pedagogical support, and hold schools accountable, depending on the type of analysis, for researchers and decision makers Geo-referential Google Maps with the geographical location of schools and their average scores for system parents to hold schools accountable Ministry of Inform policy makers, provide pedagogical support and hold schools accountable Education website Source: Ramirez 2013. Dissemination strategies should be tailored to the capabilities of education stakeholders. In order to allow all stakeholders to access and effectively utilize education data, there is a need for a dissemination strategy that takes into account the data consumption and analytical capabilities of all stakeholder groups. For instance, data should be presented in a format that is easily digestible and allows for further analysis by different stakeholders, even those with limited statistical capacity. Therefore, it may be necessary to publish descriptive and explanatory manuals and guides with the data to ensure the understanding and thus, potential for effective utilization by all stakeholders. The IEA website could be an effective channel for data dissemination. Currently, the IEA publishes data on the MoES’s website, but in order to target and tailor its online dissemination strategy the IEA should further develop its own website and consider developing a dedicated Open Data Portal. This would empower the Institute to take more responsibility in the data dissemination process and allow data users to access data through one single website. Information feedback loops foster stakeholder buy-in and data utilization. In order to facilitate and encourage data utilization and system adaptability, feedback loops are an instrumental part of any EMIS architecture. Feedback loops ensure the connection between data providers (e.g., teachers, students, parents) and grant access to a broad spectrum of data users. Such access can be achieved if information flows both from the central to the regional and local level and vice-versa. For an information loop to be effective, the education data should be presented in such a manner that all stakeholder can utilize the information and improve learning outcomes. Figure 8 provides an example of how information feedback loops could be designed. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 27 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2017 Figure 8: Information Feedback Loop Decision making Data Data and policy making collection Annual Data utilization Schools School Validation Census & Analysis Publications and other outcomes Planning and policy actions & Design Central Research/analysis of Census Government, Reporting Local Government, & School Level Feedback Source: Adapted from Abdul-Hamid 2014. Implementing individual school reports and making basic school data available will empower stakeholders to utilize EMIS data. One way to ensure that education stakeholders can make use of EMIS data is through the provision of publicly available school report cards. The reports may cover information related to infrastructure needs, structure of education system, teachers, student performance, socio- economic background of the student body, enrollment rates, etc. The school report cards should be clear, accurate, timely and useful for schools and all education stakeholders. Good practices in the use of information at school and community level – through the use of school report cards, for example – are described in detail in Cheng and Moses (2016) and Read and Manuelyan Atinc (2017). Figure 9: Screenshot of a School Report Card Source: ACARA 2016. By ensuring that every school receives individual feedback, which (aside from highly sensitive data) is accessible to the general public facilitates comparison between different schools. For instance, when SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 28 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2017 parents are faced with the decision on where to place their child, they may utilize the school report cards to identify the best fit for their child. Not only would such a feedback loop improve data access and utilization, but it would also aide schools in the achievement of their planning and development goals. Many countries such as Australia have standardized school report cards by introducing a comparable index across the nation and made the reports publicly available, which enables parents to compare index scores nationwide (ACARA 2016). Figure 9 provides a screenshot of the school report found on Australia’s mySchool website, where basic data on each individual school is available. The information here is limited to basic data, but could be expanded to include more detailed indicators. Importantly, parents do not only compare schools on the basis of raw exam schools, but take into account the socioeconomic and other characteristics of the individual schools. Data dissemination can also empower school leaders and improve the management of schools. For example, in Argentina, schools are being given user-friendly results on learning assessments and trained on how to make plans for increasing student learning. In Mexico, school directors are being equipped with data dashboards and management training to help improve evidence-based decision making in formulating educational policies and practices at school level. Such efforts are crucial to enhancing the capacity of schools and local authorities in an environment of decentralization and increased local autonomy and accountability. Longitudinal tracking of students and teachers Figure 10: Example of a Longitudinal Tracking System enables policy makers to effectively utilize education data to improve the education system in the long term. Longitudinal tracking in combination with predictive analytical tools reap high benefits for long-term planning and education policy decisions. The tracking systems needs to be directly incorporated into the EMIS infrastructure and cover students throughout their entire academic career. Currently, there are unique identifiers available such as in ULMS, but they should be designed to be more targeted towards educational attainment. Of course, a functional registry of secondary education institutions (i.e., a school registry with unique and consistent school identifiers) is also needed to Source: Adapted from Florida Department of Education 2012. effectively track educational institutions over time. The data for individual students associated with each school can then be collected longitudinally and included into the EMIS. Florida, U.S.A., has implemented a successful longitudinal student tracking system from kindergarten until graduation from university, which may serve as a model for other systems (figure 10). In the long-term, teachers, parents, students and the government will benefit from individual student portals and be able to effectively utilize individual education data. Individual student portals are often designed to capture the student’s progress and provide any easy tool for teachers, parents and students to access study-related information. By comparing student records against their classmates, but also longitudinally, academic support can be given to students in a more targeted, efficient and effective manner. Principals and school administrators actively use data to evaluate teachers, monitor school progress, and manage school plans. Policy makers use data to monitor education quality and equity, improve accountability, and gauge effectiveness of policies and programs. If possible, such a portal should SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 29 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2017 be standardized across the country and capture data that is linked to the EMIS. This should go hand in hand with the implementation of unique student identifiers, so that policy makers can gain important insights by longitudinally tracking students. In order to do so, students (and teachers) should be assigned unique identifying numbers to support longitudinal tracking efforts. Box 1 describes an example of a successful implementation of universal student portals and their utilization in Estonia. In an advanced country such as Denmark, the personal identification number is used to collect any data from the moment of birth to be used afterwards in education, training and employment. With individual data, countries need to balance the benefits from collecting such information to inform public policies with the risks associated with security in data exchanges. Box 1: Example of a Standardized Student Portal System Estonia has established a strongly digitalized economy, where public services are generally available online and the government has introduced a fully functional student portal. The eKool portal is the first informational education portal in the country and has more than 200,000 active users every day. It provides services to students, parents, teachers, schools, the government and research partners alike. It includes features such as (i.) grades, (ii.) homework assignments, (iii.) attendance/absences, (iv.) weekly reports, (v.) special communication tools between teachers, students and parents, (vi.) study material collections, (vii.) lesson descriptions and timetables and (vii.) school management reports. To improve data collection efficiencies, individual education data is directly integrated with EMIS and eKool provides data such as (i.) statutory reports and statistics, (ii.) school enrolment and allocation, (iii.) problem solving and prevention and (iv.) easy data transfer to government repositories. It is online and accessible 24 hours a day, allowing for real time monitoring. Source: eKool 2016. Academic indexing can be integrated into individual student portals. Academic indexing Figure 11: Example of Academic Indexing allows for parents and teachers to identify students High Risk 12 students (3.5%) at risk of failing to complete a specific subject, class Medium Risk 35 students (10.3%) or grade. It is an effective tool to monitor Low Risk 286 students (84.1%) *Not Available 7 students (2.1%) educational progress. In Maryland, U.S.A., a “risk index” was developed, which identifies students at risk of failure. Figure 11 illustrates the color coding for students at low, medium and high risk. When parents and students have real-time access to student learning data, as they do in Maryland, then they are able to track their own progress and identify deficiencies in their performance early on. Just as importantly, teachers may use data to track Source: Provided to authors by Cecil County, progress toward Student Learning Objectives. By Maryland, 2014. tracking each student’s progress in a computerized system, it is easier to identify and carry out necessary intervention for students at risk. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 30 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2017 Acknowledgments This report was prepared by SABER-EMIS team members Husein Abdul-Hamid (Task Team Leader), Diana Mayrhofer and Namrata Saraogi in collaboration with the Project Team consisting of Igor Kheyfets, Lucia Casap, and consultant Iryna Kogut. The report benefited immensely from the guidance, support, and comments received from the Ministry of Education and Science and the Institute of Educational Analytics, as well as excellent peer review comments from Melissa Adelman and Alina Sava. The SABER-EMIS team is especially grateful for the many parents, teachers and principals who shared their time, feedback and insights. Special thanks go to World Bank staff providing support: Manar El-Iriqsousi and Cassia Miranda. Financial support for data collection was received from the UK Department for International Development (DFID), which is gratefully acknowledged. Acronyms ACARA Australian Curriculum, Assessment and Reporting Authority DFID Department for International Development (UK) DISO State Information System of Education EDEBO Unified State Electronic Education Database EMIS Education Management Information System GDP Gross Domestic Product ICT Information and Communication Technology IEA Institute of Educational Analytics IECM Institute of Education Content Modernization ISO International Organization for Standardization IT Information Technology MoES Ministry of Education and Science MoF Ministry of Finance NGO Non-governmental Organization PISA Programme for International Student Assessment SABER Systems Approach for Better Education Results SWOT Strengths Weaknesses Opportunities Threats TIMSS Trends in Mathematics and Science Study UCEQA Ukrainian Center for Educational Quality Assessment UIS UNESCO Institute for Statistics ULMS Ukrainian Longitudinal Monitoring Survey UNESCO United Nations Educational, Scientific and Cultural Organization SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 31 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2017 References Abdul-Hamid, Husein. 2014. “SABER EMIS Framework Paper.” World Bank, Washington, DC. Australian Curriculum, Assessment and Reporting Authority (ACARA). 2016. "MySchool." ACARA, Sydney. Accessed September 07, 2016. http://www.myschool.edu.au/. ___. 2015. “School Profile: Gordon East Public School, Gordon, NWS.” Accessed January 3, 2016. https://www.myschool.edu.au/SchoolProfile/Index/94042/GordonEastPublicSchool/41270/2015. Bruns, Barbara, Deon Filmer, and Harry D; Patrinos, H. 2011. “Making Schools Work: New Evidence on Accountability Reforms.” World Bank Group, Washington, DC. Cameron, Laurie, Kurt Moses, and John Gillies. 2006. “PAPERSchool Report Cards: Some Recent Experiences.” Working Paper. USAID & EQUIP2, Washington, DC. Check My School. 2016. “CMS Initiative.” Accessed December 22, 2016. http://www.checkmyschool.org/. Cheng, Xuejiao Joy, and Kurt Moses. 2016. “Promoting transparency through information: A global review of school report cards.” UNESCO / International Institute for Educational Planning, Paris. http://unesdoc.unesco.org/images/0024/002463/246358e.pdf. du Preez, Derek. 2012. “Department for Education Plans £31M Data Aggregation Project.” Computerworld UK. Accessed October 30, 2016. http://www.computerworlduk.com/news/public- sector/3370673/department-for-education-plans-31m-data-aggregation-project/. Accessed July 30, 2014. Education Statistics (EdStats). 2016. “World Bank Education Statistics – All Indicators.” Accessed December 28, 2016. World Bank, Washington DC. http://databank.worldbank.org/data/reports.aspx?source=Education%20Statistics&preview=off. eKool. 2016. “eKool”. Accessed January 4, 2017. https://www.ekool.ee/index_en.html Florida Department of Education. 2012. “Longitudinal Tracking”. Herczyński, Jan. 2016. “Draft requirements for DISO database development”. Institute of Educational Research, Warsaw, Poland. Institute of Education Content Modernization (ICEM). 2014. “Regulation 687: General Information”. Accessed January 3, 2017. http://www.imzo.gov.ua/pro-imzo/ Institute of Educational Analytics (IEA). 2015. “Regulation 444: Institute of Educational Analytics”. ___. 2017. “About the Institute”. Accessed April 17, 2017. http://iea.gov.ua/uk/pro-institut/. International Organization for Standardization (ISO). 2016. “ISO 9000—Quality management.” Webpage. ISO, Geneva. http://www.iso.org/iso/home/standards/management-standards/iso_9000.htm. Accessed December 20, 2016. Lehmann, Hartmut, Alexander Muravyev and Klaus F. Zimmermann. 2012. “The Ukrainian Longitudinal Monitoring Survey: Towards a Better Understanding of Labor Markets in Transition.” Institute for the Study of Labor (IZA), Bonn. Ministry of Education and Science of Ukraine (MoES). 2017. “The New Ukrainian School: Conceptual Principles of Secondary School Reform.” http://mon.gov.ua/%D0%9D%D0%BE%D0%B2%D0%B8%D0%BD%D0%B8%202017/02/17/book- final-eng-cs-upd-16.01.2017.pdf. Ramirez, Maria Jose. 2013. “Disseminating and Using Student Assessment Information in Chile.” Systems Approach for Better Education Results (SABER) student assessment working paper no. 3. World Bank, Washington, DC. http://documents.worldbank.org/curated/en/2012/01/ 16238811/disseminating- using-student-assessment-information-chile. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 32 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2017 Read, Lindsay, and Tamar Manuelyan Atinc. 2017. “Information for accountability: Transparency and citizen engagement for improved service delivery in education systems.” Global Economy & Development Working Paper No. 99. Brookings Institution, Washington, DC. https://www.brookings.edu/research/information-for-accountability-transparency-and-citizen- engagement-for-improved-service-delivery-in-education-systems/. State Statistical Service of Ukraine. 2017. “Education Data.” UNESCO Institute for Statistics. 2017. “Welcome to UIS Stats”. Accessed April 17th, 2017. http://data.uis.unesco.org/Index.aspx. World Bank. 2016a. “Ukraine Overview” Accessed January 2, 2017. http://www.worldbank.org/en/country/ukraine/overview. ___. 2016b. “Project Information Document (PID) Identification/Concept Stage - Strengthening Evidence- Based Policymaking with Education Statistics and Analysis - P161312.” Report Number PIDC86561. World Bank, Washington DC. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 33 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2016 Appendix A: Summary of Policy Lever Benchmarking Policy goal Policy lever Scorea Weight Benchmark Legal framework 1.75 17% Emerging Organizational structure and 3.11 17% Advanced institutionalized processes Enabling environment Human resources 2.89 17% Established Infrastructural capacity 2.45 17% Established Budget 2.33 17% Established Data-driven culture 1.74 15% Emerging Data architecture 2.40 20% Established Data coverage 0.75 30% Latent System soundness Data analytics 0.00 15% Latent Dynamic system 2.56 15% Established Serviceability 1.40 20% Emerging Methodological soundness 2.64 25% Established Accuracy and reliability 1.48 25% Emerging Quality data Integrity 1.34 25% Emerging Periodicity and timeliness 2.33 25% Established Openness 2.08 15% Established Operational use 0.59 50% Latent Utilization in decision making Accessibility 0.94 20% Latent Effectiveness in disseminating findings 1.20 15% Emerging a. 0–0.99 = Latent; 1–1.9 = Emerging; 2–2.9 = Established; 3–4 = Advanced. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 34 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2016 Appendix B: Extended Rubric, Ukraine Scores Highlighted in Red Scoring Description of Policy levers Indicators best practices Latent Emerging Established Advanced The system contains crucial components of a comprehensive The system enabling The system lacks The system contains contains basic The system contains environment, which major components of most components of components of a crucial components of POLICY AREA 1: ENABLING ENVIRONMENT addresses related a comprehensive a comprehensive comprehensive a comprehensive policy elements and enabling enabling enabling enabling environment enables the environment environment environment functioning of an effective and dynamic system Institutionalization of system: EMIS is institutionalized as an integral part of the education system and the government Responsibility: responsibility for collecting, processing, and disseminating education statistics is given to a clearly designated institution Basic components or agency An existing legal of a legal There is an existing Most elements of a Dynamic framework: the legal framework supports A legal framework is framework or legal framework to 1.1 Legal framework legal framework are framework is dynamic and elastic so a fully functioning not in place informal support a fully in place that it can adapt to advancements in EMIS mechanisms are in functioning EMIS technology place Data supply: the legal framework mandates that schools participate in EMIS by providing education data Comprehensive, quality data: the requirement for comprehensive, quality data is clearly specified in the EMIS legal framework SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 35 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2016 Scoring Description of Policy levers Indicators best practices Latent Emerging Established Advanced Data sharing and coordination: the legal framework allows for adequate data sharing and coordination between the Ministry of Education and agencies and/or institutions that require education data Utilization: the legal framework emphasizes data-driven education policy Budget: the education system budget includes a line item for EMIS Confidentiality: the legal framework guarantees that respondents’ data are confidential and used for the sole purpose of statistics The system is The institutional institutionalized The institutional The system is structure of the within the The system is not structure of the institutionalized system is defined government, has specified in policies, system is not within the Organizational within the well-defined and what exists does clearly specified in government, has well- structure and Organizational structure and government, it has 1.2 organizational not have well-defined policies, it has some defined organizational institutionalized institutionalized processes defined processes, and has organizational organizational processes, and has processes organizational several processes; EMIS has processes, and its several functionalities processes, but its functionalities limited functionalities functionalities are beyond statistical functionalities are beyond statistical limited reporting limited reporting Qualified staff Minimum standards Some staff are The majority of staff All staff are qualified operate the system, of qualification are qualified to operate are qualified to to operate the system, Personnel: the core tasks of EMIS are and opportunities not met for the the system, and operate the system, and well-established 1.3 Human resources identified and EMIS is staffed with are available to majority of staff that limited and frequent opportunities are qualified people improve their operate the system opportunities are opportunities are constantly available to performance and and opportunities are available to available to improve improve staff retention not available to improve staff SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 36 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2016 Scoring Description of Policy levers Indicators best practices Latent Emerging Established Advanced improve their performance and staff performance performance and performance and retention and retention retention Professional development: professional retention training is available for EMIS staff Data collection: tools for data collection are available Database(s): databases exist under the The system has a The system has a well- umbrella of the data warehouse and well-defined The system has an defined infrastructure have both hardware and software infrastructure to infrastructure that to fully perform its means perform data The system lacks a The system has a Infrastructural allows it to perform data collection, 1.4 Data management system: a system is collection, well-defined basic or incomplete capacity some of its functions management, and in place that manages data collection, management, and infrastructure infrastructure in an integral dissemination processing, and reporting dissemination manner functions in an functions in an Data dissemination: data dissemination integral manner integral manner tools are available and maintained by the agency producing education statistics Personnel and professional development: the EMIS budget contains a specific budget for EMIS personnel and their professional development The system budget contains the majority Maintenance: the EMIS budget The system budget is The system budget is of required contains a specific budget for system comprehensive, The system suffers The system has a comprehensive, categories to ensure 1.5 Budget maintenance and recurrent costs ensuring that the from serious basic or incomplete ensuring that the that most parts of system is sustainable budgetary issues budget system is sustainable the system are Reporting: the EMIS budget contains a and efficient and efficient sustainable and specific budget for reporting costs efficient Physical infrastructure: the EMIS budget contains a specific budget for physical infrastructure costs SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 37 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2016 Scoring Description of Policy levers Indicators best practices Latent Emerging Established Advanced Efficient use of resources: processes and procedures are in place to ensure that resources are used efficiently A data-driven culture A data-driven culture exists that prioritizes The system has a prioritizes data as a The system suffers data management and data-driven culture A data-driven culture fundamental because there is not a utilization within and that demonstrates exists that prioritizes element of data-driven culture beyond the education Data-driven a basic appreciation data management Data-driven culture operations and that prioritizes data system, and evidence Culture of data and interest and utilization within decision making, management and of that culture is in developing and beyond the both inside and data utilization in present in daily better data education system outside of the decision making interaction and utilization practices education system decision making at all levels The system has The system has The processes and The processes and basic processes some processes and structure of EMIS structure of the The system lacks and a structure a structure, but they are sound and system are sound and POLICY AREA 2: SYSTEM SOUNDNESS processes and that do not support do not fully support support the support the structure the components of the components of components of an components of an an integrated an integrated integrated system integrated system system system The system's data The system's data structure has most The data The system's data architecture elements of the data The data architecture architecture is well structure does not includes some architecture; is well defined to 2.1 Data architecture Data architecture defined to ensure have a well-defined components; however, it has some ensure full system full system data architecture however, it is deficiencies that functionality functionality incomplete affect the system's functionality The data in the Administrative data: EMIS contains system are administrative data The data in the comprehensive and The data in the The data in the The data in the system system are far from Financial data: EMIS contains financial cover administrative, system include system include most are comprehensive 2.2 Data coverage being comprehensive, data financial, human some of the data but not all of the and cover all data and coverage is resources, and areas data areas areas Human resources data: EMIS contains limited learning outcomes human resources data data SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 38 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2016 Scoring Description of Policy levers Indicators best practices Latent Emerging Established Advanced Learning outcomes data: EMIS contains learning outcomes data Basic tools and processes are Tools and processes available, but the Tools and processes Tools and processes are available to system is not Tools and processes are available; are available to perform data capable of 2.3 Data analytics Data analytics are used to perform however, data perform data analytics analytics at different conducting limited tabulations analytics are not at different levels on a levels on a regular advanced analytical performed regularly regular basis basis steps (e.g., predictive models, projections) Quality assurance measures: the system is dynamic and maintains The system in place The system in place is quality assurance measures The system in place is not easily The system in place is not easily adaptable is elastic and easily adaptable and The system in place elastic and easily Data requirements and considerations: to changes adaptable to allow requires significant is easily adaptable, adaptable to allow for 2.4 Dynamic system mechanisms exist for addressing new /advancements in for changes time and resources but it remains changes/ and emerging data requirements data needs, as no /advancements in to accommodate reasonably complex advancements in data System adaptability: EMIS is elastic and quality assurance data needs changes and/or needs easily adaptable to allow for changes standards are used advancements and/or advancements in data needs Validity across data sources: Services provided by Services provided by information brought together from the system are valid the system are valid different data and/or statistical across data sources, across data sources, frameworks in EMIS is placed within integrate non- integrate non- the data warehouse using structural education databases education databases and consistency measures into EMIS, and Serious issues exist Inconsistencies The data are into EMIS, and archive Integration of non-education databases archive data at the related to data exist related to consistent and valid; data at the service of 2.5 Serviceability into EMIS: data from sources collected service of EMIS validity and data validity and however, some EMIS clients by by agencies outside EMIS are clients by ensuring consistency consistency concerns still exist ensuring the integrated into the EMIS data the relevance, relevance, warehouse consistency, consistency, Archiving data: multiple years of data usefulness, and usefulness, and are archived, including source data, timeliness of its timeliness of its metadata, and statistical results statistics statistics SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 39 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2016 Scoring Description of Policy levers Indicators best practices Latent Emerging Established Advanced Services to EMIS clients: services provided by the system to EMIS clients include ensuring the relevance, consistency, usefulness, and timeliness of its statistics The system has most mechanisms in place The system has the needed to collect, The system has the mechanisms The system has save, and produce required mechanisms required to collect, basic mechanisms timely, high-quality in place to collect, The system lacks save, produce, and to collect, save, information for use save, produce, and mechanisms to utilize information, and produce in decision making; utilize information, collect, save, or POLICY AREA 3: QUALITY DATA which ensures timely, quality however, some which ensures produce timely, high- accuracy, security, information; additional measures accuracy, security, quality information and timely, high- however, its are needed to and timely, high- for decision making quality information accuracy might be ensure accuracy, quality information for use in decision questionable security, and/ or for use in decision making timely information making that can be used for decision making Concepts and definitions: data fields, records, concepts, indicators, and metadata are defined and documented The methodological in official operations manuals along The methodological The methodological The methodological basis for producing The methodological with other national datasets and basis for producing basis for producing basis for producing educational basis for producing endorsed by the government educational statistics educational statistics educational statistics statistics follows educational statistics from raw data follows most Methodological Classification: defined education does not follow the basics of from raw data follows 3.1 follows required soundness system classifications are based on internationally internationally internationally internationally internationally technical guidelines and manuals accepted standards, accepted accepted standards, accepted standards, accepted standards, Scope: the scope of education statistics guidelines, or good standards, guidelines, and good guidelines, and good guidelines, and good is broader than and not limited to a practices guidelines, and practices practices practices small number of indicators (e.g., good practices measurements of enrollment, class size, and completion) SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 40 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2016 Scoring Description of Policy levers Indicators best practices Latent Emerging Established Advanced Basis for recording: data-recording systems follow internationally accepted standards, guidelines, and good practices Source data: available source data provide an adequate basis for compiling statistics Source data and Source data and Source data and statistical statistical techniques Source data and Validation of source data: source data statistical techniques Source data and techniques have follow most required statistical techniques are consistent with the definition, are sound and Accuracy and statistical techniques basic soundness elements to be are sound and 3.2 scope, and classification as well as time reliable, and reliability lack soundness and and reliability, but sound and reliable, reliable, and statistical of recording, reference periods, and statistical outputs reliability statistical outputs but statistical outputs sufficiently valuation of education statistics sufficiently portray do not portray outputs do not portray reality Statistical techniques: statistical reality reality portray reality techniques are used to calculate accurate rates and derived indicators Professionalism: EMIS staff exercise their profession with technical Education statistics Education statistics independence and without outside contained within Education statistics contained within the interference that could result in the the system are contained within the Education statistics system are mostly violation of the public trust in EMIS Education statistics guided by limited system are guided by contained within the guided by principles statistics and EMIS itself contained within the principles of all three principles of 3.3 Integrity system are not of integrity (two of system are guided by integrity (one of the integrity: Transparency: statistical policies and guided by principles the three principles principles of integrity three principles of professionalism, practices are transparent of integrity of professionalism, professionalism, transparency, and Ethical standards: policies and practices transparency, and transparency, and ethical standards in education statistics are guided by ethical standards) ethical standards) ethical standards Periodicity: the production of reports and other outputs from the data The system produces The system The system produces The system produces The system produces warehouse occur in accordance with data and statistics produces some most data and Periodicity and data and statistics all data and statistics 3.4 cycles in the education system neither periodically data and statistics statistics periodically timeliness periodically in a periodically and in a Timeliness: final statistics and financial nor in a timely periodically and in a and in a timely timely manner timely manner statistics are both disseminated in a manner timely manner manner timely manner SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 41 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2016 Scoring Description of Policy levers Indicators best practices Latent Emerging Established Advanced The system is wholly There are no signs The system is used The system is used The system is wholly utilized by different that EMIS is utilized by some education by most education utilized by different users for decision in decision making by stakeholders, but stakeholders but is users for decision POLICY AREA 4: UTILIZATION FOR DECISION MAKING making at different the majority of not for major not fully operational making at different levels of the education policy decision in governmental levels of the education system stakeholders making decision making education system EMIS stakeholders: EMIS primary stakeholders are identified and use the system in accordance with the legal The system is open The system is open The system lacks The system is open framework to the majority of The system is open to to education openness to to some education education all education User awareness: current and potential stakeholders in education stakeholders in stakeholders in stakeholders in terms 4.1 Openness EMIS users are aware of EMIS and its terms of their stakeholders in terms terms of their terms of their of their awareness outputs awareness and of their awareness awareness and awareness and and capacity to utilize User capacity: EMIS users have the capacity to utilize and capacity to utilize capacity to utilize capacity to utilize the the system skills to interpret, manipulate, and the system the system the system system utilize the data produced by the system to ultimately disseminate findings Utilization in evaluation: data produced by EMIS are used to assess the education system Utilization in governance: data produced by EMIS are used for governance purposes Data produced by Data produced by Data produced by the Data produced by Data produced by the Utilization by schools: data produced the system are used the system are used system are not used the system are system are used in by EMIS are used by schools in practice by the 4.2 Operational use in practice by the in practice by used in practice by practice by the main majority of Utilization by clients: data produced by main education education some education education education EMIS are used by clients (including stakeholders stakeholders stakeholders stakeholders stakeholders parents, communities, and other actors) Utilization by government: the system is able to produce summative indicators (derived variables) to monitor education system Understandable data: data are Education statistics The system suffers The system has The system has Education statistics 4.3 Accessibility presented in an easily digestible are presented in an from serious major accessibility minor accessibility are presented in an manner understandable accessibility issues issues issues understandable SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 42 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2016 Scoring Description of Policy levers Indicators best practices Latent Emerging Established Advanced Widely disseminated data: education manner and are manner and are statistics are disseminated beyond the widely disseminated widely disseminated Ministry of Education and/or the using clear platforms using a clear platform education statistics–producing agency for utilization, for utilization, to other EMIS stakeholders complemented by complemented by user support user support Platforms for utilization: platforms are standardized across EMIS and are customizable to user needs User support: assistance is provided to EMIS users upon request to help them access the data Dissemination strategy: national A dissemination plan governments have an information has been dissemination strategy in place implemented; Dissemination of Dissemination is The dissemination of Effectiveness in Dissemination is however, room education statistics reasonably education statistics via 4.4 disseminating neither strategic nor exists for Dissemination effectiveness: via EMIS is strategic strategic, but EMIS is strategic and findings effective improvement (for dissemination of EMIS statistics is and effective ineffective effective full effectiveness in effective relation to strategic engagement) SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 43 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2016 Appendix C: Structure of the Ukrainian Education System Source: World Bank (forthcoming). “Ukraine: Public Finance Review 2017.” SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 44 UKRAINE ǀ EMIS SABER COUNTRY REPORT |2016 www.worldbank.org/education/saber The Systems Approach for Better Education Results (SABER) initiative collects data on the policies and institutions of education systems around the world and benchmarks them against practices associated with student learning. SABER aims to give all parties with a stake in educational results—from students, administrators, teachers, and parents to policy makers and business people—an accessible, detailed, objective snapshot of how well the policies of their country's education system are oriented toward ensuring that all children and youth learn. This report focuses specifically on policies in the area of Education Management Information Systems. This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 45