100064 Papua New Guinea SABER Country Report EDUCATION MANAGEMENT INFORMATION SYSTEMS 2015 Key Policy Areas Status 1. Enabling Environment Papua New Guinea successfully established an Education Management Information System (EMIS) as the point of reference system that collects, processes, and disseminates education data on a regular basis. The National Plan for Education 2005 – 2014 set a roadmap for timely collection, management, and utilization of data in decision making. The plan set the foundation for EMIS; however, it did not institute comprehensive EMIS policies or an EMIS budget. Policies do not outline procedures to ensure student data confidentiality, nor do they define processes and procedures for sharing data with other government units. Policies do not require that information be reported back to local levels, especially schools. 2. System Soundness Using an Oracle platform, EMIS captures demographic data and some financial, infrastructure, and human resources data; however, assessment, salary, and noneducation data are not integrated. Analysis was conducted using Oracle Discoverer, but the tool is being discontinued by the vendor and rolled into the robust Oracle Business Intelligence (BI) tool. Data are sourced annually through a paper-based census, which moves from schools to district offices and on to provincial offices, before going to the z Department of Education to be entered into the system. A pilot program introduced EMIS in six provinces, giving them the ability to input and access data through a digital interface. Internal and external audits are not conducted regularly. 3. Quality Data EMIS concepts and definitions (data fields, indicators, metadata, etc.) follow a functional manual documented and approved by the government. Source data from most regions are reliable, although reporting and accuracy decline in rural locations. Systematic validation of data is conducted manually at provincial and district levels. Validation through automated systems started this year. EMIS is accessible through the Department of Education website through a dashboard, which also links EMIS data to the national education strategy. Statistics have been disseminated within 6–12 months after the start of the next school year; however, circulation of the 2015 census was delayed and may impact the timing of 2016 reports. 4. Utilization for Decision Making EMIS data are disseminated through an annual statistics book and the online dashboard. Lack of internet access and limited communication and training prevent most stakeholders outside of the national government from accessing and using EMIS. Schools are not aware of the dashboard and instead rely on their own internal methods and processes for using data. PAPUA NEW GUINEA ǀ EMIS SABER COUNTRY REPORT |2015 Introduction The SABER-EMIS assessment methodology is built on four key policy areas that are essential to EMIS and In 2011, the World Bank Group commenced a must be assessed to understand and ultimately multiyear program designed to support countries in strengthen the system. Each policy area is defined by a systematically examining and strengthening the set of policy levers (actions that help governments performance of their education systems. Part of the reach the policy area) and indicators (measuring the World Bank’s new Education Sector Strategy, the extent to which the policy levers are achieved) (Figure evidence-based initiative called SABER (Systems 1). Approach for Better Education Results), uses diagnostic Figure 1: SABER-EMIS Policy Areas and Levers tools for examining education systems and their Policy Areas Policy Levers: legal framework, organizational structure component policy domains against global standards and Enabling and institutionalized processes, human resources, best practices and in comparison with the policies and Environment infrastructural capacity, budget, data-driven culture practices of countries around the world. By leveraging System Policy Levers: data architecture, data coverage, data this global knowledge, the SABER tools fill a gap in the Soundness analytics, dynamic system, serviceability availability of data and evidence on what matters most Policy Levers: methodological soundness, accuracy and to improve the quality of education and achievement of Quality Data reliability, integrity, periodicity and timeliness better results. Utilization for Policy Levers: openness to EMIS users, operational use, Decision Making accessibility, effectiveness in disseminating findings This report discusses the results of applying the SABER– Source: Abdul-Hamid 2014. Education Management Information Systems (EMIS) tool in Papua New Guinea (PNG). The objectives of this report are to examine the system according to key A strong enabling environment lays the foundation for policy areas, identify successes and challenges in the an effective EMIS. Enabling environment refers to the system, and provide recommendations to support the laws, policies, structure, resources, and culture continued advancement of EMIS in PNG. surrounding an EMIS that make data collection, management, and access possible. In essence, this Overview of SABER-EMIS policy area is the context in which an EMIS exists. This defined scope of an enabling environment builds on Information is a key ingredient in an effective lessons learned from studies of education management education system. SABER–EMIS aims to help countries systems. improve data collection, data and system management, and data use in decision making. SABER- System soundness ensures key processes, structures, EMIS assesses the effectiveness of a country’s EMIS, and integration capabilities in an effective EMIS. with the aim of informing policy dialogue and helping Education data are sourced from different institutions, countries better manage education inputs and but all data feed into and make up EMIS. Databases processes to achieve overall efficiency and strong within an EMIS are not viewed as separate databases, learning outcomes. but as part of the whole EMIS. Key aspects of system soundness include what data are covered in EMIS and A successful EMIS is credible and operational in how they come together in the overarching system. planning and policy dialogue as well as teaching and learning. It produces and monitors education statistics Quality data establish the mechanisms required to within an education system and has a multifaceted collect, save, produce, and utilize information in an structure, comprising the technological and institutional accurate, secure, and timely manner. Data quality is a arrangements for collecting, processing, and multidimensional concept that encompasses more than disseminating data (Abdul-Hamid 2014). It is crucial for just the underlying accuracy of the statistics produced. tracking changes, ensuring data quality and timely It means that not only are the data accurate, but that reporting of information, and facilitating the utilization the data address specific needs in a timely fashion. of information in decision making. Quality data lay the groundwork for utilization. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 2 PAPUA NEW GUINEA ǀ EMIS SABER COUNTRY REPORT |2015 An effective EMIS is utilized in decision making by all Implementation can be observed through utilization of users (parents, students, teachers, principals, and EMIS by stakeholders, budget allocation, distribution of policy makers) across the education system. An EMIS human resources, availability of professional needs to be used so that measures can be taken to development activities, communication and improve educational quality. Accurate information on dissemination of information, as well as the extent of education sector performance enables the design of institutionalization across the system. Once policy more informed policies and programs. It is imperative intent and implementation are analyzed, the EMIS to understand where decision making occurs, if the assessment explores the results of these two key capacity to analyze and interpret education data exists, components, with a focus on system effectiveness and and if specific data are available to inform decisions. efficiency, in addition to strong outcomes in the areas of teaching and learning as well as management and Using the EMIS data collection instrument, policy levers planning. Strong education systems will ultimately use are scored on a four-level scale (latent, emerging, these outcomes to inform the effectiveness of policies established, and advanced) to assess the extent to and education strategies and make adjustments as which both policy intent and implementation are necessary, creating the cyclical process illustrated in Figure 3. Figure 2: SABER Scoring and EMIS Development 4 In PNG, EMIS intent and implementation were assessed Advanced through desk research and analysis of system 3 Comprehensive Established enabling applications and utilization, as well as interviews with a 2 Enabling environment, variety of stakeholders (Table 1). Emerging environment, processes, 1 processes, structure, data Basic enabling management, Latent environment, structure, data management, utilization, and Limited enabling processes, structure, data utilization in place integration in environment, with some place, with Figure 3: Policy Intent, Implementation and Outcomes Cycle, processes, management, intelligent utilization integration with Examples structure, data analytics management, utilization 9 Policies 9 Utilization 9 Vision and buy-in 9 Processes Source: Abdul-Hamid 2014. 9 Standards 9 Institutionalization 9 Strategy 9 Budget achieved (Figure 2). documents 9 Human resources 9 Communication Approach Intent Implementation Intent and Implementation The EMIS assessment examines policy intent and the degree to which intended policies are effectively implemented on the ground (Figure 3). Intent refers to the way in which EMIS and its overarching purpose are Outcomes articulated by decision makers and documented in policies and legislation, as well as standards and 9 Teaching and learning strategy documents. Assessing intent alone reveals only 9 Management part of the picture. 9 Planning and monitoring 9 Transparency 9 Governance As such, this EMIS assessment also evaluates policy execution. Implementation refers to the degree to Source: Authors. which policy intentions take place during the day-to-day activities of stakeholders (policy makers, county administrators, principals, teachers, and students). SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 3 PAPUA NEW GUINEA ǀ EMIS SABER COUNTRY REPORT |2015 Table 1: Measuring EMIS Intent and Implementation in PNG Policy intent Implementation Country Overview x Multiple meetings with x Interviews with PNG is among the world’s most culturally diverse DoE stakeholders at national, countries, home to more than 200 different cultures x Extensive review of district, and school levels and more than 860 different spoken languages. PNG has relevant policies, national x Analysis of data quality a population of 7.3 million (2013), with an estimated 40 strategies, standards, and and comprehensiveness percent under 15 years of age. planning documents x Thorough utilization assessment x Examination of Rainforest covers 75 percent of the land, and about 15 professional development percent of the country is spread across 600 islands. activities Immense mountain ranges stretch over much of the Source: Authors. mainland. More than 80 percent of the population live in rural areas and work in subsistence activities such as Methodology smallholder farming and fishing. Gross national income The EMIS assessment methodology consists of a review per capita in 2013 was $2,020, and gross domestic of written policies and technical documents as well as product growth was 5.5 percent. interviews with key stakeholders across the education system to ensure proper implementation. PNG consists of four regions, 20 integrated provinces, the autonomous province of North Solomons Research and investigation for the PNG EMIS (Bougainville), and the National Capital District (NCD). In assessment took place between March and May 2015. 1978 the constitutional parliamentary democracy The authors conducted a comprehensive review of established a provincial government system, leading to policies, as well as technical documents and other a highly decentralized education system. As such, each background materials. To further examine policy intent province has its own education plan, with different and implementation, a series of interviews and procedures for collecting and processing data. meetings took place with the following entities: The large rural population coupled with rugged terrain 1. National Department of Education (Statistics and weak infrastructure pose unique challenges to the and EMIS Unit, Information and Communication provision of quality education and the flow of Technologies [ICT] Division, Assessment Unit, information across the country. Some regions are and other education divisions) inaccessible by road. Internet saturation is low, and 2. Provincial Office representative fixed broadband penetration is below 1 percent of the 3. Sample of schools population. Various donors are supporting the government to help provide broadband Internet network and services to rural communities. In 2011, 26 percent of people in PNG had access to mobile phones, although that number has grown significantly over the past four years as a result of donor projects and greater competition among telecom providers. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 4 PAPUA NEW GUINEA ǀ EMIS SABER COUNTRY REPORT |2015 Education Overview commitments to access and quality, as well as efficient management and financing of education. Additionally, The education system in PNG has three governance the Universal Basic Education Plan 2010–2019 (DoE levels: national, provincial, and local (districts and 2010) aims to ensure that school-aged children have schools). At the national level, the Department of equal access to quality basic education and reach Education (DoE) is responsible for setting and required standards in both literacy and numeracy. implementing national policies; establishing, preserving, and improving standards of education; and ensuring In 2012, as part of the Universal Basic Education Plan, broad access to quality education. Provincial and local- the Tuition Fee Free (TFF) policy was introduced, level authorities are responsible for planning, financing, abolishing school fees and leading to a surge in staffing, and maintaining general education institutions enrollment. High enrollment numbers put considerable up to grade 12, including preschool, elementary, stress on the system, evidenced by high pupil-teacher primary, secondary, and vocational schools (DoE, PNG ratios, especially at the elementary level (Table 2), as 2004). well as textbook shortages and gaps in teacher qualifications (EFA 2015). The education cycles include elementary school (preschool to second grade), primary (third to eighth Table 2: Education Indicators, at a Glance grade), secondary (ninth to twelfth grade), and tertiary, Schools and enrollment technical, and vocational pathways (Figure 4). Flexible, Institution Schoolsa Enrollment Open, and Distance Education (FODE) is also available Elementary (preprimary) 7,017 773,807 for students who cannot attend school institutions but Primary 3,561 942,998 are able to complete diploma and certificate Secondary 212 140,123 equivalency programs independently. FODE data are Vocational 117 31,546 included in the annual Education Statistical Bulletin. Flexible Open Distance 45 11,804 Education (FODE) Several key documents serve as strategic roadmaps that Net enrollment rates guide the country’s education system and lay the Male Female Total groundwork for creation of key policies. Achieving a Elementary 63% 61% 62% Better Future: A National Plan for Education 2005–2014 (DoE 2004) and the Medium Term Development Primary 50 44 47 Strategy, 2005–2010 (Department of National Planning Secondary 9 6 7 and Rural Development 2004) reflect national Pupil-teacher ratio Elementary 49:1 Figure 4: Education Structure Primary 35:1 Secondary 31:1 Source: DoE 2013. a. Schools include government, church, and other agency schools. Source: DoE 2004. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 5 PAPUA NEW GUINEA ǀ EMIS SABER COUNTRY REPORT |2015 PNG EMIS Results needs, up to date with current technological products and standards, and deployed efficiently. This section presents the main results of EMIS diagnostics described in the previous sections. Results With the goal of providing all school-age children with and scores for each policy area are presented, along access to quality basic education, the Universal Basic with supporting evidence. Education Plan 2010–2019 (UBE 2010–19) gave way to the TFF policy, which had significant implications for Policy Area 1: Enabling Environment EMIS. TFF relieved households from the burden of school fees and positioned the public financing system Emerging zz|| to cover costs up to grade 12. The TFF application PNG’s enabling environment was assessed in the calculation runs in the EMIS database, using data following areas: (1) legal framework, (2) organizational collected through the national school census. Not only structure and institutionalized processes, (3) human do schools need to submit data through the census to resources, (4) infrastructural capacity, (5) budget, and receive TFF funds, but the data they submit classify (6) data-driven culture. them in a certain funding bracket. This incentive led to an increase in reporting of some school data. Based on Three core policies guide and influence EMIS efforts in interviews conducted during SABER-EMIS data PNG: (1) Achieving a Better Future: A National Plan for collection (2015), the challenge has been that some Education 2005–2014 (NEP 2005–14), (2) the ICT Policy, schools now exaggerate enrollment numbers in an and (3) the Universal Basic Education Plan 2010–2019 attempt to reach a higher bracket of TFF funding. For and resulting TFF policy. NEP 2005–14 presents the this reason, sound validation processes at the District vision, strategy, and roadmap for the country’s Education Office (DEO) and the Provincial Education education system. It builds off of the previous 10-year Office (PEO) levels, as well as automated tools to flag plan and aligns with other strategic development potentially inaccurate data, are of critical importance. documents nationally and internationally (e.g., Education for All, Millennium Development Goals). UBE and TFF successfully provided more students with National education targets set forth in the Plan are access to education; however, the surge in enrollment tracked in EMIS. numbers has put tremendous pressure on the education system as a whole and specifically on the NEP 2005–14 strengthened EMIS infrastructure, ability of schools to provide quality education. EMIS is processes, standards, and practices. It called for a not currently equipped to track national quality centralized system for collection and storage of data to indicators, but given the challenges in the provision of be in place by 2006 and set new requirements for data quality education, greater incentive may exist to collection and management; for example, a new census position EMIS to integrate learning outcomes data. was established. Additionally, the Plan set goals for data utilization in planning and decision making, as well as The importance of data utilization is articulated in dissemination of findings. The next five-year education plans and policies; however, this commitment to data- plan is close to finalization and will soon be announced. driven decision making is not as strong in practice. Decision makers tend to use EMIS for tracking and NEP mandated an ICT policy and plan, which further reporting, but few signs indicate that EMIS is actively helped to establish EMIS by defining key technical being used as a tool for planning. Additionally, EMIS areas. Under the ICT Policy, technical guidelines for the training is demand-driven, and a very limited number of system are addressed in areas such as software requests have been made in the area of data utilization procurement and development, copyright compliance, for planning. maintenance and service contracting, inventory management, licensing, security and disaster recovery, Processes to share data with other government and compatibility and integration. Additionally, the ICT departments are neither clearly defined nor detailed in Policy calls for technologies to be aligned with business policies. For example, DoE and the Department of Health share data frequently, but no policy is in place to SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 6 PAPUA NEW GUINEA ǀ EMIS SABER COUNTRY REPORT |2015 streamline or even automate these transactions. There is assigned to support EMIS; however, he or she also is potential here to increase efficiency and drive better holds another position and responsibilities. utilization of data. Policy Area 2: System Soundness Currently the government of PNG (GoPNG) does not have a policy that calls for an EMIS budget line; instead Emerging zz|| the ICT and the Policy, Planning, and Research Division PNG’s EMIS soundness was assessed in five critical budgets support EMIS staff and resources. Additionally, areas: (1) data architecture, (2) data coverage, (3) data most of the financing for EMIS is supported through analytics, (4) dynamic system, and (5) serviceability. donor-financed operations supported by the government. This makes strategic planning for EMIS The infrastructure supporting EMIS in PNG has difficult because funding timelines and amounts are improved dramatically over the last decade. EMIS inconsistent and unpredictable. Core EMIS staff and launched in 2004–2005 using a combination of seasonal staff are covered under the larger DoE budget, Microsoft Excel and Access systems. In 2005–2006, DoE while the ICT budget covers EMIS maintenance, identified Oracle as a best-in-class provider and reporting, web-platform updates, and physical transitioned the entire department to Oracle infrastructure. Without an explicit EMIS budget, the Application Express, a web-based software EMIS Unit lacks the autonomy and planning capabilities development environment that runs on an Oracle to strategically advance EMIS to the next level. database. All data were effectively migrated onto the new system. The department hosts two physical servers Additional policy gaps were identified in the area of (one for redundancy) and two storage devices on site. confidentiality. Policies do not currently mandate that Each physical server has 27 virtual servers with 6 respondents’ data be kept secure, nor do they define terabytes of storage. Data center speed is 200 megabits processes and procedures to keep data secure. Further, per second (Mbps), with a secondary link on 100 Mbps. no laws provide a guide to student or parent/guardian rights in accessing student data. The continued growth of EMIS data architecture, and functionality in general, is very much dependent on Policies that guide the flow of information back to funding. The EMIS Unit and the ICT Division are strong schools do not exist. Feedback loops create an partners, with the ICT team responding quickly to EMIS information cycle that brings EMIS analysis back to the Unit requests. The EMIS Unit is adept at running the school level and can ultimately improve response rates system with limited resources but could do much more and accuracy of data. Schools are eager to use data but with additional resources and the support of an are unable to access it. As a result, many schools use overarching EMIS Policy with an EMIS budget. their own internal systems for data processing and utilization. Budget pressures limit the extent to which the EMIS Unit can engage its external vendor and make At the national level, the EMIS Unit consists of 14 staff necessary upgrades to the Oracle system. Supporting members and resides under the Policy, Planning, and the EMIS Unit is an external firm, Datec, which was Research Division of DoE. Additionally, the ICT identified through a tender process and due diligence. Department, consisting of 10 staff including four Datec has been a supportive external firm, with the only database experts, supports the EMIS team. At the limitation being the lack of funding to pursue further national level, the EMIS team has an organizational EMIS development. Additionally, the EMIS Unit structure with specific roles, responsibilities, and a previously used Oracle Discoverer for querying, structured workflow. Further, work processes are reporting, data analysis, and web publishing, but Oracle periodically reviewed to maintain and improve is no longer supporting Discoverer and is in the process efficiency. At the provincial level, skill levels as well as of transitioning clients to Business Intelligence (BI). The organizational structure and institutional processes only challenge with an Oracle system is that it is quite vary. The lack of staff to support EMIS poses significant expensive; thus it is critical that the EMIS Unit, the challenges. For example, in some provinces, one person SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 7 PAPUA NEW GUINEA ǀ EMIS SABER COUNTRY REPORT |2015 Policy and Planning Division, and DoE ensure that the population. The challenge is that with low internet necessary financing is in place to maintain the system. saturation and limited communication materials about the Dashboard, it is not frequently used. Many people PNG’s EMIS data are secure. The system utilizes a large cannot access it or are not aware that it exists. database that holds national school census data in addition to other DoE applications such as the TFF application for school subsidies and the quarterly Box 1: Good Practice in Provincial EMIS Access: The Case return application for school-level monthly reports of of Milne Bay student attendance. No security violations have occurred in recent years. Oracle tracks audit threats, One province that has been especially successful on-site servers are backed up daily, and an offsite with the pilot initiative is Milne Bay, a maritime backup takes place weekly. province consisting of four districts and serving approximately 70,000 students. Part of Milne Bay’s PNG is currently piloting a program that provides EMIS success rests in linking EMIS with provincial access in six provinces (Box 1). Apart from those strategic planning, which targets three areas: (1) provinces, education data are entirely in paper format Access, (2) Quality, and (3) Management. Milne Bay from schools until they reach DoE at the national level. also developed both district-level and provincial- A good practice in EMIS implementation is the level processes (including formal checklists) for reduction of manual validation and transfer of data by census review, validation, protocols for making inputting data directly into EMIS through a computer- changes to existing data, and data entry. The result or tablet-based system as early in the data collection is a 1 percent nonresponse rate for completion of process as possible. The pilot project enables provinces census forms by schools. The province is also eager to input data into EMIS, access real-time data from their to test EMIS at the district level by putting province, and run their own reports. DoE provides the computers in district offices with direct access to hardware, software, and training necessary to access EMIS. and use the system. Provinces use their own budget to cover internet and service provider bills that are roughly Milne Bay’s success can also be attributed to the K 40,000 ($14,700) per year. The districts that are most following factors: (1) a dedicated EMIS staff, (2) successful with the rollout are those that have at least buy-in from the highest level of the PEO, (3) one dedicated EMIS staff member. For some provinces, strategic vision, (4) operational focus on provider payments go through DoE, although, as one transparency, good governance, utilization, and province found, it is more efficient for the province to sustainability, and (5) established data management pay the fee directly to the provider so as to avoid processes (e.g., validation, revision, entry). EMIS bottlenecks or run the risk of lapsed payments, which data are regularly communicated to members of could lead to problems such as power outages. the provincial parliament and the provincial assembly, with some decision makers requesting The continued rollout of this initiative, including data directly from the provincial EMIS team. EMIS dedicated financing to ensure its sustainability and data are currently reporting transitional rates and ongoing training for provinces, carries the potential to infrastructure needs to guide planning decisions for significantly strengthen EMIS in PNG. Provinces construction of new schools and to fill access and participating in the pilot should also be encouraged to quality gaps. share information with districts and schools and train Milne Bay EMIS Workflow these local stakeholders on data utilization. EMIS is available to the public via the DoE website (education.gov.pg), through the EducationInfo DevInfo Dashboard. The dashboard is designed to promote fact- based decision making based on reliable data and report education performance to the general SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 8 PAPUA NEW GUINEA ǀ EMIS SABER COUNTRY REPORT |2015 Data analytics are not automated in the current EMIS; follow logic, have fixed methodology, and have a well- instead, the EMIS Unit extracts data and runs analysis defined purpose (Abdul-Hamid 2014). Table 3 compares in Oracle Discoverer (soon to be Oracle BI). That said, data coverage best practices with data coverage in PNG. descriptive analytics and data tabulations are frequently run. Planning analysis such as projections, predictive Table 3: Data Coverage, Best Practice and PNG models, and scenario analysis is not taking place. Type Best practice PNG x Demographic x Demographic EMIS data coverage currently includes administrative x Health x Some Admin. data as well as some financial and human resources x Attendance (enrollment, learning data data. The census collects data from state-run schools repeat, dropout, etc.) x Some school and schools managed by church agencies such as the x School level Anglicans, Roman Catholics, Lutherans, and United and x Budgets and revenues x Some Financial x Spending budgets and Evangelical Alliance. It captures school details (e.g., data x Cash transfers, subsidies revenues name, location, registration), enrollment and x Unit cost per student graduation data, financial data (e.g., sources of funding, x General demographics x Some general bank account information), infrastructure data, teacher Human x Salaries demographics information (e.g., registration number, year started, resources x Performance evaluation qualification), and additional information such as data x Professional existence of a Board of Management and information development on instructional tools such as textbooks. Extended data x Classroom assessments on school finances, such as school spending and salary Learning x National assessments outcomes information, are not included. Lack of learning x International data outcomes data creates a significant gap in the extent to assessments which EMIS can inform and support provision of quality Source: Adapted from Abdul-Hamid 2014. education. Currently assessment data sit on a Microsoft Access Policy Area 3: Quality Data database, effectively siloed from outside data. In 2004 the Assessment Unit (formerly the Measurement Established zzz| Services Branch) launched FoxPro as the primary data The quality of data captured by PNG’s EMIS was management system. In 2008 an unsuccessful effort assessed in four areas: (1) methodological soundness, was made to integrate Oracle and FoxPro so that (2) accuracy and reliability, (3) integrity, and (4) assessment data could link with other DoE databases. periodicity and timeliness. Dialogue between the Assessment Unit and the EMIS Unit is strongly encouraged as well as efforts to explore During the design of the current EMIS, DoE developed integration of data. an EMIS Operations Manual, complete with metadata, data fields, and indicators that mapped back to NEP GoPNG uses a number of systems for financial 2005–14. Metadata were defined such that they would management and payroll; however, they are not integrate with the GoPNG national system, PNGInfo. integrated with EMIS. These systems include the This level of sophistication gave EMIS a strong Integrated Financial Management System, the payroll foundation for good data. To build upon that management system, and the Provincial Government foundation, the EMIS Operations Manual should be Accounting System. reviewed and updated and reincorporated into routine use. A comprehensive EMIS should include not only administrative data but also financial, human The primary mechanism for EMIS data collection is the resources, and learning data (Table 3). This information national school census, which gained greater response should be available at both the individual and aggregate rates after being linked with the TFF education level. The type of data entered into the system needs to subsidy. The census collects school details such as SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 9 PAPUA NEW GUINEA ǀ EMIS SABER COUNTRY REPORT |2015 finances, enrollment, progression, repeaters, responsible for validating, endorsing, and transmitting information on vulnerable children and children with data. special needs, school infrastructure information, teacher data, and some additional information such as In 2015 EMIS also started using automated processes to whether there is a Board of Management. With the validate data and flag discrepancies. Both manual and exception of the provinces participating in the pilot automated validation processes are essential and program, all provinces send census data in paper format should continue to be strengthened and reinforced. to the national government to be input into EMIS. Each With the linking of census data and the TFF subsidy, year, TFF has a National Education Minimum School Fee some schools have inflated enrollment numbers to rate set by the National Education Board. This rate is receive more money. Validation procedures are also input into the TFF application, running on the EMIS underway to adjust for previously misreported database, to calculate school subsidies for all schools in geographical data (Box 2). Sound validation procedures PNG. The TFF application uses the rate and final census catch these reporting issues early in the data supply data—enrollment, school information, school level, chain and prevent much larger quality issues from school accounts, school locality, school type, and arising. sector—to calculate the amount of subsidy and disburse it directly into school bank accounts. Box 2: Where's my School Pilot Program DoE publishes an annual education statistics report with a message from the Secretary articulating the The EMIS Unit recently launched Where's my intention to use the report as a way to track progress School, a beta program that uses geographic toward NEP 2005–14 and to inform relevant policies locations to enable users to view satellite imagery and programs. Data include public and church schools and maps of PNG schools. The Where’s my School through secondary school as well as technical, application will eventually alert National/Provincial vocational, and teacher education and FODE. Education offices and church agencies as well as other stakeholders to the existence and location of EMIS data are currently released in two primary ways: schools. This will pave the way for a better the EducationInfo Dashboard and the Annual Statistics understanding of equity in the distribution of Book. Various education stakeholders are made aware education services to schools and students in PNG. of the release of this information through Currently the program is slow, and the same announcements sent by post. Secretary’s Circular internet access limitations apply. Additionally, the announcements also share important updates that quality of data is inconsistent. The strategy to come directly from the Office of the Secretary. These collect geographic data was paired with material announcements are usually linked to a formal policy distribution to schools. Vendors were given Global and therefore communicate issues of significance. Positioning System (GPS) devices and asked to record the location at the point of delivery. Some Data are validated at multiple points as they flow from vendors failed to register, and others recorded schools to DoE. School principals complete the National inaccurate locations. The GPS was supposed to be School Census and submit the paper form to the DEO. automated within the EMIS, but setbacks in data The DEO is the first line of quality control, validating the quality have delayed the program. Nonetheless, the data before sending them on to the Provincial Division program has strong potential, especially as efforts of Education (PDoE). The PDoE also reviews and are being made to validate and clean the existing validates data and then provides an official location data. When fully functional, this endorsement before sending them on to the EMIS Unit transparent platform has the potential to contribute at DoE, with the exception of pilot program provinces, to greater accountability across the education which input data directly into the system. Data cannot system. move on from the PEO without the official endorsement. Standards Officers—DoE staff assigned to district and provincial offices—are the key personnel SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 10 PAPUA NEW GUINEA ǀ EMIS SABER COUNTRY REPORT |2015 The EMIS Unit does its best to provide professional conduct reduce breakdowns in data security and development on data collection and management at privacy. district and provincial levels, although resource constraints exist. Districts and provincial offices should EMIS is linked to the national education plan. Key share relevant training with schools, which would help indicators are tracked in EMIS, and progression toward the EMIS Unit reach its goal of 100 percent coverage for the national target as well as time series data are the census; however, training does not reach all levels disseminated on the EMIS Dashboard (Table 4). of the education system. The nonresponse rate for schools is currently 15 percent, mostly from schools in Table 4: Framework Indicators by Plan, Compared with remote regions or those with new and untrained staff. National Targets Outcome Indicator Baseline Current Target Urban schools, primarily secondary schools, often have A.1.ACCESS: Every six- Net Admission 11.5 32.9 100 their own internal systems for collecting, processing, year-old child enrolled Rate and using student data that are entirely outside of at elementary prep. A.2.ACCESS: All children Gross 77.1 143.2 100 EMIS. enroll at elementary Admission prep. Rate Australian Department of Foreign Affairs and Trade B.4.RETENTION: A Gross 45.3 65.0 77 greater number of Completion advisors provide professional development at the children completing a Rate national level through continuous on-the-job coaching. full basic education Advisors are placed in various government offices and B.5.RETENTION: All Gross 69.8 98.3 96 children have the Enrollment share specialized skills over two-year terms, with opportunity to complete Rate (Prep. to options for extension. Apart from advisors, a significant a full quality primary Grade 8) gap is seen in technical training for the EMIS Unit. The education of nine years to grade 8 upcoming rollout of Oracle BI software is an example of B.6.RETENTION: All Net 53 74.0 85 a technical area that could be the focus of a children have the Enrollment opportunity to complete Rate (Prep. to professional development training module. a full nine years of basic Grade 8) education to grade 8 At this time, the flow of data is largely unidirectional, D.10.MANAGEMENT: % Teachers 57.4 100 Appropriately qualified Primary by from school to the national level, with limited teachers in all Type of feedback loops. Feedback loops move the flow of classrooms Qualifications information back to schools to share analysis. This is an D.11.MANAGEMENT: Student- 31.3 35.3 34 Effective use of Teacher Ratio EMIS good practice that has the potential to engage resources Primary schools and increase quality and response rates. (Grade 3 to Grade 8) D.16.MANAGEMENT: % Permanent 100 The EMIS Unit operates with high levels of Students taught in a Classrooms professionalism and ethical standards. Staff are conducive environment encouraged to attend professional conferences. D.18.MANAGEMENT: Public 5.6 −99 Education provided with Expenditure Further, an internal peer review maintains the quality of sufficient funds on Education DoE publications. Access to data is restricted to EMIS as % of Total Government staff that require access to perform their duties. For the Expenditure pilot program, provinces can manipulate only their own E.19.EQUITY: Equal Gender Parity 0.8 0.8 100 data. To ensure that professional conduct continues at opportunities for both Index boys and girls the same level, the EMIS Unit may consider establishing Source: DoE EducationInfo Dashboard, a code of professional ethics and ethical standards. http://www.education.gov.pg/Staff/Devinfo%20Dashboard/indicator.html. Documenting clear guidelines for what constitutes unethical behavior is worthwhile. These formal, approved statements ensure consistent ethical standards during team growth and turnover. Further, approved guidelines and standards on professional SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 11 PAPUA NEW GUINEA ǀ EMIS SABER COUNTRY REPORT |2015 Policy Area 4: Utilization for Decision making a request through the relevant provincial Making education office. Provinces that are part of the pilot program are gaining real-time access to EMIS data and Emerging zz|| have the ability to input, manage, and extract data, although all six pilot provinces are not yet fully utilizing The utilization of PNG’s EMIS was assessed by EMIS. examining four areas: (1) openness, (2) operational use, (3) accessibility, and (4) effectiveness in disseminating To announce key EMIS milestones or the annual push findings. of new education data, the EMIS Unit sends an announcement via postal mail to a list of education A considerable gap is found between national and stakeholders including provincial education heads subnational staff when it comes to awareness and (with extra copies to be shared with districts), line capacity to use EMIS. Within DoE, top management departments, universities, and libraries. This and, to a lesser degree, middle management staff use communication plays an important role in keeping EMIS, especially for monitoring and reporting needs. stakeholders informed and supportive of EMIS The Policy, Planning, and Research Division, where the developments. Broadening the number of recipients, EMIS Unit resides, is one of the top users of EMIS data. increasing the frequency of mailings, and adding e-mail Outside of DoE, most requests for EMIS data from communication as well as monthly or quarterly general national government entities come from the National updates and “How To” guides would help to further Economic and Fiscal Commission, the National engage stakeholders across the country. Statistical Office, the Department of National Planning and Monitoring, the National Research Institute, and The EMIS Unit has the capacity to interpret, the Department of Health. manipulate, and use data on a daily basis; further, the team does its best to share knowledge with colleagues The EducationInfo Dashboard consists of a homepage, at DoE and at the provincial level. That said, an influx in the dashboard, and framework indicators by plan. The funding for capacity-building exercises at all levels of dashboard allows users to search by sector and the education system is key to the continued indicator. Sector includes each education cycle development of EMIS. The EMIS Unit requires access to (elementary, primary, basic, secondary, vocational) and training on cutting edge programs, such as Oracle BI, as education totals. The dashboard is ideal for users who well as other best practices in EMIS. Most of the Unit’s are exploring specific indicators. Users can disaggregate training currently comes from independent research. an indicator by gender, district, or province and view DoE would benefit from targeted training on how to use trends over the last five years. Indicators are shared EMIS for planning. through different graphics, usually a PNG map, a trend line, and a bar chart. The dashboard has links where Efforts to train provinces on EMIS should continue. users can download graphics or data, but downloads do Designing and delivering EMIS training for both DEOs not always work. Possibly the largest disadvantage to and schools would engage these critical stakeholders in the dashboard is that users cannot download aggregate the EMIS process. Alerting local-level stakeholders to data. To download datasets, users must contact the the EMIS vision and roadmap is an important first step. EMIS Unit directly to submit a request. Most schools are not aware of the progress of EMIS and the availability of EMIS data. Most schools do not know The Department of National Planning and Monitoring that even if they cannot access the dashboard, they can maintains PNGInfo, which aims to serve as a single send a request to the EMIS Unit for data. Bringing local point of access for all government data. Currently stakeholders into the process, keeping them requests are made through PNGInfo for EMIS data, but consistently informed, and offering training when the two databases are not yet fully integrated. possible, even though a “Train the Trainer” network (Box 3), would likely improve overall response rates and At the provincial level, school boards use EMIS data, data quality. which they may access through the internet or by SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 12 PAPUA NEW GUINEA ǀ EMIS SABER COUNTRY REPORT |2015 Box 3: Train the Trainer Network are outside the scope of the national EMIS, evidence that schools are actively collecting and using these data A good practice in EMIS training is the reveals that in some places school-level capacity around implementation of a “Train the Trainer” network. data is quite strong. As PNG’s EMIS progresses, this This is a down-stream network strategy in which level of sophistication could be leveraged. For example, central or province-level officers train clusters of when EMIS reaches local schools, schools that have local-level officials (district and school staff), who in been implementing their own tracking systems will be turn conduct the training with their local-level well positioned to train less advanced schools on colleagues. Breakdowns in this strategy occur when utilization techniques. trainers move on to other positions, so it is important to identify the right trainers and train a large enough Box 4: School Data Utilization: The Case of Jubilee group on a consistent basis. Trainers should be Secondary excited about the potential of EMIS and should be committed to working in their current position for Jubilee Secondary School is a church agency school the foreseeable future. If possible, providing trainers in the NCD. It receives the TFF subsidy. Jubilee with a small extra stipend is recommended. reports the following data: Source: Authors. x National Census to DoE (annually) x Enrollment and staffing to National Catholic The EMIS Unit, with the support of ICT, has a vision for Education Office (monthly and quarterly) EMIS to be fully integrated with DoE and external x Assessment data to the Assessment Unit government departments and used regularly from the (biquarterly) school level to the highest levels of central x Report cards to parents (quarterly) government decision making. This vision is a driving force that powers incremental advances in EMIS. To be compliant with the above reporting requirements, while also achieving school-level EMIS is increasingly used by decision makers, learning and management outcomes, Jubilee evidenced in part by the growing number of requests maintains its own database. Teachers collect and that the EMIS Unit receives for training on how to use track student learning outcomes (e.g., short-term the system. At the provincial level, and especially tests, assignments, projects) and behavioral data. among the provinces participating in the EMIS pilot, education stakeholders are increasingly coming to the Even more importantly, data are actively used. provincial office to request data to guide decision Students with low grades or behavioral issues making. Representatives from Milne Bay remarked that, receive a series of interventions including meeting at times, the governor or members of parliament come with parents and reviews by the Academic Review to the office to request data. Committee. This is an example of using data proactively to prevent repetition and dropouts. Many schools are maintaining their own independent Administrators also use the database to track data management processes in addition to delivering teacher absenteeism and teacher in-service training, on formal data reporting requirements. This takes as well as to manage the budget and plan and track place more frequently at secondary schools, especially progress toward Jubilee’s School Learning those with some digital infrastructure (e.g., computers, Improvement Plan. internet access) and buy-in from the principal. In these schools, an Excel database is often maintained Jubilee Secondary demonstrates an advanced level (although paper forms are also frequently used) that of data utilization, but it takes place outside of tracks key indicators currently not included in the EMIS. Efforts to bring school-level utilization into national EMIS. Indicators such as student behavior, EMIS should be explored. teacher absenteeism, and student learning outcomes are captured and tracked (Box 4). Although these data SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 13 PAPUA NEW GUINEA ǀ EMIS SABER COUNTRY REPORT |2015 Schools and clients (e.g., parents, communities, and students) are not using EMIS. Schools are actively reporting data but rarely receiving any feedback. They are unaware of and/or unable to access the EducationInfo Dashboard. PNG’s EMIS is used operationally in evaluation and governance and by government (especially central government), but schools and clients mark a gap in operational use (Table 5). Table 5: Operational Use, Best Practice, and PNG Utilization Best practice PNG x School performance x Some school x Student performance performance x Limited student x Growth reports performance In x Diagnostic reports x Graduation evaluation x Graduation rates rates x Transition rates x Transition rates x Teacher performance x Policy decisions x Some policy x Accountability x Limited In x Planning accountability governance x Management x Some planning x Limited management x Academic x Limited performance academic x Teacher performance By schools performance x Management x Comparison with other schools x Parents access EMIS x Limited access x Communities access by clients EMIS By clients x Use data to make decisions x Use data to demand quality x Ratios x Some ratios x Infrastructure x Some capacity infrastructure By x Quality/outcome x Some government indicators quality/outcome x Spending efficiency indicators x Teacher salaries x Equality x Equality indicators indicators Source: Adapted from Abdul-Hamid 2014. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 14 PAPUA NEW GUINEA ǀ EMIS SABER COUNTRY REPORT |2015 Recommendations and Proposed Activities Table 6: PNG EMIS Rankings This section presents a set of recommendations and Emerging proposed activities based on the assessment of EMIS in 1. Enabling Environment zz|| PNG (Table 6). Recommendations and activities aim to improve overall EMIS functionality in a sustainable and Emerging 2. System Soundness effective manner to ensure better access and use of zz|| information for decision making, planning, and student learning. The Strengths, Weaknesses, Opportunities, Established 3. Quality Data and Threats (SWOT) profile (Figure 5) summarizes key zzz| points from the needs assessment and informs recommendations. Emerging 4. Utilization for Decision Making zz|| Figure 5: PNG EMIS SWOT Profile Strengths Weaknesses x Strong desire from decision makers for data-driven x Lack of comprehensive EMIS policies decision making x Lack of EMIS-specific budget x Education strategy (NEP 2005–14) lays the x Lack of integration with other databases/information foundation for EMIS and links EMIS with national systems (e.g., assessment and finance) goals x Lack of individual student-level data x EMIS is established at the central level x Manual processing of data in most provinces x Strong data validation procedures x Weak internet access, especially in remote areas x Talented human resources at national level, strong x Poor accuracy and reporting in remote locations EMIS Unit and ICT Department x Lack of regular audits x Sound platform, data architecture, analysis tools x Lack of internet access and limited communication x EMIS concepts and definitions follow a functional and training prevent most stakeholders outside of manual documented and approved by the the national government from accessing and using government the online dashboard x EMIS online dashboard exists and statistics book is x Limited feedback loops for data and information to published annually flow back to schools Opportunities Threats x Continued rollout of EMIS digital interface in x Intentional false reporting of data to receive greater provinces subsidy from TFF program x Next national education plan to further extend EMIS x Heavy dependence on donors for funding in PNG x Lack of integration of EMIS with other education x Continued improvements in ICT infrastructure information systems (e.g., assessment and finance) throughout the country and especially in remote x Lack of coordination with other government locations departments x EMIS Unit, with ICT Division support, has strong x Staff turnover vision for EMIS advancement x Oracle is expensive, and shifts such as moving from x Schools are eager to receive data, and many have Discoverer to BI are costly their own internal data utilization processes, which x Lack of data confidentiality procedures will make integration with EMIS much smoother x Not actively reviewing and updating the EMIS operations manual could cause setbacks in methodological soundness SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 15 PAPUA NEW GUINEA ǀ EMIS SABER COUNTRY REPORT |2015 EMIS in PNG has come a long way, and the basic An EMIS Policy should define key areas such as an processes and infrastructure that support collection, EMIS budget, data collection procedures, data sharing management, and dissemination of data are strong. and coordination, integration with external databases, Establishing sound processes and infrastructure is no professional development at national and subnational small task, and DoE should be commended for the EMIS levels, confidentiality, comprehensive and quality achievements to date. The following recommendations data, penalties for false reporting, and utilization of summarize key improvements that have the potential data at all levels of the education system. Given that to continue the exciting growth trajectory of EMIS in the EMIS Unit did not exist at the time that the ICT PNG. Policy was drafted, the ICT Policy does not fully support EMIS. The next five-year national education plan will be The commitment to establishing EMIS and a qualified published in the coming months. The plan will advance EMIS Unit would be bolstered by an equal EMIS with a new roadmap as well as a log frame that commitment to establish EMIS policies and an EMIS aims to strengthen EMIS indicators and tracking budget. The biggest obstacle to the enabling mechanisms. It also provides an opportunity to initiate environment for EMIS in PNG is the lack of official EMIS development of more comprehensive EMIS policies. policies and a standalone EMIS budget. For EMIS to reach the next level in its development, these gaps must Significant resources have been invested into EMIS be addressed. Basic policies that mandate EMIS, processes, structure and software; however, that position the EMIS Unit as the official body responsible investment is at risk if it is not supported by regular for education data, call for an EMIS budget, and set internal and external audits, coupled with the requirements around dissemination and utilization will resources to act on feedback from audits. Audits are catapult EMIS forward and ensure institutionalization of currently taking place on an ad hoc basis, often as a the system. result of available funds. Commitment to and investment in internal and external audits not only NEP 2005–14, as well as supporting policies such as the protect the larger investment in EMIS, but also defend ICT Policy and TFF, catalyzed the development of a EMIS when the reliability of data and analysis in the much stronger EMIS, supported by a qualified team, system is questioned good infrastructure, and sound processes. To continue this momentum, and to support and empower the EMIS The unit of measurement in the system should be Unit in building a long-term and sustainable strategy for expanded to the student. Currently EMIS does not have EMIS development, the next five-year period should unique identifiers for students. As a result, it is not institutionalize EMIS through an overarching EMIS capable of linking student performance with teachers. Policy (Box 5). Teacher registration numbers are recorded in the census and could be the basis for a teacher identifier in Box 5: Rationale for a Guiding EMIS Policy the future. Course codes are also not included in the system. Student identifiers are generally used in EMIS policies drive effectiveness and safeguard the established and advanced systems and enable system in various ways. Policies that mandate EMIS and longitudinal tracking and greater insight into the impact EMIS resources establish continuity and sustainability for of inputs (e.g., professional development, changes in the system. Further, policies that specify responsibilities and roles for data collection and management prevent curriculum, policies, programs) on student learning interference from external agencies. Similarly, policies outcomes. Student identifiers must be designed that guide processes and procedures for working with carefully to ensure privacy and security. This entails a other units and other sources of data contribute to process whereby identifiable data are replaced by improvements in quality, efficiency, and integration. nonidentifiable data so as to protect the individual Policies can also help to establish the supply of data into identity of each student. Ultimately, student-level data the EMIS by requiring schools to submit data in a timely help answer more detailed policy questions that can manner and positioning the EMIS team as the primary inform better decision making (Box 6). data collection authority. Source: Abdul-Hamid 2014. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 16 PAPUA NEW GUINEA ǀ EMIS SABER COUNTRY REPORT |2015 Box 6: Example of Data Collected by EMIS in Fiji To be fully operational in decision making and planning as well as teaching and learning, PNG EMIS EMIS in Fiji (FEMIS) captures individual student data will need to strengthen integration capabilities, both entered at the school level, which contains information at a technical level as well as through stakeholder such as student identification number, registered birth dialogue and consensus building. Dialogue with other number, parent details, gender, date of birth, home units, especially the Assessment Unit, will lay the situation (household income, electricity), school groundwork for integration. One approach in attendance, record of school fees, and financial collaborating with different units or departments is to assistance. In addition, it captures health records of conduct a workshop designed to identify the key each student, including special needs data. FEMIS also questions that each party needs the data to answer to links to the national teacher data system and effectively achieve goals and monitor outcomes. Once assessment data system. These links help answer a questions are identified, the group works backwards to range of questions, such as: which children with ensure that the system is able to capture the necessary disabilities, in which settings, and under what circumstances, are achieving what educational data to answer these questions. outcomes? And, which teachers with what qualifications are creating environments that result in Investments should be made in capacity building at all good learning outcomes? levels. A common threat to the progression of an EMIS is that funding goes toward infrastructure but does not Source: Sprunt 2014. go into training. Education stakeholders at all levels of the system should be trained. This includes engaging The unidirectional nature of EMIS in PNG, from schools local-level staff, especially school principals and to DoE, needs to evolve to include feedback loops that teachers, as well as training national government carry information back to the local level. Although a decision makers on how to use EMIS for planning. The good first step is to establish a strong flow of data from EMIS Unit also requires continuous training to stay up schools to provinces and finally to the national level, to date with relevant tools and techniques. with validation procedures at each stage, it is critical for EMIS to institute feedback loops that carry information PNG EMIS would gain momentum and strength by back down the chain to the local level (Figure 6). developing a communications strategy to share the Although data are available online, limited internet vision, key activities, and engagement opportunities access prevents school officials, local authorities, and with education stakeholders across the country. communities from accessing data. Feedback loops Schools and districts should receive targeted increase utilization of data at the local level and communication materials so that they gain an improve the frequency and accuracy of source data. understanding of the national strategy around Brochures, pamphlets, or briefs that share simple education data and PNG’s EMIS. Schools are a critical information, both national analysis as well as relevant part of this strategy because they are the source of local data, and promote the online dashboard would data. Bringing them into the process and recognizing have long-term benefits for EMIS and the education their important role through simple communication system. materials has the potential to build excitement around education data and encourage increased participation. Figure 6: EMIS Information Cycle Data Data Decision making and User feedback should be collected to inform future collection utilization policy making improvements to the EducationInfo Dashboard. The Data Inputs School warehouse dashboard is a critical point of dissemination, but like any online interface, it should continue to evolve based Student learning, other Parents/Students Research outcomes, and policy on user feedback and the evolving indicators and Teachers Analysis actions outcomes set by the education system. When possible, School Management Reporting conducting focus groups and user testing of the site Feedback could provide information on how to improve the user Source: Abdul-Hamid 2014. experience. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 17 PAPUA NEW GUINEA ǀ EMIS SABER COUNTRY REPORT |2015 Acknowledgments NCD National Capital District NEP National Plan for Education This report was prepared by SABER-EMIS team PaBER Pacific Benchmarking for Education Results members Sarah Mintz and Namrata Saraogi, under the PDoE Provincial Division of Education supervision of Husein Abdul-Hamid (Senior Education PEO Provincial Education Office Specialist, Education Global Practice, World Bank). The PNG Papua New Guinea report was developed in collaboration with Fred SABER Systems Approach for Better Education Brooker (Senior Education Specialist, Australian Results Department of Foreign Affairs and Trade) and in SWOT Strengths, Weaknesses, Opportunities, and partnership with the Pacific Benchmarking for Threats Education Results (PaBER) initiative. 1 With PaBER, TFF Tuition Fee Free SABER has assessed various parts of the education UBE Universal Basic Education system including School Accountability and Autonomy, Student Assessment, Teacher Policies, and now EMIS. References Similar to SABER, the PaBER initiative links policy with implementation, identifies areas to strengthen policy, Abdul-Hamid, Husein. 2014. “SABER EMIS Framework expands knowledge dissemination, and improves the Paper.” World Bank, Washington, DC. quality of education and student performance across Education for All (EFA). 2015. “National Review Report: the pacific. The partnership has run pilot programs in Papua New Guinea.” DoE, Port Moresby. Papua New Guinea, Samoa, and the Solomon Islands. Papua New Guinea Department of Education (DoE). 2004. “National Plan for Education 2005–2014.” Special thanks to colleagues providing insights and DoE, Port Moresby. feedback: Stephen Close, Manar El-Iriqsousi, Cassia ———. 2005. “ICT Policy.” DoE, Port Moresby. Miranda, and Michael Melamed. The report benefited immensely from the guidance and support of Regina ———. 2013. “2013 Education Statistics.” DoE, Port Mabia (PaBER Coordinator). The SABER EMIS team is Moresby. especially grateful to the Papua New Guinea ———. 2015. “Education for All 2015 National Review Department of Education, especially James Agigo Report: Papua New Guinea.” DoE, Port Moresby. (Acting Assistant Secretary, Research, Evaluation and ———. n.d. EducationInfo Dashboard. DoE, Port Statistics Division), Thomas Podarua (Assistant Moresby. Secretary, ICT Division), and Priscilla Rasehei (Manager, http://www.education.gov.pg/Staff/Devinfo%20Da EMIS Unit). shboard/index.html. Sprunt, B. 2014. “Efforts to Improve Disability Acronyms Disaggregation of the Fiji Education Management BI Business Intelligence (Oracle) Information System.” Working Paper. Nossal DEO District Education Office Institute for Global Health, University of DoE Department of Education Melbourne, Melbourne. EMIS Education Management Information System FEMIS EMIS in Fiji FODE Flexible, Open, and Distance Education GPS Global Positioning System GoPNG Government of Papua New Guinea ICT Information and Communication Technologies 1 PaBER is funded by the Australian government and is coordinated through the Educational Quality and Assessment Programme (formerly the South Pacific Board for Educational Assessment). SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 18 PAPUA NEW GUINEA ǀ EMIS SABER COUNTRY REPORT |2015 Appendix A: Summary of Policy Lever Benchmarking Policy area Policy lever Scorea Weight Benchmark Legal framework 1.45 15% Emerging Organizational structure and 2.67 15% Established institutionalized processes Enabling environment Human resources 2.53 15% Established Infrastructural capacity 2.42 15% Established Budget 2.67 15% Established Data-driven culture 2.29 10% Established Data architecture 2.86 20% Established Data coverage 1.34 30% Emerging System soundness Data analytics 1.33 15% Emerging Dynamic system 1.87 15% Emerging Serviceability 2.23 20% Established Methodological soundness 3.12 25% Advanced Accuracy and reliability 2.59 25% Established Quality data Integrity 1.02 25% Emerging Periodicity and timeliness 1.67 25% Emerging Openness 1.22 15% Emerging Operational use 1.04 50% Emerging Utilization in decision making Accessibility 1.94 20% Emerging Effectiveness in disseminating 1.40 15% Emerging findings a. 0–0.99 = Latent; 1–1.9 = Emerging; 2–2.9 = Established; 3–4 = Advanced. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 19 PAPUA NEW GUINEA ǀ EMIS SABER COUNTRY REPORT |2015 Appendix B: Extended Rubric, PNG Scores Highlighted in Red Scoring Description of Policy levers Indicators best practices Latent Emerging Established Advanced The system contains crucial components of a comprehensive The system enabling The system lacks The system contains contains basic The system contains environment, which major components of most components of components of a crucial components of POLICY AREA 1: ENABLING ENVIRONMENT addresses related a comprehensive a comprehensive comprehensive a comprehensive policy elements and enabling enabling enabling enabling environment enables the environment environment environment functioning of an effective and dynamic system Institutionalization of system: EMIS is institutionalized as an integral part of the education system and the government Responsibility: responsibility for collecting, processing, and disseminating education statistics is given to a clearly designated institution Basic components or agency An existing legal of a legal There is an existing Most elements of a Dynamic framework: the legal framework supports A legal framework is framework or legal framework to 1.1 Legal framework legal framework are framework is dynamic and elastic so a fully functioning not in place informal support a fully in place that it can adapt to advancements in EMIS mechanisms are in functioning EMIS technology place Data supply: the legal framework mandates that schools participate in EMIS by providing education data Comprehensive, quality data: the requirement for comprehensive, quality data is clearly specified in the EMIS legal framework SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 20 PAPUA NEW GUINEA ǀ EMIS SABER COUNTRY REPORT |2015 Scoring Description of Policy levers Indicators best practices Latent Emerging Established Advanced Data sharing and coordination: the legal framework allows for adequate data sharing and coordination between the Ministry of Education and agencies and/or institutions that require education data Utilization: the legal framework emphasizes data-driven education policy Budget: the education system budget includes a line item for EMIS Confidentiality: the legal framework guarantees that respondents’ data are confidential and used for the sole purpose of statistics The system is The institutional institutionalized The institutional The system is structure of the within the The system is not structure of the institutionalized within system is defined government, has specified in policies, system is not clearly the government, has Organizational within the well-defined and what exists does specified in policies, well-defined structure and Organizational structure and government, and it 1.2 organizational not have well-defined it has some organizational institutionalized institutionalized processes has defined processes, and has organizational organizational processes, and has processes organizational several processes; EMIS has processes, and its several functionalities processes, but its functionalities limited functionalities functionalities are beyond statistical functionalities are beyond statistical limited reporting limited reporting Qualified staff Minimum standards Some staff are The majority of staff All staff are qualified operate the system, of qualification are qualified to operate are qualified to to operate the system, Personnel: the core tasks of EMIS are and opportunities not met for the the system, and operate the system, and well-established 1.3 Human resources identified, and EMIS is staffed with are available to majority of staff that limited and frequent opportunities are qualified people improve their operate the system opportunities are opportunities are constantly available to performance and and opportunities are available to improve available to improve improve staff retention not available to staff performance staff performance performance and SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 21 PAPUA NEW GUINEA ǀ EMIS SABER COUNTRY REPORT |2015 Scoring Description of Policy levers Indicators best practices Latent Emerging Established Advanced improve their and retention and retention retention performance and Professional development: professional retention training is available for EMIS staff Data collection: tools for data collection are available Database(s): databases exist under the The system has a The system has a well- umbrella of the data warehouse and well-defined defined infrastructure have both hardware and software infrastructure to The system has an to fully perform its means perform data The system lacks a The system has a infrastructure that Infrastructural data collection, 1.4 Data management system: a system is collection, well-defined basic or incomplete allows it to perform capacity management, and in place that manages data collection, management, and infrastructure infrastructure some of its functions dissemination processing, and reporting dissemination in an integral manner functions in an integral functions in an Data dissemination: data dissemination manner integral manner tools are available and maintained by the agency producing education statistics Personnel and professional development: the EMIS budget contains a specific budget for EMIS personnel and their professional development Maintenance: the EMIS budget contains The system budget a specific budget for system contains the majority maintenance and recurrent costs The system budget is The system budget is of required comprehensive, The system suffers The system has a comprehensive, categories to ensure 1.5 Budget Reporting: the EMIS budget contains a ensuring that the from serious basic or incomplete ensuring that the that most parts of specific budget for reporting costs system is sustainable budgetary issues budget system is sustainable the system are and efficient and efficient sustainable and Physical infrastructure: the EMIS budget efficient contains a specific budget for physical infrastructure costs Efficient use of resources: processes and procedures are in place to ensure that resources are used efficiently SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 22 PAPUA NEW GUINEA ǀ EMIS SABER COUNTRY REPORT |2015 Scoring Description of Policy levers Indicators best practices Latent Emerging Established Advanced A data-driven culture A data-driven culture exists that prioritizes The system has a prioritizes data as a The system suffers data management and data-driven culture A data-driven culture fundamental because there is not a utilization within and that demonstrates a exists that prioritizes element of data-driven culture beyond the education Data-driven basic appreciation data management Data-driven culture operations and that prioritizes data system, and evidence Culture of data and interest and utilization within decision making, management and of that culture is in developing better and beyond the both inside and data utilization in present in daily data utilization education system outside of the decision making interaction and practices education system decision making at all levels The system has The system has some The processes and The processes and basic processes and processes and a structure of the structure of EMIS are The system lacks a structure that do structure, but they system are sound and POLICY AREA 2: SYSTEM SOUNDNESS sound and support processes and not support the do not fully support support the the components of structure components of an the components of components of an an integrated system integrated system an integrated system integrated system The system's data The system's data structure has most The data architecture The system's data architecture elements of the data The data architecture is well defined to structure does not includes some architecture; is well defined to 2.1 Data architecture Data architecture ensure full system have a well-defined components; however, it has some ensure full system functionality data architecture however, it is deficiencies that functionality incomplete affect the system's functionality Administrative data: EMIS contains administrative data The data in the system are The data in the Financial data: EMIS contains financial comprehensive and The data in the The data in the The data in the system system are far from data cover administrative, system include system include most are comprehensive 2.2 Data coverage being comprehensive, financial, human some of the data but not all of the data and cover all data Human resources data: EMIS contains and coverage is resources, and areas areas areas human resources data limited learning outcomes Learning outcomes data: EMIS contains data learning outcomes data SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 23 PAPUA NEW GUINEA ǀ EMIS SABER COUNTRY REPORT |2015 Scoring Description of Policy levers Indicators best practices Latent Emerging Established Advanced Basic tools and processes are Tools and processes available, but the Tools and processes Tools and processes are available to system is not Tools and processes are available; are available to perform data capable of 2.3 Data analytics Data analytics are used to perform however, data perform data analytics analytics at different conducting limited tabulations analytics are not at different levels on a levels on a regular advanced analytical performed regularly regular basis basis steps (e.g., predictive models, projections) Quality assurance measures: the system is dynamic and maintains quality The system in place The system in place is assurance measures The system in place is not easily The system in place is not easily adaptable is elastic and easily adaptable and The system in place is elastic and easily Data requirements and considerations: to changes adaptable to allow requires significant easily adaptable, but adaptable to allow for 2.4 Dynamic system mechanisms exist for addressing new /advancements in for changes time and resources it remains reasonably changes/ and emerging data requirements data needs, as no /advancements in to accommodate complex advancements in data System adaptability: EMIS is elastic and quality assurance data needs changes and/or needs easily adaptable to allow for changes standards are used advancements and/or advancements in data needs Validity across data sources: information brought together from different data and/or statistical Services provided by frameworks in EMIS is placed within the Services provided by the system are valid data warehouse using structural and the system are valid across data sources, consistency measures across data sources, integrate integrate Integration of noneducation databases noneducation noneducation into EMIS: data from sources collected databases into EMIS, Serious issues exist Inconsistencies exist The data are databases into EMIS, by agencies outside EMIS are integrated and archive data at related to data related to data consistent and valid; 2.5 Serviceability and archive data at the into the EMIS data warehouse the service of EMIS validity and validity and however, some service of EMIS clients Archiving data: multiple years of data clients by ensuring consistency consistency concerns still exist by ensuring the are archived, including source data, the relevance, relevance, consistency, metadata, and statistical results consistency, usefulness, and usefulness, and Services to EMIS clients: services timeliness of its timeliness of its provided by the system to EMIS clients statistics statistics include ensuring the relevance, consistency, usefulness, and timeliness of its statistics SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 24 PAPUA NEW GUINEA ǀ EMIS SABER COUNTRY REPORT |2015 Scoring Description of Policy levers Indicators best practices Latent Emerging Established Advanced The system has most mechanisms in place The system has the needed to collect, The system has the mechanisms The system has save, and produce required mechanisms required to collect, basic mechanisms timely, high-quality The system lacks in place to collect, save, produce, and to collect, save, and information for use mechanisms to save, produce, and utilize information, produce timely, in decision making; collect, save, or utilize information, POLICY AREA 3: QUALITY DATA which ensures quality however, some produce timely, high- which ensures accuracy, security, information; additional measures quality information accuracy, security, and and timely, high- however, its are needed to for decision making timely, high-quality quality information accuracy might be ensure accuracy, information for use in for use in decision questionable security, and/ or decision making making timely information that can be used for decision making Concepts and definitions: data fields, records, concepts, indicators, and metadata are defined and documented in official operations manuals along with other national datasets and endorsed by the government The methodological The methodological The methodological The methodological The methodological basis for producing basis for producing basis for producing Classification: defined education system basis for producing basis for producing educational statistics educational educational statistics classifications are based on technical educational statistics educational statistics from raw data statistics follows the follows most Methodological guidelines and manuals does not follow from raw data follows 3.1 follows basics of required soundness Scope: the scope of education statistics internationally internationally internationally internationally internationally is broader than and not limited to a accepted standards, accepted standards, accepted standards, accepted standards, accepted standards, small number of indicators (e.g., guidelines, or good guidelines, and good guidelines, and good guidelines, and guidelines, and good measurements of enrollment, class size, practices practices practices good practices practices and completion) Basis for recording: data-recording systems follow internationally accepted standards, guidelines, and good practices Source data: available source data Source data and Source data and Source data and Source data and Source data and Accuracy and 3.2 provide an adequate basis for compiling statistical techniques statistical techniques statistical statistical techniques statistical techniques reliability statistics are sound and lack soundness and techniques have follow most required are sound and reliable, SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 25 PAPUA NEW GUINEA ǀ EMIS SABER COUNTRY REPORT |2015 Scoring Description of Policy levers Indicators best practices Latent Emerging Established Advanced Validation of source data: source data reliable, and reliability basic soundness elements to be sound and statistical outputs are consistent with the definition, statistical outputs and reliability, but and reliable, but sufficiently portray scope, and classification as well as time sufficiently portray statistical outputs statistical outputs do reality of recording, reference periods, and reality do not portray not portray reality valuation of education statistics reality Statistical techniques: statistical techniques are used to calculate accurate rates and derived indicators Professionalism: EMIS staff exercise their profession with technical Education statistics Education statistics independence and without outside contained within Education statistics contained within the interference that could result in the the system are contained within the Education statistics system are mostly violation of the public trust in EMIS Education statistics guided by limited system are guided by contained within the guided by principles statistics and EMIS itself contained within the principles of all three principles of 3.3 Integrity system are not guided of integrity (two of system are guided by integrity (one of the integrity: Transparency: statistical policies and by principles of the three principles principles of integrity three principles of professionalism, practices are transparent integrity of professionalism, professionalism, transparency, and Ethical standards: policies and practices transparency, and transparency, and ethical standards in education statistics are guided by ethical standards) ethical standards) ethical standards Periodicity: the production of reports and other outputs from the data The system produces The system The system produces The system produces The system produces warehouse occur in accordance with data and statistics produces some data most data and Periodicity and data and statistics all data and statistics 3.4 cycles in the education system neither periodically and statistics statistics periodically timeliness periodically in a periodically and in a Timeliness: final statistics and financial nor in a timely periodically and in a and in a timely timely manner timely manner statistics are both disseminated in a manner timely manner manner timely manner The system is wholly There are no signs The system is used The system is wholly The system is used utilized by different that EMIS is utilized by most education utilized by different by some education users for decision in decision making by stakeholders but is users for decision POLICY AREA 4: UTILIZATION FOR DECISION MAKING stakeholders, but making at different the majority of not fully operational making at different not for major policy levels of the education in governmental levels of the education decision making education system stakeholders decision making system EMIS stakeholders: EMIS primary The system is open The system lacks The system is open The system is open to The system is open to stakeholders are identified and use the to education openness to to some education the majority of all education 4.1 Openness system in accordance with the legal stakeholders in terms education stakeholders in education stakeholders in terms framework of their awareness stakeholders in terms terms of their stakeholders in terms of their awareness and SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 26 PAPUA NEW GUINEA ǀ EMIS SABER COUNTRY REPORT |2015 Scoring Description of Policy levers Indicators best practices Latent Emerging Established Advanced User awareness: current and potential and capacity to of their awareness awareness and of their awareness capacity to utilize the EMIS users are aware of EMIS and its utilize the system and capacity to utilize capacity to utilize and capacity to utilize system outputs the system the system the system User capacity: EMIS users have the skills to interpret, manipulate, and utilize the data produced by the system to ultimately disseminate findings Utilization in evaluation: data produced by EMIS are used to assess the education system Utilization in governance: data produced by EMIS are used for governance purposes Data produced by Data produced by Data produced by the Data produced by the Data produced by the Utilization by schools: data produced by the system are used the system are used system are used in system are used in system are not used in 4.2 Operational use EMIS are used by schools in practice by the in practice by some practice by the practice by the main practice by education main education education majority of education education Utilization by clients: data produced by stakeholders stakeholders stakeholders stakeholders stakeholders EMIS are used by clients (including parents, communities, and other actors) Utilization by government: the system is able to produce summative indicators (derived variables) to monitor education system Understandable data: data are presented in an easily digestible manner Widely disseminated data: education Education statistics Education statistics are statistics are disseminated beyond the are presented in an presented in an Ministry of Education and/or the understandable understandable education statistics–producing agency manner and are The system suffers The system has The system has manner and are widely 4.3 Accessibility to other EMIS stakeholders widely disseminated from serious major accessibility minor accessibility disseminated using a Platforms for utilization: platforms are using clear platforms accessibility issues issues issues clear platform for standardized across EMIS and are for utilization, utilization, customizable to user needs complemented by complemented by user user support support User support: assistance is provided to EMIS users upon request to help them access the data SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 27 PAPUA NEW GUINEA ǀ EMIS SABER COUNTRY REPORT |2015 Scoring Description of Policy levers Indicators best practices Latent Emerging Established Advanced Dissemination strategy: national A dissemination plan governments have an information has been dissemination strategy in place Dissemination of Dissemination is implemented; The dissemination of Effectiveness in Dissemination is education statistics reasonably however, room exists education statistics via 4.4 disseminating neither strategic nor Dissemination effectiveness: via EMIS is strategic strategic, but for improvement (for EMIS is strategic and findings effective dissemination of EMIS statistics is and effective ineffective full effectiveness in effective effective relation to strategic engagement) SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 28 PAPUA NEW GUINEA ǀ EMIS SABER COUNTRY REPORT |2015 www.worldbank.org/education/saber The Systems Approach for Better Education Results (SABER) initiative collects data on the policies and institutions of education systems around the world and benchmarks them against practices associated with student learning. SABER aims to give all parties with a stake in educational results—from students, administrators, teachers, and parents to policy makers and business people—an accessible, detailed, objective snapshot of how well the policies of their country's education system are oriented toward ensuring that all children and youth learn. This report focuses specifically on policies in the area of Education Management Information Systems. This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 29