67660 The World Bank Human Development Network - Education System Assessment and Benchmarking for Education Results (SABER) ANTIGUA AND BARBUDA Education Management Information System (EMIS) COUNTRY REPORT Emilio Porta, Jennifer Klein, Gustavo Arcia and Harriet Nannyonjo February 2012 Acknowledgements This report was prepared by a team led by Emilio Porta, Senior Education Specialist at the Human Development Network/Education at the World Bank; and consisting of Gustavo Arcia, Consultant to the Human Development Network/Education of the World Bank and Senior Economist at Analítica LLC in Miami, Florida; Jennifer Klein, Consultant to the Human Development Network/Education at the World Bank, and Harriet Nannyonjo, Senior Education Specialist, LCSHE, World Bank. The report was prepared under the guidance of Elizabeth King, Robin Horn and Chingboon Lee. The views expressed here are those of the authors and should not be attributed to the World Bank Group. All data contained in this report is the result of collaboration between the authors, the Organization of Eastern Caribbean States, and participants in the benchmarking exercise. All errors are our own. This benchmarking study arose from an active partnership between the Education Reform Unit of the Organization of Eastern Caribbean States (OECS) and the World Bank. The benchmarking exercise was done during an OECS workshop conducted in Castries, St. Lucia, from January 23 to January 28, 2011, with the participation of government officials from Antigua & Barbuda, the Commonwealth of Dominica, Grenada, St. Kitts & Nevis, St. Lucia, and St. Vincent & the Grenadines. A delegate from Montserrat also attended as an observer. The workshop and benchmarking exercise were done under the invaluable leadership of Marcellus Albertin, Head of the Education Reform Unit (OERU) at the OECS. His unflagging support, enthusiasm, and institutional supervision were fundamental for the cooperation of all participants and for the success of the workshop. To him we owe a great deal of gratitude. We would like to thank the OERU staff that helped us with workshop logistics, especially Emma Mc Farlane- Jouavel and Beverly Pierre. We would also like to thank the workshop participants: Doristeen Etinoff, Priscilla Nicholas, and Patricia George from Antigua & Barbuda; Ted Serrant, Robert Guiste, and Weeferly Jules from Dominica; Pauleen Finlay, Michelle Peters, and Imi Chitterman from Grenada; Gregory Julius from Monserrat; Quinton Morton, Ian Gregory, and Laurence Richards from St. Kitts & Nevis; Kendall Khodra, Nathalie Elliott, Sisera Simon, Evariste John, and Valerie Leon from St. Lucia; Dixton Findlay, Keith Thomas, and Junior Jack from St. Vicent & Grenadines; Darrel Montrope, Jacqueline Massiah, Sean Mathurin, and Loverly Anthony- Charles from the OECS. Abbreviations EMIS Education Management Information System MOE Ministry of Education OECD Organization for Economic Cooperation and Development OECS Organization of Eastern Caribbean States SABER System Assessment and Benchmarking for Education Results SEAT SABER EMIS Assessment Tool UIS UNESCO Institute for Statistics UNESCO United Nations Educational, Scientific, Cultural Organization ANTIGUA AND BARBUDA EMERGING  Aspect of Data Quality Benchmark Prerequisites of Quality Emerging ¤¤¢¢ Assurances of Integrity Emerging ¤¤¢¢ Methodological Soundness Emerging ¤¤¢¢ Accuracy and Reliability Emerging ¤¤¢¢ Serviceability Latent ¤¢¢¢ Accessibility Emerging ¤¤¢¢ 1 BACKGROUND Education Data in Antigua and Barbuda Education data in Antigua and Barbuda is collected through both paper and electronic questionnaires. The electronic questionnaires are collected from three secondary schools through the AbusSTAR EMIS software that was developed in Barbados. The Ministry of Education (MOE) has plans to expand the use of the EMIS software to all secondary schools, but currently continues the use of an annual paper data collection questionnaire in the absence of a fully digitized EMIS throughout the country. FACILITIES AND EQUIPMENT. The Ministry of Education provides EMIS hardware but the software is provided and maintained by a private company from Barbados under contract with the Government. The electronic EMIS system is web-based and accessible only with a password. This system only currently exists in three secondary schools, and the use of this equipment and software will be expanded throughout the country. EMIS STAFF. The Ministry of Education provides data entry clerks and a management team. A school management team is trained to carry out functions at the school level. EMIS DATA. The electronic EMIS has three modules: a student module, a module for managing furniture and materials, and a teacher administration module. DATA COLLECTION. Data are collected in all public and private schools, but only 50% of the private schools send their data to be included in the EMIS. DATA PROCESSING. The data from the three secondary schools that submit data electronically are aggregated manually at the planning unit and verification is done at the Ministry of Education in collaboration with school principals and the EMIS team. Errors are corrected as soon as they are detected and officers in the planning unit design the output tables. PUBLICATIONS. The EMIS project leader coordinates data publications and reports the EMIS results to the MOE, schools, parents, international agencies and other stakeholders. The planning unit prepares the Statistical Digest. 2 The EMIS in Antigua and Barbuda EMERGING:  In January 2011, Antigua and Barbuda’s EMIS was assessed using the SABER- Figure 1. SABER EMIS Scores in the OECS EMIS Assessment Tool (SEAT) and overall the EMIS was categorized as EMERGING (0.46). Among the six Organisation of Eastern Caribbean States (OECS) countries, Antigua had the lowest overall score (Table 1) but the country’s OECS rankings varied for each Aspect of Quality (Figure 1). Even though Antigua’s scores on all of the Aspects of Quality were lower than the OECS average, Antigua’s scores were not the lowest score of the OECS countries on three of six Aspects. Antigua was only slightly below the OECS average on Assurances of Integrity (0.53). The next sections of this country report will analyze Antigua’s performance on the sub-components of each Aspect of Quality in order to present a detailed portrait of the strengths and weaknesses of the EMIS and many concrete actions that the country can take to improve education data quality. Table 1. SABER EMIS Scores in the OECS Countries (2011) OECS Dominica Antigua Grenada St. Kitts St. Vincent St. Lucia Average Pre-Requisites 0.70 0.52 0.68 0.66 0.45 0.64 0.61 of Quality Assurances of 0.58 0.53 0.61 0.44 0.50 0.64 0.55 Integrity Methodological 0.83 0.50 0.67 0.67 0.83 0.67 0.69 Soundness Accuracy and 0.70 0.48 0.58 0.75 0.53 0.58 0.60 Reliability Serviceability 0.61 0.29 0.50 0.79 0.43 0.68 0.55 Accessibility 0.47 0.47 0.69 0.61 0.36 0.56 0.53 Overall 0.65 0.46 0.62 0.65 0.52 0.63 0.59 Latent Emerging Established Mature 0 – 0.3 0.31 - 0.59 0.6 - 0.79 0.8 - 1 3 PREREQUISITES OF QUALITY EMERGING:  The Prerequisites of Quality are still EMERGING (0.52) in Figure 2. Prerequisites of Quality Antigua (Figure 2). With one statistician responsible for the EMERGING collection and dissemination of data, Antigua’s roles and    responsibilities are somewhat clear despite the lack of law related specifically to education statistics (Table 2, 0.1). There are also no formal agreements on data sharing, but an informal agreement exists where institutions share data upon request (0.2). The confidentiality of individual data exists despite a lack of laws to support it (0.3). Antigua could improve the Prerequisites of Quality by formalizing these informal arrangements. The staff, facilities, computing resources, and financing are somewhat commensurate with EMIS activities and these resources are efficiently managed, but the EMIS would be more effective with updated computers and additional resources (0.5 / 0.6). Processes are not yet in place to focus on or monitor data quality in Antigua (0.8). There is a commitment to quality among management, but there is a need for more tangible strategies and interventions. For example, there are no formal reviews of data quality and little user feedback on data quality (0.9). Antigua & OECS Table 2. Prerequisites of Quality: Subcomponents Benchmark Barbuda Average Responsibility for collecting and disseminating education data is Emerging 0.1 0.50 0.75 clearly specified  Emerging 0.2 Data sharing and coordination among different agencies are adequate 0.50 0.50  Individual/personal data are kept confidential and used for statistical Established 0.3 0.75 0.79 purposes only  Statistical reporting is ensured through legal mandate and/or Established 0.4 0.75 0.58 measures to encourage response  Staff, facilities, computing resources, and financing are Established 0.5 0.75 0.63 commensurate with the activities  Processes and procedures are in place to ensure that resources are Established 0.6 0.75 0.63 used efficiently  Education statistics meet user needs and those needs are monitored Established 0.7 0.75 0.75 continuously  Emerging 0.8 Processes are in place to focus on quality 0.50 0.63  Latent 0.9 Processes are in place to monitor the quality of data processes 0.00 0.33  Processes are in place to deal with quality considerations in planning Emerging 0.10 0.25 0.58 the stat program  Mechanisms exist for addressing new and emerging data Emerging 0.11 0.25 0.54 requirements  4 ASSURANCES OF INTEGRITY EMERGING:  Antigua’s Assurances of Integrity are still Figure 3. Assurances of Integrity in the OECS EMERGING (0.53) and the country scored just below the OECS average (0.55) (Figure 3). This score was Antigua’s highest on any Aspect of Quality. EMERGING    Only informal mechanisms protect the professional independence of the data producing institution, which can ensure that statistics are produced on an impartial basis (Table 3, 1.1). The terms and conditions under which statistics are collected, processed, and disseminated are not available to the public (1.5) and the public is not informed about internal access to preliminary data (1.6). The professionalism of EMIS staff is currently ESTABLISHED through guidelines for staff behavior but these guidelines are not actively enforced (1.9). Also, while staff are recruited and promoted based on professional credentials, professionalism could be further promoted by encouraging staff to publish and by establishing a peer review process (1.2). Antigua earned a MATURE score in one sub- component by giving advance notice of major changes in methods, sources, and techniques as soon as the decision is made (1.8). Antigua & OECS Table 3. Assurances of Integrity: Subcomponents Benchmark Barbuda Average Emerging 1.1 Statistics are produced on an impartial basis 0.25 0.38  Emerging 1.2 Professionalism of staff is actively promoted 0.50 0.42  Choices of data sources and statistical techniques are made solely by Established 1.3 0.75 0.83 statistical considerations  Agency is entitled to comment on erroneous interpretation and misuse Established 1.4 0.75 0.58 of statistics  Latent 1.5 Terms and conditions are available to the public 0.00 0.33  Public is aware of internal governmental access to statistics prior to Latent 1.6 0.00 0.38 their release  Established 1.7 Products of education statics agency are clearly identified 0.75 0.50  Advanced notice is given of major changes in methodology, source Mature 1.8 1.00 0.71 data, and statistical techniques  Guidelines for staff behavior are in place and are well known to the Established 1.9 0.75 0.83 staff  5 METHODOLOGICAL SOUNDNESS EMERGING:  In terms of Methodological Soundness, Antigua’s Figure 4. Methodological Soundness in the EMIS is EMERGING (0.50). Antigua scored below OECS countries the OECS average (0.69) and had the lowest score of the OECS countries (Figure 4). Antigua’s highest sub-component score was on classification systems: Antigua follows the EMERGING International Standard Classification of Education    (ISCED) in most education sector data except expenditure data (Table 4, 2.3). Expanding the use of ISCED to expenditure data would ensure complete consistency with ISCED and improve Antigua’s score on this subcomponent. Despite an ESTABLISHED score on classification systems, Antigua’s structure, concepts, and definitions do not have proper documentation and may not be consistent with regional and international standards (2.1) established by the UNESCO Institute for Statistics (UIS) and the OECS Education Reform Unit (OERU). Currently, Antigua’s EMIS produces between 71 and 90 percent of UIS indicators annually, which results in a EMERGING benchmark on the scope of statistics sub-component (2.2). Expanding the scope of statistics produced to 100 percent of UIS and OECD indicators is ideal and can enable additional domestic, regional, and international education policy analysis. Antigua & OECS Table 4. Methodological Soundness: Subcomponents Benchmark Barbuda Average Overall structure, concepts and definitions follow regionally and Emerging 2.1 internationally accepted standards, guidelines, and good 0.25 0.83  practices Scope is in accordance with international standards, guidelines, Emerging 2.2 0.50 0.42 or good practices  Classification systems are consistent with international Established 2.3 0.75 0.83 standards, guidelines, or good practices  6 ACCURACY AND RELIABILITY EMERGING:  The Accuracy and Reliability of Antigua’s EMIS data Figure 5. Accuracy and Reliability is EMERGING (0.48) (Figure 5). Antigua’s EMIS was only LATENT on one sub-component – timeliness – which indicates that a foundation for Accuracy and Reliability exists in Antigua. EMERGING    The foundation is firmly ESTABLISHED on three sub- components. Sound statistical methods are used in data transformations and adjustments, but they are not documented (Table, 5, 3.6). Also, statistical discrepancies in intermediate data are investigated most of the time (3.8) and systematic processes are in place for monitoring errors and omissions in statistical outputs, but the results are not made public (3.9). Antigua could improve on this Aspect by improving the timeliness of the data exchange, the comprehensiveness of the data collected, and the standardization of the scope and definitions of source data (3.1-3.3). Also, Antigua could validate intermediate results against other information (3.7) and conduct studies comparing preliminary data to revised data to analyze statistical methods and inform the process (3.10). Antigua & OECS Table 5. Accuracy and Reliability: Subcomponents Benchmark Barbuda Average Source data are obtained from comprehensive data collection that Emerging 3.1 0.50 0.58 takes into account country-specific conditions  Data are reasonably confined to the definitions, scope, classifications, Emerging 3.2 0.25 0.50 and time of recording required  Latent 3.3 Source data are timely (6 months after event) 0.00 0.46  Other data sources, such as censuses, surveys, and administrative Emerging 3.4 0.50 0.42 records, are routinely assessed  Data compilation employs sound statistical techniques to deal with Emerging 3.5 0.50 0.79 data sources  Other statistical procedures (data editing, transformations, and Established 3.6 0.75 0.63 analysis) employ sound statistical techniques  Intermediate results are validated against other information where Emerging 3.7 0.50 0.67 applicable  Statistical discrepancies in intermediate data are assessed and Established 3.8 0.75 0.92 investigated  Statistical discrepancies and other potential indicators or problems in Established 3.9 0.75 0.71 statistical outputs are investigated  Studies and analyses of revisions are carried out routinely and used Emerging 3.10 0.25 0.33 internally to inform the processes  7 SERVICEABILITY LATENT:  The Serviceability of Antigua’s EMIS was categorized as LATENT (0.29), but the score was only 0.02 away Figure 6. Serviceability in the OECS from earning an EMERGING benchmark. Antigua was MATURE in one sub-component – Periodicity – because the census of enrolment, teachers, schools, and financial data are produced annually (4.1). Still, with four LATENT scores, there are many areas that Antigua could improve. For example, Antigua could: LATENT Ü Make administrative school census data available 2   months after the initiation of the school year. Currently these data are not made available until 6 to 12 months later (4.2). Ü Check all data for consistency and regularly cross- check the data (4.3). Ü Increase the availability of time series data to more than 10 years. Less than five years are currently available (4.4). Ü Compare statistics to other data sources such as household surveys to evaluate consistency (4.5) Ü Establish a regular and transparent schedule or revisions (4.6) Ü Produce and clearly identify preliminary and revised data (4.7). Antigua was the only OECS country to receive a LATENT average score on any Aspect of Quality, but by taking some of the actions mentioned above, Antigua could significantly improve the score and avoid inconsistencies and errors that could potentially damage the credibility of EMIS. Antigua & OECS Table 6. Serviceability: Subcomponents Benchmark Barbuda Average Mature 4.1 Periodicity follows dissemination standards 1.00 0.96  Emerging 4.2 Timeliness follows international dissemination standards 0.50 0.63  Emerging 4.3 Statistics are consistent within the dataset 0.50 0.71  Statistics are consistent or reconcilable over a reasonable Latent 4.4 0.00 0.54 period of time  Statistics are consistent or reconcilable with those obtained Latent 4.5 0.00 0.33 through other data sources and/or statistical frameworks  Latent 4.6 Revisions follow a regular and transparent schedule 0.00 0.21  Latent 4.7 Preliminary and/or revised data are clearly identified 0.00 0.46  8 ACCESSIBILITY EMERGING:  The Accessibility of Antigua’s EMIS is EMERGING (0.47) but Antigua’s sub-component scores ranged from LATENT to MATURE. Antigua earned a MATURE score on data Figure 7. Accessibility in the OECS presentation: Statistics are clearly presented to users with tables and charts that show the underlying data and disaggregation (5.1). During the last two to four years, data were available electronically and there was a yearbook ready EMERGING for dissemination (5.2). Metadata is made available to users    upon request (5.6). Most of the time data are released to all users simultaneously (5.4), but there is no pre-announced schedule for data releases (5.3). Most releases identify a contact person in case of required assistance, but assistance is not monitored (5.8). Data catalogs (5.7) and catalogs of publications and other services (5.9) are not produced in Antigua. Without these catalogs, users are not able to clearly identify and request the information or services that they need. Accessibility is one of the key missions of an EMIS because it creates and maintains the public image of the EMIS and enables greater accountability. It is imperative for all levels of administration in Antigua to focus on developing a more accessible EMIS by establishing a schedule for data releases and producing catalogs, brochures, and manuals to give users better access to EMIS data, metadata, and assistance. Antigua & OECS Table 7. Accessibility: Subcomponents Benchmark Barbuda Average Statistics are presented to facilitate proper interpretation and Mature 5.1 1.00 0.96 comparisons (layout, clarity of texts, tables, and charts)  Established 5.2 Dissemination media and format are adequate 0.75 0.54  Emerging 5.3 Statistics are released on a pre-announced schedule 0.25 0.38  Established 5.4 Statistics are made available to all users at the same time 0.75 0.79  Emerging 5.5 Statistics not routinely disseminated are made available upon request 0.50 0.75  Documentation on concepts, scope, classifications, basis of recording, data sources, and statistical techniques is available, and differences Established 5.6 0.75 0.58 from internationally accepted standards, guidelines, or good practices  are annotated Latent 5.7 Levels of detail are adapted to the needs of the intended users 0.00 0.38  Emerging 5.8 Contact points for each subject field are publicized 0.25 0.38  Catalogs of publications and other services, including information on Latent 5.9 0.00 0.00 any charges, are widely available  9