Sitakhela Likusasa Impact Evaluation Evaluating the Effectiveness of Incentives to improve HIV Prevention Outcomes for Young Females in Eswatini Standard Operating Procedure - # 5 Learner admission and attendance data quality assurance (Education Incentives) Document 5 in a series of 20 Standard Operating Procedures Version date 10 May 2019 Status Final © International Bank for Reconstruction and Development / The World Bank 1818 H Street NW, Washington DC 20433 Internet: www.worldbank.org; Telephone: 202 473 1000 This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of the Executive Directors of The World Bank or other partner institutions or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Nothing herein shall constitute or be considered to be a limitation upon or waiver of the privileges and immunities of The World Bank, all of which are specifically reserved. Rights and Permissions This work is available under the Creative Commons Attribution 4.0 International (CC BY 4.0) unported license https://creativecommons.org/licenses/by/4.0/. Under the Creative Commons Attribution license, you are free to copy, distribute and adapt this work, including for commercial purposes, under the following conditions: Attribution – Please cite the work as follows: Sitakhela Likusasa Impact Evaluation: Standard Operating Procedure 0 – Introduction. Washington DC: World Bank. License: Creative Commons Attribution CC BY 4.0 Translations – If you create a translation of this work, please add the following disclaimer along with the attribution: This translation was not created by The World Bank and should not be considered an official World Bank translation. The World Bank shall not be liable for any content or error in its translation. All queries on rights and licenses should be addressed to the Office of the Publisher, The World Bank, 1818 H Street NW, Washington DC, 20433, USA; fax: 202-522-2625; email: pubrights@worldbank.org. Page ii Standard Operating Procedure - # 5 Learner admission and attendance data quality assurance (Education Incentives) NERCHA – National Emergency Research Council on HIV and AIDS authors: Muziwethu Nkambule and Tengetile Dlamini World Bank authors: Marelize Görgens and Wendy Heard IHM Southern Africa authors: Mthokozisi Dlamini, MINISTRY OF EDUCATION AND MINISTRY OF HEALTH TRAINING National Reference Laboratory, and Swaziland National AIDS Programme (SNAP) KINGDOM OF ESWATINI KINGDOM OF ESWATINI Main study implementation partner SGBV counselling and follow up For baseline survey For baseline survey Page iii [Type text] Table of Contents 1 Introduction to this Standard Operation Procedure (SOP) ........................................................... 1 2 Understanding the importance of quality learner data ................................................................. 1 3 Routine data quality assessment (RDQA)? ................................................................................. 1 Implementation phases for the learner admission and attendance data quality assessment ............... 3 Phase 1: .................................................................................................................................... 3 Phase 2: .................................................................................................................................... 5 Phase 3: .................................................................................................................................... 7 Phase 4: .................................................................................................................................... 8 4 References ................................................................................................................................. 9 Annex A:......................................................................................................................................... 10 Annex B: ......................................................................................................................................... 14 Table of Figures and Tables Figure 1: The three main components of a RQDA ............................................................................. 1 Figure 2: Tracking the flow of the learner attendance data, from data source to database ................... 2 Figure 3: Suggested framework for the final data quality audit report ................................................ 8 Figure 4: Conceptual framework for data quality assurance; data management and reporting systems within the Education Sector, functional areas and data quality.......................................................... 14 Figure 5: The six key stages of the data management cycle ............................................................. 16 Figure 6: Instructions from Admissions Register ............................................................................. 19 Figure 7: Instructions provided in the School Attendance Register .................................................. 21 Table 1: Dimensions of data quality ................................................................................................ 15 Table 2: Observations from June 2016 field visit ............................................................................ 24 1 Introduction to this Standard Operation Procedure (SOP) The introduction and critical background information relating to the Sitakela Likusasa Study is provided in SOP#0 (Introduction to the Sitakela Likusasa Study). This SOP aims to provide information with regards to: This SOP applies to the “in-school” categories of study participants, as school admission (enrolment) and attendance need to be monitored throughout the study in all arms. This SOP describes how the learner admission and attendance data can be assessed, on a limited scale, and verified and will point to areas to consider in order to improve the quality of the learner admission and attendance data and data collection systems. This SOP should be read in conjunction with:  SOP # 3: Education Incentives – Incentives for School Enrolment and Attendance  SOP # 6: Standard operating procedures for collecting school enrolment and school attendance information  Annex B to SOP # 5: The importance of ensuring data quality 2 Understanding the importance of quality learner data Data quality is a cornerstone of accountability in programs. Ensuring that the correct Study participants receive the incentives they qualify for, is dependent, to some extent, on reliable data on learner admission into school and learner attendance data. This SOP provides guidelines to conduct spot checks, using a data quality assessment approach, on (learner admission and attendance) data quality and provides feedback on possible areas for strengthening the systems to improve the quality of this data. The review of individual learner and class data at school level must be checked, for errors to be minimised. Regular reviews of the data are needed to validate the data and build trust in the resulting information to be generated for the system. It is recommended that a 10% random sample of schools be visited each school term , resulting in 30% of schools per annum, and school records be checked and confirmed against the study information captured. Errors, challenges and blockages should be identified and suggestions made to avoid these in future and areas identified to be addressed to strengthen the EMIS and in turn improve the data quality. 3 Routine data quality assessment (RDQA)? RQDA allows one to determine data quality at any given time and provides the opportunity to develop and implement strategies to address and prioritise gaps. The process consists of asking pointed questions on data quality and data management processes. Figure 1: The three main components of a RQDA Page 1 Data •Are data management systems and procedures in place to ensure data quality? managment review Data •Are the data being collected accuarate? verification assessment •Should there be problems with data management, how can we proceed? Develop a •What areas should be prioritised for improvement? data quality •Who is responsible for following through on the actions? action plan For the purpose of this SOP, the general RDQA will be adapted to focus on learner admission and learner attendance data, as this is the learner information that is required for the purposes of the Study. The purpose of the revised RDQA is to assess, on a limited scale, if schools and the Project data management team, are collecting, capturing and reporting data accurately and on time, and to double- check the reported results. To do this, a sample of schools will be drawn and these will be visited. During the site-visit to the sampled schools the data on learner admission and learner attendance will be checked, and f required a head count of learners carried out. Back at the project office, the data provided will then be traced to see if it has been correctly linked, recorded, captured and manipulated. Three types of data verification are proposed and they will be integrated into the audit processes; trace and verify, where the critical numbers will be checked against the original copy of the source document (school registration lists and copies of attendance registers submitted) and these are then compared with the reported numbers; Cross-checks, where the reported numbers are checked against other data sources (such as project, school and EMIS records); and Spot-checks, where visits will be carried out to targeted schools and recorded data (project records) checked with the actual learner numbers (head counts) and where appropriate other school records (Admission and attendance register). Since for the Study, no aggregated data is used, there is no need to provide for data quality checks at the aggregation levels. However, aspects of the data management cycle will be assessed and possible weaknesses identified. Figure 2: Tracking the flow of the learner attendance data, from data source to database Class 1 Class 1 Class11 Class School A Class 1 Attendance Register Class 1 Region 1 Class11 Class Class From 1 1 Attendance School B Register Class 1 Project Class11 Class Database School C Class Class 1 1 Attendance Register Class 1 Class 1 Class11 Class Form 1 Attendance Region 2 School D Register Class 1 Class Class1 School E Class11 Form 1 Attendance Register Details of individual learner attendance records are required to flow up to the Project Database, Page 2 not summaries prepared at different levels The RQDA will be guided by certain implementation phases, with specific steps within each phase. Implementation phases for the learner admission and attendance data quality assessment The implementation of the adjusted Routine Data Quality Assessment (RDQA) can be identified by the following phases: Review Preparation & School visits output & Completion Initiation conducted findings Preparation & Phase 1: Initiation The audit team will be required to lead on all the steps within this phase. Step 1 & 2 will require the assistance of the database manager. This phase occurs prior to the audit team going out to schools, and will be required to be completed every school term. It is suggested the audit is conducted within the 3rd month of the start of the new school term, as data from the previous term will be checked and the data management team need to have processed this data. Page 3 The steps within Phase 1 are: PHASE 1 1. Obtain an electronic listing of all the schools the Study participants Preparation are drawn from. Ensure the listing contains the minimum of the following and Initiation details:  EMIS Number 1. Obtain  Official School Name electronic  Region listing of  School contact details schools  Enrolment by grade and gender (year of data) 2. Confirm 2. Confirm and document the scope of the RQDA identifying what it is and document that is to be checked: the scope of  which reporting period to be included, the RQDA  what data and level of aggregation – attendance and admission, and 3. Draw a  size of the sample. random The purpose of the random selection sample to 3. With the school listing provided in sample is to produce quantitative estimates select 30% of step 1, using the statistical technique of data quality that can be viewed as the schools of random sampling and computer indicative of the quality of data in all applications such as MS Excel, schools not simply the selected schools. STATA, SPSS (or others), draw a sample of 30% of the schools, allowing 4. Obtain for replacement. Random Sampling offers the most powerful method for National and drawing inferences about data quality for a country or program. The selected Regional schools will be representative, i.e. the schools selected will be similar to the authorisation entire population of schools in terms of the attributes that affect data quality – & inform size and location. Regional Guidance 4. Obtain National and Regional authorisation for the impending site Officers visits and data quality assessment to be undertaken. Ensure that the relevant RQDAObtain Regional Guidance Officers are also informed, as they will play a critical role national and 5. Identify in the field work. The Regional Guidance Officers to also inform the selected and orientate schools of the planned visits, but should not share too much details, just team of field provide an indication of when the team will be visiting and that the Head workers Teacher/Principal is to be present. 6. Review 5. Constitute the field team, ensuring that each member understands the documentation role she or he needs to play and that all have received the necessary training & prepare and orientation. Each team member must be fully conversant with the required procedures and instruments to be used for the RDQA. summaries 6. Review the documentation previously provided by the schools 7. Prepare included in the sample, along with any related data analysis or information detailed plan products produced. For each school to be visited ensure the Field Team for site visits members have the following: & deal with  A summary of the learner attendance data that needs to be logistics verified. This would include for each of the learners participating in the Study, an indication of the class and stream the Study participant is in and the Page 4 total number of days they were absent for during the term under review. It is critical that the names of thee learners involved in the Study are NOT to be provided the Head Teacher, Regional Guidance Officer or anyone else outside the Study Team  Copies of the lists of the learners enrolled at the selected school. Calculate a total for each grade/class and stream (if provided) by gender. 7. Deal with all logistics required to prepare for the site visits and draw up a detailed plan for the site visits: including the dates and timing of the visits, assigning team members to the various schools. Each term 1/3 of the selected sample (10% of the school population) are to be visited, so that over the course of the year, the 30% identified as the sample will be covered. The 10% to be visited each term are to be randomly assigned to each of school trimesters. Select ion of School visits Phase 2: schoo ls to be are visite d conducted The field team leader will be required to drive all the steps within this phase. Members of the field work team will be critical in this phase. The school principal/head teacher for each of the selected schools will be involved. To be undertaken each term according to the plan devised in Phase 1 (Step 7). The steps within Phase 2 are: PHASE 2 1. Trace and record reported data Once at the school, after dealing with School visits protocol issues, request the school to provide the attendance registers for all are conducted grades and streams for the period under review, the admission register and copies of the current class lists (or lists of learners enrolled at the school by grade and stream). Ask to be allowed time (approximately 45 minutes) to 1. Trace and work with these documents on your own (as a Field Work team). In this time: record  Complete the relevant sections of the questionnaire (annex A), reported data to determine whether the key elements of reporting and recording learner admission and attendance are being implemented at the selected schools. This 2. Conduct a would include availability and use of the required registers, completeness, head-count of whether changes to enrolment are recorded in the attendance register and learners admission register, how errors are dealt with, level of detail, etc. Aspects of data quality (table 2 within the Annex) will be considered.  Look at lists of learners enrolled in the school, against class 3. Calculate lists, against names recorded in attendance registers and against the admission data quality list provided at the back of the attendance register; compare this with the data statistics previously provided for the school – ensuring that the same period of time are under review. Cross check the count of learners across the various data 4. Conclude sources, and calculate the differences. the school  Record any differences found and try and substantiate or visit explain the discrepancies. For example, o If a difference is found in the number of learners Page 5 enrolled in a grade – new learner/s may have enrolled or learner/s may have left (dropped out) the school. For this check to see if the attendance register (for the current term) reflects any learners leaving or being admitted. The admission register can also be checked for this information. o Should a difference be found in the data recorded for the learner attendance of Study participants – changes may have been made to the data or there may have been a transcription error. For this check the attendance register, to see if alterations have been made, redo the calculated totals and compare with the data captured in the data base. o Another possible reason for a discrepancy could be simple data entry or arithmetic errors in calculations. 2. Conduct a head count of learners, by following this process :  Request that access to the school for learners be controlled for the period of the count – no learners to enter or leave the premises.  Learners are requested to stay in the same classroom for the time the count is conducted – preferably with teacher supervision (for secondary schools they will be required to move into their register class groups). In the classroom ask the boys and girls to sit separately, and to stay seated in the place assigned to them. No “breaks” to be taken. Learners may continue with work while the field team moves through the group.  Two members of the Field Work team then move through each class and count and separately record the number of boys and girls counted for each class and stream. If their counts agree they move to the next classroom. If their counts do not agree they are to repeat the count until their numbers recorded by sex tally.  For each class visited the Field Work team embers need to also count and record the number of learners absent, teachers and learners can be asked who is absent or “missing” on that day.  This process is repeated till all learners are counted. The numbers can then be compared with the information provided in the attendance register.  Differences are to be calculated and recorded for each stream and grade. 3. Enter the required key data into a spreadsheet or calculate the required ratios and percentages: a. Result verification ratio for learner enrolment (school) = verified count of learner enrolment of school reported count of enrolment of school b. Result verification ratio for learner enrolment (grade) = verified count of learner enrolment of grade reported count of enrolment of grade c. Result verification ratio for learner attendance (Study participants) = verified count of learner attendance of Study partipants reported count of elearner attendance of Study partipants d. Percent difference for cross checks in learner attendance (Study participants) = 100 1 Page 6 4. Provide some brief feedback to the Head Teacher and thank all involved at the school for their assistance and willingness to cooperate with the activities that had to be undertaken and the sharing of the schools documents and records. Indicate to the Principal/Head Teacher that a brief report on issues of data quality and system compliance will be prepared and that this will be shared with the school. Ensure that the Field Team has all the required documentation before leaving the school, and that all aspects of the questionnaire are completed. Review output & Phase 3: findings The Field Work team will be required to drive all the steps within this phase. This phase occurs when the Field Work team return from the school visits, every school term. It is suggested that the review is conducted as soon as the team is back in from fieldwork, or when a batch of visits are completed, should the visits be spread over time. The steps within Phase 3 are: PHASE 3 1. Ensure that all aspects of the questionnaire are complete. Use the Review notes made and the analysis of findings to draft a report. Review the graphs output and generated by the questionnaire tool, these should also be included in the findings report. 1. Draft The report should list the findings and then link these to recommendations to findings and address the issues of concern. The findings should stress the positive aspects recommend- of the school reporting system and the data management, as well as the dations weaknesses identified by the Field Work team. Remember the main purpose of the RQDA is to improve data quality, so focus on this with the 2. Close out recommendations. All findings should therefore point to this and provide meeting innovative controls and effective steps to ensure that data are collected consistently and reliably in order to comply with the components of quality data (table 2 within Annex B). The recommendations need to strengthen the design and implementation of school record keeping and data management of these critical records. All findings must be backed by documentary evidence. In the recommendation notes, cite the evidence that points to a threat to data quality. One or more recommended action should be provided to address this or avoid it in future. Some of the recommended actions could be time referenced. For example: Supervision checks : The lack of regular supervisory checks of the attendance register data could lead to potential errors in calculations and the aggregation of data. Recommendation/s: The school to appoint a senior teacher, per grade (or for smaller school per phase), to check the attendance registers across all streams for that particular grade on a weekly basis, before submitting them to Head Teacher. 2. Once all the school site visits are completed, for a particular cycle, the Field Work Team Leader should conduct a close out meeting with the data manager, those involved in M&E and other Page 7 stakeholders to share the results and present the preliminary findings and recommendations. Together the team could also discuss further steps to improve data quality. The agreements reached at the close out meeting, and further recommendations made or suggested changes to the preliminary findings and recommendations, must be documented. Phase 4: Completion The Field Work team leader will be required to drive all the steps within this phase. This phase occurs when the Field Work team have completed a cycle of school visits, and conducted the close-out meeting for Phase 3. The steps within Phase 4 are: 1. To draft the final report of the RDQA processes, including the PHASE 4 findings and recommendations for addressing the gaps and weakness Completion highlighted through the audit. It is at this stage that the recommended changes accepted at the close-out meting be made to the report. 1. Draft final A suggested framework for the audit report is provided below. RDQA report 2. Disseminate and share the report with relevant groups and 2. Share the communicate the findings. When the findings are shared it is recommended RDQA report that together an action plan to address the identified gaps should be developed. & develop The action plan should be clear and include what action needs to be take, time action plan frames – start and end date, who will take the lead and what measures will be put in place to verify or measure that the action has been completed. 3. Initiate follow up of It is suggested that when the results of the RQDA are shared and disseminated, recommended stakeholders are given the opportunity to provide comments. These can be actions integrated into the report. 3. As appropriate, initiate follow up procedures to ensure the agreed changes are effected. It is suggested that the identified recommended actions be categorised as either minor data quality issues or major data quality issues. Different timeframes are to be determined to address the two categories –perhaps 6 months for the minor issues and a year for the major issues. If subsequent visits are made to the schools, during this period, it is suggested that follow up in made on actions the schools were required to complete. For major issues it is recommended that the person/team tasked with implementing the required recommendations or changes should report on a quarterly basis, so that progress or hindrances could be monitored. Figure 3: Suggested framework for the final data quality audit report Page 8 I. Executive Summary II. Introduction & Background a. Project Background b. Purpose of the RQDA c. Scope of the RQDA d. Selection of schools e. Description of the data collection and reporting system III. Assessment of school records and data management system a. Describe steps taken in RDQA b. Summary statistics c. Key findings, identified strengths and weaknesses IV. Verification of data on learner admission and learner attendance a. Description of the data verification steps b. Assessment of data quality c. Key findings, identified strengths and weaknesses V. Table of recommendations and areas for suggested improvement VI. Responses to the RQDA Source: Adapted from the Measure (2008) Data quality audit tool – guidelines for implementation 4 References 1. Education Foundation (2001) School Records Management 2. Education Foundation (2002) EMIS Improvement Project – Learner Information Module 3. MEASURE (2008) Data Quality Audit Tool – Guidelines for implementation 4. NESIS (1995) School data collection training Manual 5. PACT (2005) Building Monitoring, Evaluation and Reporting Systems for HIV/AIDS Programs 6. PACT (2014) Field Guide for Data Quality Management -Monitoring, Evaluation, Results and Learning Series Publications (Module 2) 7. SADC (2013) SADC EMIS Norms and Standards Peer Review Assessment Report - Eswatini 8. Eswatini Ministry of Education (u.d.) Admissions Register 9. Eswatini Ministry of Education (u.d.) Attendance and Mark Register (E.A.5) 10. Eswatini Ministry of Education (1988) A guide to school regulations and procedures – including the Education Rules of 1977 11. The Global Fund to fight AIDS, Tuberculosis and Malaria, PEPFAR, USAID. WHO, UNAIDS, MEASURE Evaluation (2008) Routine Data Quality Assessment Tool (RDQA) – Guidelines for implementation for HIV, TB & Malaria Program 12. UNESCO (2013) Monitoring and evaluation of the education sector response to HIV and AIDS – guidelines for the construction and use of core indicators Page 9 Annex A: Questionnaire to be used for the RQDA Sitakhela Likusasa Impact Evaluation Spot Check Questionnaire Learner Admission and Attendance Data Quality Assessment School Name Region EMIS No Reporting period being verified Start date End Date From EMIS Data: Year of Data _____________________ Learner Numbers Class/Std 1 Class/Std 2 Class/Std 3 Class/Std 4 Class/Std 5 Class 6 Class 7 Sub total Total Boys Girls Total From School Submission to the Project: Date of Data _____________________ Learner Numbers Class/Std 1 Class/Std 2 Class/Std 3 Class/Std 4 Class/Std 5 Class 6 Class 7 Sub total Total Boys Girls Total In the table above (School reported learner numbers), with an * indicate the class/Std which has study participants. Indicate number of study participants that attend this school: _________________ School Visit Date Team Leader RGO Remember to: Confirm the number of class groups/streams per grade/std , the enrolment by grade and gender, whether any learners have left the school during the course of the year (give details, name, stream and date left) and if there are any new entrants (during the year) after admissions were closed(give details, name, stream and date learner joined the school) Request the school to provide access to: The Admission Register Attendance register for all classes/streams Copy of current school class lists for all classes/streams A space where the field team can work with the school records (Privately) Indicate that a head count of learners may be require Learner numbers & number of class groups/streams indicated by school: Class/Std 1 Class/Std 2 Class/Std 3 Class/Std 4 Class/Std 5 Class 6 Class 7 Sub total Total Boys Girls Total Streams Number of learners who left the school (during the course of the year) Boys Girls Total Number of new entrants - learners who joined the school after formal admission at the start of the year Boys Girls Total Page 10 SCHOOL ADMISSION REGISTER: (Tick appropriate option) Are all learners admitted to school (new entrants) entered into the admission register? Yes No Are all the required details provided for the new entrants (no missing information) Yes No Are admission numbers assigned to all the entries in the register? Yes No Is the register up to date? (check against the class lists provided, especially for the first class/std offered) Yes No Does the register indicate when learners leave the school during the year? Yes No Should errors or omissions be found – provide possible explanation or reason for the differences found Describe error or omission Account for possible difference SCHOOL CLASS LISTS No of class lists checked during the visit: Class/Std 1 Class/Std 2 Class/Std 3 Class/Std 4 Class/Std 5 Class 6 Class 7 Total No of class lists where number of learners listed match the enrolment figures Account for possible differences provided by EMIS (only complete if the year of data is the same) provided by school (on arrival) provided by school (for earlier project request) indicated in the class register indicated in the summary at the rear of the class register Tick appropriate option For the entry class/std, are all learners admitted entered on the class list? (against admission register) Yes No Are all the required details provided for the entries (no missing information) Yes No Was the class list created specifically for the project or is it a routine list used by the school Created Routine If routine, is the class list up to date? (new entrants and school leavers included) Yes No Should errors or omissions be found – provide possible explanation or reason for the differences found Describe error or omission Account for possible difference SCHOOL ATTENDANCE REGISTER: No of attendance registers checked during the visit: Class/Std 1 Class/Std 2 Class/Std 3 Class/Std 4 Class/Std 5 Class 6 Class 7 Total For each of the following indicate the number of the attendance registers checked that match the criteria : Names fully match class list Admission numbers (attendance register are cross referenced to attendance register) Absence of learners is indicated correctly (codes are used for absenteeism) Evidence that the register has been checked by a colleague Register is up to date (allow for 1 day not yet completed) Register indicates when learners leave (including date) Number of school days are reported correctly Totals/summary calculations are provided for the end of the week Totals/summary calculations provided are calculated accurately Totals/summary calculations are provided for the end of the term Totals/summary calculations provided for the end of term are calculated accurately If end of the year, Totals/summary calculations are provided for the end of the year If end of the year, Totals/summary calculations provided for the end of the year are calculated correctly At the back, the admission list is provided accurately, including the admission number If end of the year, Totals/summary calculations (at the back) are provided for the end of the year If end of the year, Totals/summary calculations (at the back) provided are transferred correctly Page 11 Should errors or omissions be found – provide possible explanation or reason for the differences found Describe error or omission Account for possible difference DETAILS FOR STUDY PARTICIPANTS Number of study participants enrolled at this school Number of classes/standards that the study participants are enrolled in Number of study particpants that appear on the class lists Number of study particpants that appear in the class register Calculate the number of days the identified study participant/s were absent for – term by term Number of study participants that have absent days reflected inaccurately in the register Number of study participants that have absent days reflected inaccurately in the project database Should errors or omissions be found – provide possible explanation or reason for the differences found Describe error or omission Account for possible difference HEAD COUNT Where differences in the reported number of learners are found between – Class list, attendance register & project records a head count is to be conducted across all learners. The count of learners by class stream/group and gender is to be undertaken, separately, by two field workers. Count to be repeated until the numbers recorded tally. The following tally sheet can be used for each class group/stream to be head counted. Boys Girls Learner number on class list (a) Class/Standard Learner number in register (b) Number of learners marked present (c) Group/stream identified Number of learners in admission list (back of register) (d) Learner count by field workers (FW) FW 1 – 1st FW 2 – 1st FW 1 – 2nd FW 2 – 2nd FW 1 – 3rd FW 2 – 3rd Agreed Count Count Count Count Count Count total Boys Girls Calculated Difference (Compared against the head count) (e – a) Register – only (e – c) (e – d) Compared against Head Class Admission those present for the head count Count (e) list (a) Value As % Value As % list (d) Value As % the day (c) Boys Girls Should errors be found – provide possible explanation or reason for the differences found Describe error Account for possible difference Page 12 CALCULATION OF DATA QUALITY RATIOS Use the data collected to make the following calculations: a. Result verification ratio for learner enrolment (school) = verified count of learner enrolment of school reported count of enrolment of school b. Result verification ratio for learner enrolment (class/std) = verified count of learner enrolment of class/std reported count of enrolment of class/std c. Result verification ratio for learner attendance (Study participants) = verified count of learner attendance of Study partipants reported count of learner attendance of Study partipants d. Percent difference for cross checks in learner attendance (Study participants) = verified count of learner attendance of Study partipants 100 x reported count of learner attendance of Study partipants 1 FEEDBACK TO HEAD TEACHER As a field work team, reflect on the findings (especially the tables that capture the errors and omissions and possible reasons for the differences) and agree on the items to report back to Head Teacher on, and suggestions to be made to improve learner attendance and admission data. SCHOOLS RESPONSE Fieldworker to capture the response of the school when feedback is provided Page 13 Annex B: Understanding the importance of quality learner data & processes in Eswatini Data quality is a cornerstone of accountability in programs. Ensuring that the correct Study participants receive the incentives they qualify for, is dependent, to some extent, on reliable data on learner admission into school and learner attendance data. Having poor or unreliable data creates long-term cots and unforeseen effects compared with the costs and benefits of having good data. Practical and affordable strategies exist for generating timely and reliable data on learners, but appropriate investment is needed to develop the capacity to collect, manage, analyse, disseminate and use information. In general the quality of reported data is dependent on the underlying data management and reporting systems, or for the Education sector EMIS (Education Management Information System); stronger systems should produce better quality or more reliable data. In order to produce learner data of good quality, school recording and reporting systems and data flows need to be functional, dependable and consistent. The various points at which this data is then aggregated also need to reliable and operational: the regional and central (head office) level. When assuring the quality of learner data it is important to (1) verify the quality of the data, (2) assess the system that produces that data, and (3) develop plans to improve both if required. 1 Figure 4: Conceptual framework for data quality assurance; data management and reporting systems within the Education Sector, functional areas and data quality Dimensions of Quality Quality Accuracy, Completeness, Reliability, Data Timeliness, Confidentiality, Precision, Integrity (and Validity & Ethics) Functional components of EMIS needed to ensure Data Quality (Annex A) EMIS / Data management i. EMIS capabilities, roles and responsibilities and reporting system Central EMIS Unit Reporting levels ii. Training iii. Data reporting requirements Intermediate aggregation iv. Indicator & data definitions level (e.g. Regions) v. Data collection and reporting tools vi. Data management processes vii. Data quality mechanisms and controls School Records viii. Links with national reporting system Source: Adapted from the GFTAM (2008) Routine data quality assessment tool In order to ensure data quality, reviews are to be conducted at all levels. As all points of data collection and EMIS processes are disposed to errors resulting from data collection, processing and transmission, it is of utmost important to have data quality reviews. Concepts of data quality, data management and data assessment What do we understand as data quality? Data quality makes reference to the worth of accuracy of the data collected. Data quality emphasizes the high standard required of data capture, verification and analysis. Ensuring high quality data is important and requires a structured and purposeful approach at each step along the way. Issues relating to data quality need to be thought through to ensure that suitable quality standards are maintained. The process of checking data quality is referred to as routine data quality assessment 1 These steps are identified in the GFATM: Routine Data Quality Assessment Tool Guidelines (2008) Page 14 (RDQA) or a data quality audit. RDQAs help identify where data quality is poor and should point to possible solutions. Criteria often used to assess data quality Data quality is commonly assessed according to five key criteria: 1) accuracy or validity, 2) reliability, 3) integrity, 4) precision and 5) timeliness. Others that are added include: 6) completeness, 7) confidentiality and 8) ethics. Table 1: Dimensions of data quality Criterion Explanation or operational definition Pointers for data quality issues Accuracy or Also known as validity. Accurate data are Data validity issues could result from: Validity considered correct: the data measure what  Definitional issues for data being they are intended to measure. Accurate data collected minimize errors (e.g., recording or  Respondents having trouble interviewer bias, transcription error, understanding the questions asked of sampling error) to a point of being them negligible.  Data is incomplete or illegible  Data altered in transcriptions  Sampling or representation errors  Respondents under pressure to answer ‘correctly’ or provide the information Reliability For a data set to be reliable, data collection Data reliability issues could result from: processes must be stable and consistent over  Different tools used to collect the same time, with reliable internal quality controls in data place and data procedures handled in a  Recording inconsistency between staff transparent manner. members providing data  Instances of wrong or missing data not reported or considered Integrity Data have integrity when the system used to Data integrity issues could result from: generate them is protected from deliberate  Someone has tried to bias or influence bias or manipulation for political, personal or the outcomes of the data other reasons.  Unreasonable time pressures are placed For a data set to have integrity, the data must on data collection and collation be accurate and free of error introduced by either human or technological means, either wilfully or unconsciously. Precision This means that the data have sufficient Data precision issues could result from: detail. For example, learner attendance  Is data aggregated without peer review requires the number of individuals who  Is only aggregated data collected – attended school, by grade/form and sex of despite efforts to collect disaggregated the individual. An information system lacks data precision if it is not designed to record the sex of the individual. Timeliness Data are timely when collected frequently Timeliness issues could result from: enough and they are up-to-date (current).  Decisions made without data – due to Timeliness is affected by: (1) the rate at delays or unavailability which the program’s information system is  Data being out of date to be relevant or updated; (2) the rate of change of actual of value program activities; and (3) when the information is actually used or required. Source; Adapted from definitions provided in PACT (2014) and GFATM (2008) Page 15 While the five criterion discussed in the table above are considered as the foundation for under- standing data quality , the following additional factors are at times also considered: Completeness: means that an information system from which the results are derived is appropriately inclusive: it represents the complete list of eligible persons or units and not just a fraction or part of the list. No data is missing, no responses incomplete or due to other data quality issues, found to be unusable. (Euphoria!) Confidentiality: means that clients are assured that their data will be maintained according to national and/or international standards for data. This means that personal data are not disclosed inappropriately, and that data in hard copy and electronic form are treated with appropriate levels of security (e.g. kept in locked cabinets and in password protected files). Ethics: are the rules or standards governing the conduct of a person collecting, collating, reporting on, or utilizing data, and represent our sta ndard of what’s “right”. Ethical issues include concerns of informed consent, the protection of privacy and confidentiality and misrepresentation or falsification of data. What is data management? Data management considers how data moves along the track; from controlling the data collection process; to how data are brought together and analysed; determining the most appropriate levels of aggregation, presentation and dissemination format; and ensuring data utilization. Figure 5: The six key stages of the data management cycle Data source Data Data utilisation collection Data Data reporting collation Data analysis To gauge the quality of a given data set, you need to understand exactly where the data originated and the strengths and weakness of the data at that source. Primary data sources present the least risk to potential errors as the data is raw, un-manipulated and first-hand. To ensure data quality and permit multisite data comparisons, the data collection effort usually needs to be structured so that all data are collected the same way from one data collection site to the next. Credibility of the process is also very important in the data collection stage. Selecting appropriate tools and methods, training data recorders, having clear instructions on how to collect data and how to capture information, and checking the accuracy of the data generated are all part of a quality data collection process. Ensuring that data collection tools have data entry “checks” where possible will help to keep data that are collected clean. Checks can include random spot checks undertaken by a supervisor. Page 16 Data collation is the process of assembling data into a format for the purpose of analysis using either electronic or manual tools. Collation may involve data coding to make it easier to manipulate the data. Collation also refers to the process of consolidating information from various sites or offices and building completed data sets. All these processes introduce an opportunity for mistakes to be made and errors to be introduced into the data. Thus, careful management of the collation process is critical to maintaining data quality. Data analysis is the process of examining, reviewing, and evaluating data sets to assess a hypothesis. Analysis enables data users to test underlying hypotheses or assumptions and to predict relationships in order to understand and evaluate their programs. The data analysis period is also an important time for a data quality review. Although the primary purpose of data analysis is to advance understanding and gain insights into the program, the analysis process also provides—from a data quality viewpoint—valuable feedback on the adequacy and completeness of our data and its relevance, validity, and precision. Reporting involves compilation of descriptive information, presenting data analysis as useful knowledge. Key to the reporting process is to ensure that data presented are relevant to the different target audiences and that the report has integrity—in other words, that it accurately presents the data set and results. From a data quality perspective, accurate presentation of the findings—without an overzealous spin overstating the results or purposive exclusion of information to mislead an audience. Data usage refers to the process of making timely, data-driven decisions. For this to happen efficiently, relevant high-quality data must be made accessible in a timely manner to key decision makers. This means that decision makers (including project teams) must know of the existence of the data set, can locate it, and can easily import it into their working environment. Usually, the data are most useful to decision makers after analysis and synthesis into an understandable, relevant report. Source; Adapted from PACT (2014) Filed Guide for data quality management – monitoring, evaluation, results and learning seried publication (module 2) Identified constraints of learner admission and attendance data and systems within Eswatini Ministry of Education Schools are required to have information about their learners if they are to make appropriate decisions about resources and educational support programme. School Management Teams require this information for school based planning, administration, governance and self-evaluation. From the time a child enters schools, records begin to follow the new learner. Besides the information provided by the parent or guardian, the school itself creates additional information that describes the learner’s involvement in education, extracurricular activities and other relevant experiences. This information is all contained within the school records system. In addition to schools requiring information, learner information for management and planning functions are also required at Regional and Head Office level, to ensure that national policies have been successfully implemented. Learner numbers remain the biggest cost driver for the Education Sector, so the quality of learner information is critical. The Eswatini Ministry of Education has well-structured processes in place to guide schools on collecting, tracking and reporting on learner admission and attendance at public schools. There are two critical documents, both are standardised for all public schools, both provided to schools by the Page 17 government at no cost and both have clear guidelines and instructions: the admissions register and the class attendance register. The Admissions Register This is a permanent record in which details of every learner admitted to a Schools tend to differ on school is recorded. The Register is designed as a comprehensive listing of understanding when a learner is “admitted”. the schools former and current learners and is necessary to keep track of school enrolment. The source of information entered into the admissions register is obtained through the application process – either through the completion of application forms (which are school specific), letters for admission, transfer cards or learners (along with a parent, guardian or sibling) presenting themselves at an interview. The intended purpose of the learner Admissions Register is to:  Record all learners admitted to a school by entering summary details of learners who have been accepted for enrolment.  Provide an overview of the school enrolment by individual leaner  Show leaner movement from the first enrolment to the time they leave the school  Assist in developing the school enrolment or class lists.  Assign a unique admission number to each learner. This number should never change, while the learner is at that school. In the Eswatini context, the admissions register is a bulky, hard covered, bound book, issued by the Ministry of Education. The cover contains clear instructions on how the Admissions Register is to be created and maintained. Page 18 Figure 6: Instructions from Admissions Register Source: Eswatini Ministry of Education Admissions Register Page 19 The instructions provided in the Admissions Register, as well as the “Guide to school regulations and procedures” indicate that school leavers should be reflected in the Admissions Register as soon as the learner leaves the school. Once Admissions are concluded and the Admissions Register finalised, the intended next step is that from the attendance register the class lists are created (or updated for subsequent grades), and once these are finalised the details are transcribed into the Attendance Register, both to track learner attendance, and at the back of the Attendance Register to record admissions into the specific class for that year. The Attendance Register The Attendance Register provides a daily record of the attendance of each learner in each class in the school. The Attendance Register is based on the class list, prepared from the entries in the Admissions register and the learner records of the previous years (failures and promotions). The names of all the learners should be entered alphabetically, according to Surname, and by gender. The admission number is an important cross reference between the Admissions Register and the Attendance Register, and must be entered into the Attendance Register for each learner. The intended purpose of the attendance register is to:  Provide the class teacher or school management with information about an individual learner’s attendance or absence at a glance  Determine the average class attendance  Identify those leaners who need attention due to irregular school attendance  Show the size and composition of a class group Schools tend to not always indicate the Non-attendance, or absenteeism, is recorded on a daily basis and reason for absenteeism. A learner is also marked present if s/he only should be differentiated between those learners who are absent presents her/himself for the due to illness (marked with an S), those absent with leave (school registration period or time when the informed or reason for absenteeism) (marked with an L), and attendance register is completed. . those absent (marked with an X). The class teacher and school management then follow up on irregular school attendance. If the learner leaves the school, the learner is to be struck off the Attendance Register indicating “left, the date and reason”. At the end of each week, summaries of learner attendance are calculated for the week and the Attendance Register is submitted to School Management for checking. The checks are basically for compliance to ensure that the registers are being completed diligently. It is also at this point that Management may be alerted to irregular attendance. At the end of each term, the class teacher is required to complete a summary of attendance and this is completed at the back of the Attendance Register. In some schools school management teams collate these summary tables for all grades, to get a complete and comparative picture of learner attendance across all classes and grades. At the end of the year, the final assessment marks attained by the learner is completed in the final exam results schedule, provided in the Attendance Register, and there is an indication if the child is promoted or has failed the grade. The Admission list, within the Register, also has to be updated to include the total number of days attended in the year, for each learner. Page 20 In the Eswatini context, the Attendance Register is combined with the mark register in a large (A3+) book issued by the Ministry of Education. The cover contains clear instructions on how the Attendance Register is to be completed. Figure 7: Instructions provided in the School Attendance Register Source: Eswatini Ministry of Education Attendance and Mark Register Different processes observed in capturing learner attendance at Primary and Secondary Schools, during the June 2016 field visit. Given that in general, in primary schools, a class teacher is assigned to a group of learners who then teaches most of the subjects to this class, i.e. the class stays with the same teacher for the majority of the day, the daily attendance register is completed by the nominated class teacher. Attendance is generally recorded directly into the Attendance Register during the first few sessions of the school day. However in Secondary schools, given that the learners move between teachers and according to a structured timetable that often does not follow a week, different systems are introduced to ensure that the attendance register is completed. One system allows for a class to be assigned to a registration teacher, who may or may not teach the learners. The time timetable then allows for a registration period at the start of each day, or after assembly, and the registration teacher is then requi red to complete the attendance register and deal with administrative issues in the time assigned for ‘registration”. Another system observed includes the use of class monitors, who are provided with a class list of learners. With this system, the class monitors mark off learners who are present or absent, and generally this is completed for every lesson during the day, so one can track learner attendance throughout a day. Some schools who use this system, request the teacher to sign off against the list at the end of the teaching period. The class list is then either returned to a nominated class teacher or straight to the office. Even with this class monitor system, the assigned class registration teacher is required to track down the class within the first few periods in order to be able to complete the attendance register. It is only at the end of the week, in some schools that the daily registers completed by the class monitors are compared with the formal attendance register completed by the registration teacher. This is not done consistently across schools. Some schools indicated, should they use the system where class monitors complete the register, unannounced spot checks will be conducted by senior teachers, who at odd times of the day, during the term will call for the class list tracking learner attendance. They will then conduct their own check of learners who are in attendance and compare this with the class monitors list. It appears that schools tend to use the class monitor system to try and track, or discourage bunking off or truancy rather than strengthening the attendance register data. Page 21 EMIS and national statistics on admissions and attendance Like many countries in the Region, EMIS within Eswatini, relies on an annual school census to collect the required data from schools to collate national statistics and provide information to support planning, management, reporting and policy implementation and tracking. Currently the EMIS Annual School Census form makes allowance for tracking school admissions (new entrants), school leavers (drop outs) and school enrolment, by gender, grade and age. The current EMIS Annual School Census makes no provision for reporting on learner attendance. When the Head of EMIS was interviewed, he indicated that an earlier version of the EMIS Annual School Census form included tracking absenteeism due to reason, but this was dropped as the data was found to be unreliable. It was indicated that no effort had been made to improve or adjust school records to provide this data and no training had been provided. Currently the Central Statistics Bureau, through their household surveys collects information on school attendance to be able to report on the Net Attendance Rate. This information is obtained through a question included in the survey that asked the head of the household whether the children in the home attend school. There are no follow up questions on the frequency of school attendance, or any way in which this information is confirmed or validated. There are new developments that may put additional pressure on the EMIS and school record systems, to provide more detailed and accurate information on learner absenteeism and learner enrolment, and improve learner based record keeping: The adoption of UNESCO Within this M&E framework, countries are going to be required to Indicators for the monitoring report on school attendance amongst orphaned learners. While the and evaluation of the education framework indicates that this data will be collected through sector response to HIV and population based surveys, it may with time be transferred to the AIDS Ministry of Education to provide this data. As part of their admission processes and school records, schools will be required to be able to report separately on orphans, according to age category. Eswatini has committed to implement this framework, and preparatory work has already begun. The adoption of the SADC The SADC EMIS Norms and Standards code contains 17 EMIS Norms and Standards minimum norms and standards covering policy and legal frameworks, resource availability and utilization, statistical processes and education information reports. SADC Ministers of Education adopted this code at their annual meeting in Kinshasa, Democratic Republic of Congo, in March 2010 and by doing so, committed themselves to adhere to these norms and standards, thereby improving their EMIS. These norms and standards are directed at improving data quality and strengthening EMIS. Eswatini has already undertaken the peer review assessment and has started working towards implementing the recommendations of the peer review. National introduction of SAMS The Eswatini Ministry of Education has negotiated with the South - School Administration African Government to implement the SA-SAMS system in Page 22 Management System Eswatini. The MOU is about to be signed (June 2016). SAMS allows for the computerisation of all aspects of school administration, management and governance. The programme has a strong EMIS focus to assist schools in the completion of the Annual School Census. The system is based on the creation of individual learner records, which follows the learner throughout her/his career within the education sector, which could be from early childhood development through to tertiary studies. The introduction of SAMS in Eswatini will required that detailed, individual learner records are created and maintained by schools. Learner attendance is also considered to be a critical component of SAMS and schools are likely to be required to maintain learner attendance records on SAMS. Schools moving towards Some schools we visited in Eswatini (Field trip June 2016) had computerized administrative made the move to a computerised administrative system. In doing systems this, they felt that the paper based records – School Admission Register and School Attendance and Mark Register – were outdated and duplicated the work they did on the computer system. At one school the Admission Register has been totally disregarded, since all learner registration is done within the computer software, however the paper based school Attendance Registers were maintained, but on a weekly basis this information was captured into the computer system, against each individual learner record. A distinct advantage of the computerised system for the Study is that when registering for the Study, learners could be asked to produce their most recent automated school report. The school report, generated by the computer system, could be used to validate the learner’s enrolment at a school, but is sure to also include a data field that captures the number of school days missed due to absenteeism. Issues that may impact on data quality As previously indicated, the Eswatini Ministry of Education has well-structured processes in place to guide schools on collecting, tracking and reporting on learner admission and attendance at public schools. The admissions register and the class attendance register are both standardised for all public schools, are provided to schools by the government at no cost and have clear guidelines and instructions. In addition the “Guide to School regulations and Procedures” read together with the Education Rules of 1977, provide a clear legal framework regarding learner admissions and maintaining attendance registers. If the school followed these regulations and instructions provided, to the letter, there would be no qualms about the quality of learner admission and attendance data at public schools. However, over the years the monitoring of these regulations and instructions provided have slipped and with that cracks have been created that do impact on the quality of learner admission and attendance data. The table below points to some of the issues that are considered to possibly impact on learner admission and attendance data quality. Page 23 Table 2: Observations from June 2016 field visit With the Admissions Register, schools tend to differ on understanding when a learner is “admitted”. Some wait to complete the admissions register well into the first term, when attendance has stabilized and learners attend regularly, while others will complete the attendance register as a learner apply to the school. Some of these learners may then not Observations attend the specific school as they have applied to several schools, to ensure a place is from the Field secured. While the school regulations indicate that when a learner leaves a school, for any reason, this must be recorded immediately in the Admissions Register, this is rarely done. None of the Admission Registers that we observed during our school visits (June 2016) were up to date in recording school leavers, and in some instances even late learner admissions (transfers in throughout the year) were not recorded. While the instructions in the Attendance Register are also clear about when and how to record when a learner leaves the school, this varied across the schools visited. Some wait 6 days, others wait 14 days, others 2 weeks, while others wait till the parent/guardian reports to the Office. While this affects learner enrolment data it also affects the calculation of the weekly and termly summary attendance data and indicators the schools need to calculate. Schools tend to not always indicate the reason for absenteeism. It was indicated it is difficult to tell this as they are not always informed, or the parent reports to the office but the class teacher is required to complete the learner attendance register. A learner is marked present if s/he only presents her/himself for the registration period, or time when the attendance register is completed. The inverse applies, if a learner arrives just after registration has been completed (e.g. they are late on the day) they are marked as absent. Learner attendance appears to be determined by being present when the learner attendance register is completed. Some schools make reference to temporary admission registers, class lists and attendance registers. Often these are used for the first few weeks of the new year, until learner enrolment has settled down. The duplication and double reporting system may result in errors creeping in. It is apparent that schools find little value in the Admission Register. Data is not complete and after the class lists are compiled the Admission Register is not updated until the new entrants are issued the next year. Schools report they only use the Admission Register in order to allocate a registration number to learners. Often learners who joint the school Observations mid-year are not even entered into the Admission register. from the Field Schools indicated that the Admission Register is never check by officials visiting the school. At times the Attendance register is checked, or when a specific case regarding a learner is investigated. The school may be asked to produce the Attendance Register to check the attendance record of the learner being investigated. One school shared a good practice, to help improve the quality of the data. Senior teaches were requested to check the registers of teachers within the grade, during the course of the week, before these were submitted to the principal for control. Senior teachers would do spot check to ensure that those who were absent were marked as absent, and also were required to check the calculations and summaries prepared by the teachers in the grade. The principals indicated this did help control the quality of the data and also ensured that registers were completed on a daily basis, as teachers did not know when senior teachers would check the register. The summary calculations that are required at the end of the week and term, in order to close the Attendance Register are confusing and misleading. It was also indicated that teachers do not know why it is required as it is not used by anyone. None of the schools visited indicated that they provided training or orientation for teachers on how to complete the registers. It is assumed they are taught while at college. Teachers Page 24 confirmed they are taught while at College, however they felt the instructions are clear. Teachers who attend university are not taught how to complete registers, they merely ask colleagues to help and interpret the instructions. One newly appointed Head Teacher (1 month in) interviewed had no knowledge of what was required of her in terms of checking on registers or completing the Admissions Register. The older, more experienced principals indicated, when they check the registers on a weekly basis they provide feedback to young teachers, if errors are picked up. It appears that class lists are prepared in a very ad hoc manner. This is disturbing given the importance given to the class list – as this appears to become the official record of learners admitted or registered at a school. One school reported that learners (new entrants at a primary school) were counted off into three groups and then the teacher asked the learner for their details and created the class list this way – this was then entered into the Attendance Register and the Admissions Register, along with the Admission list in the Attendance Register. No checks were made against admission registers, application forms or support documents. After interviewing some Head Teachers it became apparent that one was left with a bit of a question about which came first – the Class List or the Admission Register. One of the recommendations with the SADC EMIS review is that it should be compulsory when learners register to present their birth ceritifcate – this could be used to confirm the name and age of learners and will assist in improving data quality. From the Study fieldwork team side, the loose use of terminology should be avoided. When wanting the details of learners admitted into the school, or registered within the school, there should be a request for details of learners admitted into the school - not enrolment. Enrolment is termed to be the count of learners enrolled at a school, or in a class/grade. Also it is suggested that the duplication of effort be avoided. For the Study purposes, the school should be asked to submit their admission lists from the class Attendance Register, for all grades. To ensure that the Grade and Stream is indicated. Schools should not be requested to create a new list of learner details for the purpose of the Study. Use existing records and information products. All regulations and guidelines, and observations, mentioned earlier, only apply to public schools. If Study participants should attend private schools, these operate independently and different records, procedures and practices may apply. Page 25