Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00005431 IMPLEMENTATION COMPLETION AND RESULTS REPORT TF-A4773 ON A SMALL GRANT IN THE AMOUNT OF US$500,000 TO THE Ministry of Finance of the Dominican Republic FOR A Strengthening the Capacity to Produce and Use Quality Education Statistics Grant (P163049) March 24, 2021 Education Global Practice Latin America And Caribbean Region Regional Vice President: Carlos Felipe Jaramillo Country Director: Michel Kerf Regional Director: Luis Benveniste Practice Manager: Emanuela Di Gropello Task Team Leader(s): Monica Yanez-Pagans ICR Main Contributor: Wendy de León ABBREVIATIONS AND ACRONYMS DR Dominican Republic EMIS Education Management and Information System ICE Educational Quality Index (�ndice de Calidad Educativa) ICR Implementation Completion and Results Report ICT Information and Communications Technology INAIPI National Institute for Comprehensive Early Childhood Care (Instituto Nacional de Atención Integral a la Primera Infancia) IRI Intermediate Results Indicator ISR Implementation Status and Results Report MINERD Ministry of Education of the Dominican Republic (Ministerio de Educación de la República Dominicana) NSDS National Strategy for the Development of Statistics NSO National Statistics Office OCI Office of International Cooperation (Oficina de Cooperación Internacional) OEI Organization of Ibero-American States (Organización de Estados Iberoamericanos) PDO Project Development Objective SABER Systems Approach for Better Education Results SIGERD Dominican Republic’s Information System for Education Sector Management (Sistema de Información para Gestión Educativa de la República Dominicana) SSGRD Statistical Service Group TERCE Third Regional Comparative and Explanatory Study (Tercer Estudio Regional Comparativo y Explicativo) UNESCO United Nations Educational, Scientific and Cultural Organization TABLE OF CONTENTS DATA SHEET .......................................................................................................................... 1 I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 3 II. OUTCOME .................................................................................................................... 10 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 16 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 17 V. LESSONS LEARNED AND RECOMMENDATIONS .............................................................. 18 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 19 ANNEX 2. PROJECT COST BY COMPONENT ........................................................................... 24 ANNEX 3. RECIPIENT COMMENTS ........................................................................................ 25 ANNEX 4. SUPPORTING DOCUMENTS .................................................................................. 31 The World Bank Strengthening the Capacity to Produce and Use Quality Education Statistics (P163049) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name Strengthening the Capacity to Produce and Use Quality P163049 Education Statistics Country Financing Instrument Dominican Republic Investment Project Financing Original EA Category Revised EA Category Organizations Borrower Implementing Agency Ministry of Education of the Dominican Republic - Ministry of Finance of the Dominican Republic MINERD Project Development Objective (PDO) Original PDO The objective of the Project is to strengthen MINERD’s capacity to produce high-quality data, consolidate data from different data systems and use that data to inform stakeholders in the education sector. Page 1 of 31 The World Bank Strengthening the Capacity to Produce and Use Quality Education Statistics (P163049) FINANCING FINANCE_TBL Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) Donor Financing TF-A4773 500,000 500,000 500,000 Total 500,000 500,000 500,000 Total Project Cost 500,000 500,000 500,000 KEY DATES Approval Effectiveness Original Closing Actual Closing 03-Apr-2017 07-Apr-2017 07-Apr-2019 30-Sep-2020 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 26-Mar-2019 0.22 Change in Results Framework Change in Loan Closing Date(s) Reallocation between Disbursement Categories Change in Implementation Schedule 07-Apr-2020 0.50 Change in Loan Closing Date(s) KEY RATINGS Outcome Bank Performance M&E Quality Highly Satisfactory Satisfactory Substantial RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 19-Sep-2017 Satisfactory Satisfactory 0.00 02 09-Feb-2018 Satisfactory Satisfactory 0.15 Page 2 of 31 The World Bank Strengthening the Capacity to Produce and Use Quality Education Statistics (P163049) 03 01-Aug-2018 Satisfactory Satisfactory 0.15 04 15-Feb-2019 Moderately Satisfactory Moderately Satisfactory 0.22 05 29-Aug-2019 Moderately Satisfactory Moderately Satisfactory 0.22 06 07-Apr-2020 Moderately Satisfactory Moderately Satisfactory 0.50 07 25-Sep-2020 Satisfactory Satisfactory 0.50 ADM STAFF Role At Approval At ICR Regional Vice President: Jorge Familiar Calderon Carlos Felipe Jaramillo Country Director: Tahseen Sayed Khan Michel Kerf Director: Amit Dar Luis Benveniste Practice Manager: Reema Nayar Emanuela Di Gropello Task Team Leader(s): Juan Baron, Elena Maria Roseo Monica Yanez Pagans Wendy Elizabeth De Leon ICR Contributing Author: Samayoa I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES Context 1. Country Context at Appraisal. In 2017, the Dominican Republic (DR) was coming off rapid growth over the previous decades, which had translated into a reduction in poverty rates and income inequality. Between 1992 and 2013, the Dominican Gross Domestic Product (GDP) grew at an average annual rate of 5 percent, while between 2014 and 2016 growth rates accelerated to an average of 7 percent per annum. The average growth rate of the GDP in the DR during the period between 2014 and 2016 was among the highest in the Latin America and the Caribbean region. This sustained economic growth had translated into a significant reduction in poverty rates from 30 percent in 2000 to 20 percent in 20161 and a reduction in income inequality measured by the Gini coefficient from 0.52 in 2000 to 0.46 in 2016.2 2. Education Sector Context . At appraisal, poor student learning outcomes were the main challenge faced by the Dominican education system. Results from regional assessments of learning 1 The reported poverty rates are calculated using the international lower-middle-income poverty line for Latin America and the Caribbean established at $5.50-a-day in 2011 PPP prices (https://www.worldbank.org/en/topic/poverty/lac-equity- lab1/poverty/head-count). 2 World Bank (2016). Building a better future together: Dominican Republic Policy Notes. October. Washington, D.C: USA. Page 3 of 31 The World Bank Strengthening the Capacity to Produce and Use Quality Education Statistics (P163049) outcomes in 2013 had confirmed the lag in comparison to other countries in the region. According to results from the Third Regional Comparative and Explanatory Study ( Tercer Estudio Regional Comparativo y Explicativo, TERCE),3 the DR had low learning results in reading and mathematics in third and sixth grade. For third grade, the results showed that 74 percent and 85 percent of students were at level 1 in reading and mathematics, respectively. 4 For sixth grade, the results showed that 38 percent5 and 80 percent of students were in level 1 for reading and mathematics, respectively. 6 To improve learning outcomes, the DR needed, among other things, to build the foundations to formulate and implement quality-enhancing public education policies. In line with that goal, the DR’s Ministry of Education ( Ministerio de Educación de la República Dominicana, MINERD)7 embarked on a plan to improve its capacity to produce high-quality data and to integrate and use existing data for effective management and decision making. 3. As proof of its strong commitment to the aforementioned goal, MINERD started strengthening coordination mechanisms with the National Statistics Office (NSO) under the implementation of the National Strategy for the Development of Statistics (NSDS) 2013-2016. MINERD’s National Office for Planning and Educational Development implemented concrete actions aimed at strengthening its information systems in coordination with the NSO, such as, for example, the modernization of the Education Management and Information System (EMIS); improving the flexibility of the DR’s Information Systema for the Management of the Education System ( Sistema de Información para Gestión Educativa de la República Dominicana, SIGERD); the creation of operational manuals for schools and school districts; and the development of a school quality index, management indicators at the school level, among others. 4. Despite the Governmental commitment to increase capacity to produce and use data for decision making in the education sector, several challenges remained. The most important were: (i) the integration of different data systems (e.g. the school decentralization system, the school management system, the student learning assessments system, the República Digital Program),8 and their alignment with the Education Chapter of the NSDS and international commitments on the production of education statistics (e.g. the Sustainable Development Goals); (ii) a quality assurance system for data production and collection that integrated protocols, responsibilities, training material, training data quality audits, and other procedures; and (iii) data dissemination, analysis and use, including open data initiatives that allowed access to multiple stakeholders in a user-friendly way, using the variety of visualizations and 3 The TERCE study analyzes the knowledge and skills of students based on their specific curricula and ranks their performance based on the learning objectives of each country. It is coordinated by the UNESCO Regional Bureau for Education in Latin America and the Caribbean. 4 Performance levels range from Level 1 (the lowest) to Level 4 (the highest). Level 1 refers to the minimum proficiency levels for reading and mathematics. 5 For sixth grade, 54 percent were at Level 2 for reading (total of 92 percent in Level 1 and Level 2). 6 These low outcomes were later confirmed and subsumed into the “learning poverty� metric created by the World Bank in 2019, estimating that 81 percent of Dominican children at late primary age are not proficient in reading, after adjusting for the out-of- school population. See World Bank (2019); Dominican Republic Learning Poverty Brief. 7 MINERD is responsible for the service delivery of public pre-university education (initial, basic and intermediate education), and has a governance role over all pre-university schools (both public and private). 8 República Digital is a government program that comprises all policies and actions to promote the incorporation of ICT in productive, educational, and governmental processes and of services to citizens. The program was created by Presidential Decree 258-16 on September 16, 2016. Page 4 of 31 The World Bank Strengthening the Capacity to Produce and Use Quality Education Statistics (P163049) analytical tools available.9 6. Bank rationale for supporting the Project. Based on the existing challenges, the Project aimed to finance the design and implementation of a set of activities that would integrate data systems, set up systems to guarantee the quality of data, and improve data use, analysis, and dissemination. The Project would help inform the design of education reforms and decision-making through improved availability and use of information and statistical data in the education sector. Finally, the Project would also contribute to a more efficient use of public resources through its support to a more informed decision making in the public education system. The Project was aligned with the Country Partnership Strategy FY15-FY18.10 7. The Project would be implemented by the MINERD through the Office of International Cooperation (Oficina de Cooperación Internacional, OCI), which is the unit responsible for coordinating and overseeing internationally funded projects in the education sector. The OCI would coordinate with other offices within MINERD, including the Planning and Statistics Office and the Information and Communication Technology Office. Project Development Objectives (PDOs) 8. The Project Development Objective (PDO) was to strengthen MINERD’s capacity to produce high- quality data, consolidate data from different data systems and use that data to inform stakeholders in the education sector. The PDO remained the same during the implementation period. Key Expected Outcomes and Outcome Indicators 9. The Project’s design included three indicators to assess progress towards the achievement of the PDO, as follows: • PDO Indicator 1: New protocols for data collection and data quality verification are available, disseminated and piloted. • PDO Indicator 2: EMIS systems are improved to consolidate different statistical data from the Ministry and new modules for data collection are developed and incorporated. • PDO Indicator 3: Open data portal is available to allow all stakeholders to access timely and relevant education data and analytics, including user-friendly reports. 10. The indicators remained the same for the duration of the Project. 9 The Open Data Readiness Assessment (ODRA) for the Dominican Republic is available at: http://opendatatoolkit.worldbank.org/docs/odra/odra_republica_dominicana.pdf 10 The Project contributed to results area 4 “Promoting equitable, efficient, sustainable management of public resources� and results area 5 “Strengthening social service delivery.� Contributions would be achieved by improving data quality and education statistics for evidence-based decision making that would support the efficient use of resources and its allocation, increase accountability, and contribute to the long-term reforms of MINERD aimed at improving service delivery. Page 5 of 31 The World Bank Strengthening the Capacity to Produce and Use Quality Education Statistics (P163049) Components Component 1: Integration and improvement of MINERD’s information systems 11. The objective of this component was to support the improvement of SIGERD with the integration of different information systems and the establishment of internal procedures and quality assurance initiatives. This component included support to two main subcomponents: (i) integration of systems; and (ii) data production quality assurance. Each of these subcomponents had specific activities planned to be funded, as follows: • Subcomponent 1.1: Integration of systems- Integration of different information systems into SIGERD, through: (i) carrying out an audit of SIGERD’s quality of information based on selected schools; (ii) identifying and integrating MINERD’s management systems, statistics and indicators in accordance with the NSDS; (iii) developing an action plan to incorporate early childhood development indicators into MINERD’s management system; (iv) carrying out selected system assessments and cleaning of databases; (v) strengthening and expanding SIGERD, including the development of new modules for better data use at school and district levels; (vi) strengthening MINERS’s decentralization directorate management system and integrating with SIGERD; (vii) benchmarking SIGERD using the Systems Approach for Better Education Results- Education Management Information Systems (SABER-EMIS) instrument; (viii) organizing national workshops and south-south exchanges with other countries; (ix) developing a multi-sectorial plan for the development of vocational education statistics for MINERD and the Ministry of Labor, aligned with the NSDS; and (x) carrying out capacity-building activities. This subcomponent contributed to achieving objective number 2 of the PDO (to strengthen the capacity of MINERD to consolidate data from different data systems) through the achievement of PDO indicator 2 (EMIS systems are improved to consolidate different statistical data from the Ministry and new modules for data collection are developed and incorporated). • Subcomponent 1.2: Data production quality assurance- Establishment of internal procedures and data quality assurance initiatives, through: (i) unifying and defining the coding of schools and other MINERD entities; (ii) developing the education sector information and communications technology (ICT) strategy under República Digital; (iii) developing and disseminating internal procedures for indicators, data collection and quality assurance for MINERD’s staff; (iv) carrying out an assessment of MINERD’s decentralized units capacity to produce and use data under SIGERD; (v) carrying out workshops with sector stakeholders to share and discuss the results of the assessment; and (vi) carrying out capacity-building activities. This subcomponent contributed to achieving objective number 1 of the PDO (to strengthen the capacity of MINERD to produce high-quality data) through the achievement of PDO indicator 1 (new protocols for data collection and data quality verification are available, disseminated and piloted). Component 2. Training and dissemination of information 12. The objective of this component was to strengthen the capacity of MINERD to produce quality data, make it available to the public and use it to inform decision-making and system stakeholders. This component included two main subcomponents: (i) training to improve data quality production and use; Page 6 of 31 The World Bank Strengthening the Capacity to Produce and Use Quality Education Statistics (P163049) and (ii) improvement of data dissemination (open data). Each of these subcomponents also had specific activities planned to be funded, as follows: • Subcomponent 2.1: Training to improve data quality production and use by MINERD’s staff through: (i) developing training modules; (ii) carrying out training on data entry and data use for school directors and MINERD staff; and (iii) designing training modules for dissemination on student learning assessment results. This subcomponent contributed to achieving objective number 3 of the PDO (to strengthen the capacity of MINERD to use data to inform stakeholders in the education sector). • Subcomponent 2.2: Improvement of education data dissemination (open data) through: (i) carrying out knowledge exchange initiatives with other countries to develop a strategy for education data dissemination and analysis; (ii) designing and improving an open data website; (iii) providing technical assistance for the development of an open data website, and visualization and analytics databases; (iv) strengthening the use of indicators and information disseminated through the open data website; (v) carrying out workshops to present and disseminate the open data website among internal and external stakeholders of the education sector; and (vi) carrying out consultations with stakeholders on data availability and effective communications means. This subcomponent contributed to achieving objective number 3 of the PDO (to strengthen the capacity of MINERD to use data to inform stakeholders in the education sector) through the achievement of PDO indicator 3 (open data portal is available to allow all stakeholders to access timely and relevant education data and analytics, including user-friendly reports). Component 3. Project implementation 13. The objective of this component was to support project implementation. The activities supported under this component included support for project administration, project audits, procurement, financial management and basic monitoring and evaluation activities to be carried out by MINERD. 14. All three components of the Project included capacity-building activities as part of the terms of reference of the consultancies and included on-the-job knowledge transfer from consultants to MINERD’s staff, instead of workshops only. The objective was to create capacity from within MINERD. Project Restructuring First Restructuring 15. The Project had a Level 2 restructuring in March of 2019. The amount disbursed at restructuring was US$0.22 million (44 percent of grant had been disbursed). The restructuring entailed the following: i. Change in results framework: With the restructuring, MINERD prioritized capacity-building activities for its staff over consultancies. To reflect the prioritization of training and workshops over consultancies, the end target value of the Intermediate Results Indicator (IRI) “number of consultancies executed and/or being executed� was revised downwards Page 7 of 31 The World Bank Strengthening the Capacity to Produce and Use Quality Education Statistics (P163049) from 18 to 11. The target was also revised because the team estimated that 11 outputs would be produced, even if the number of contracts to complete these outputs was larger. ii. Revisions to project activities: Revisions were made to activities within Component 1. One activity was dropped as it was no longer needed; two activities were dropped because they were outside MINERD’s direct control; and one activity was added to reflect a high-priority need. These changes had a limited impact on the PDO, as approximately 17 of the Project’s original 20 activities would still be carried out as planned. The proposed restructuring revised project activities under Component 1 were as follows: • The activity “benchmarking of the Dominican EMIS system using SABER-EMIS instrument� was dropped, as MINERD had sufficient understanding of the EMIS based on existing diagnostics, including one from the United Nations Educational, Scientific and Cultural Organization (UNESCO) and one from the Inter-American Development Bank. • The activity “development of a multisectoral plan for the development of vocational education statistics between the MINERD and Ministry of Labor and aligned with the NSDS� was dropped, as it was outside the direct control of MINERD and the required intersectoral collaboration was not expected to be feasible within the timeframe of the Project. • The activity “developing the education sector ICT strategy under República Digital was dropped, as it was outside of the purview of MINERD to develop such a strategy. • A new activity was added, “diagnosis of MINERD’s physical information archiving systems� to focus on identifying ways to improve archiving systems in which large amounts of data and information were underutilized. iii. Reallocation between disbursement categories: The restructuring reallocated approximately US$62,000 in funds from Category 1 (Consulting and Non-Consulting Services) and Category 3 (Operating Costs) towards Category 2 (Training and Workshops), to reflect the MINERD’s prioritization of capacity building for its own staff. iv. Extension of closing date: The proposed restructuring extended the closing date of the Project by one year, from April 7, 2019 to April 7,2020. The extension of twelve months would compensate the one-year delay in beginning implementation. In fact, upon reaching effectiveness (April 2017), the implementation of the Project had begun a year later (March 2018) due to delays in completing a few key fiduciary processes, including opening the designated accounts and obtaining the specimens of signature. Second Restructuring 16. The Project had a second Level 2 restructuring in May of 2020. The amount disbursed at restructuring was US$0.50 million. The restructuring only entailed the extension of the closing date. The Recipient requested an extension of the closing date of the Project due to the delays caused to the implementation process by the COVID-19 pandemic, especially the social distancing measures adopted in the country to control the spread of the virus. The restructuring extended the closing date of the Project by about 6 months, from April 7, 2020 to September 30, 2020 so that all activities that were affected by the COVID-19 pandemic could be completed as planned. Page 8 of 31 The World Bank Strengthening the Capacity to Produce and Use Quality Education Statistics (P163049) Theory of Change (Results Chain) 17. The Project’s Theory of Change is represented in the following results chain: Long-Term Main activities Outputs PDO Outcomes Outcome Component 1: Subcomponent 1.2 Component 1: Subcomponent Objective 1: MINERD’s - Perform data quality audit in 298 1.2 capacity to produce high- schools to compare physical records and - Carry out a quality audit of quality data is records in the SIGERD to find student data registry from the strengthened. inconsistencies. school level included in SIGERD. - Develop protocols to monitor - Develop new protocols for data - New protocols for data enrollment, teachers and administrative collection and quality collection and data quality staff and infrastructure. verification. Create guidelines verification are available, - Create guidelines and instruments for and instruments for the disseminated and piloted data collection and quality assurance. implementation of the new (Project Development - Carry out virtual workshops for protocols. Objective -PDO- indicator MINERD’s technicians on the - Train MINERD technicians on 1). implementation of the guidelines and the new guidelines and the use of the instruments (10 instruments. workshops for 350 technicians). Production of Component 1: Subcomponent 1.1 Component 1: Subcomponent Objective 2: MINERD’s high-quality - Diagnosis of information system 1.1 capacity to consolidate data and performance to analyze inefficiencies - Design datasets of different data from different informed and improve the operation of SIGERD. MINERD information systems, systems is strengthened. stakeholders - Design and integrate datasets of allowing their integration with for improved different information systems: SIGERD, SIGERD. - EMIS systems are evidence- human resources, and student’s learning - Integrate MINERD’s improved to consolidate based decision- assessments (national standardized management systems with different statistical data making in the tests). SIGERD. from the Ministry, and new education - Expand the SIGERD by developing two -Develop functionalities through modules for data sector. new modules: student and teacher new modules for better data use collection are developed functionalities (student grades, redesign for teaching (student and teacher and incorporated (PDO of student file, administration of module and administrative indicator 2). teachers' schedules) and administrative module). functionalities (inventories, school management). Component 2: Subcomponent 2.1 Component 2: Subcomponent Objective 3: MINERD’s - Implement a training program on data 2.1 capacity to use data to entry and use for school principals and - Train MINERD staff and inform stakeholders in the MINERD staff (52 workshops with more principals on data entry and the education sector is than 2,300 principals and technicians information system. strengthened. trained). Component 2: Subcomponent Component 2: Subcomponent 2.2 2.2 - Open data portal is - Develop dynamic dashboards to display - Design an open data website to available to allow all information for internal and external access information in an open, stakeholders to access users. Create an open data website to dynamic way. timely and relevant access information. - Create a user intelligence education data and - Create a user intelligence network network. Train users on the analytics, including user- called Red de Inteligencia. Organize information system. friendly reports (PDO workshops to integrate de network, - Draft new internal regulations indicator 3). train its members, and disseminate the for inputting, processing and Page 9 of 31 The World Bank Strengthening the Capacity to Produce and Use Quality Education Statistics (P163049) open data website. Twenty workshops using EMIS data and the Strategic to train 600 technicians and network Intelligence System. Currently, focal points. SIGERD does not have its own - Develop the Educational Quality Index internal regulation. (ICE) - Design an Education Quality Index based on EMIS data. 18. There were two critical assumptions that needed to hold for the achievement of the PDO: a. The core technical staff would continue to work with MINERD or would train newly hired staff to continue with SIGERD updates. b. The trained users (User Intelligence Network) would generate information demand and increase social accountability and transparency of MINERD. This would also lead to informed decision- making from MINERD authorities. II. OUTCOME Assessment of Achievement of the PDO 19. This section assesses the degree of achievement of each of the following three outcomes of the Project: • To strengthen the capacity of MINERD to produce high-quality data. • To strengthen the capacity of MINERD to consolidate data from different data systems. • To strengthen the capacity of MINERD to use data to inform stakeholders in the education sector. Objective 1: To strengthen the capacity of MINERD to produce high-quality data 20. The Project’s first objective was to strengthen the capacity of MINERD to produce high-quality data. Achievement of this objective was measured by PDO Indicator 1: “New protocols for data collection and data quality verification are available, disseminated and piloted.� The baseline was “No�, and the target was “Yes.� This target was achieved. 21. Objective 1 was supported by Component 1. A summary of the key activities completed that substantiate the achievement of this objective are below: • Under component 1, the Project carried out an audit of SIGERD’s quality of information based on a sample of 298 schools. The auditors compared physical records of the schools with the digital records from SIGERD to identify inconsistencies and verify data quality. The consulting firm that was hired to carry out this audit work (Statistical Service Group, SSGRD) found that around 3 percent of students were not registered in SIGERD, with the under-registration being larger for pre-primary students.11 116 percent of pre-primary students, 4 percent of primary students and 1 percent of secondary students were not registered in SIGERD. Page 10 of 31 The World Bank Strengthening the Capacity to Produce and Use Quality Education Statistics (P163049) • Based on the findings of the data quality audit, the Project established new protocols for data collection and quality verification. Protocols were created and tested to monitor enrollment, teaching and administrative staff and infrastructure, which included a step-by-step guide of the verification procedure. A decision was made to have school supervisors carry out quality audits twice a year moving forward. • Monitoring instruments were developed for the implementation of the new protocols in SIGERD. An explanatory video was also created. The instruments created included the following: (i) monitoring the registration of enrolled students; (ii) monitoring the registration of teachers; and (iii) monitoring school infrastructure. • MINERD technicians were trained on the new guidelines and instruments to ensure adequate implementation and sustainability. Virtual workshops for MINERD’s technicians on the implementation of the guidelines and the use of the instruments were held. Ten workshops were carried out to train 350 technicians. The school supervisors and directors were trained to implement the data verification instruments and to correct inconsistencies in order to guarantee the collection of high-quality data. The protocols also established that schools will be visited twice a year to cross-check data and correct potential inconsistencies. 22. This outcome was fully achieved and its efficacy rating is High. New protocols for data collection and data quality verification were created, disseminated, and piloted. Monitoring instruments to implement the new protocols were developed and MINERD technicians were trained on the new instruments and guidelines. The protocols ensure correction of potential inconsistencies to maintain the SIGERD information system up to date with reliable data. Objective 2: Strengthen the capacity of MINERD to consolidate data from different data systems 23. The Project’s second objective was to strengthen the capacity of MINERD to consolidate data from different data systems. Achievement of this objective was measured by PDO Indicator 2: “EMIS systems are improved to consolidate different statistical data from the Ministry, and new modules for data collection are developed and incorporated.� The baseline was “No�, and the target was “Yes�. This target was achieved. 24. Objective 2 was supported by Component 1. A summary of the key activities completed that substantiate the achievement of this objective are below: • Component 1 supported the creation of datasets (data structures) from different MINERD’s information systems – like human resources and quality assessment (national standardized tests) data - to allow for their integration with the SIGERD. The datasets are now able to connect to a data warehouse, which allows the Power Business Intelligence application to provide users with different pieces of information requested. Users are provided three levels of access: strategic (directors from MINERD’s central offices), tactical (analysts), and operational. The existing data structure provides the necessary inputs for the creation of dashboards disseminated in the open data website (see objective 3). Page 11 of 31 The World Bank Strengthening the Capacity to Produce and Use Quality Education Statistics (P163049) • In addition, functionalities were developed through new modules for better management and data use in the teaching (student and teacher functionalities) and administrative fields. The new modules were instrumental to consolidate data from different sources. The SIGERD was expanded by developing two new modules: the student and teacher module (student grades, redesign of student file to facilitate registration, administration of teachers' schedules to optimize human resources and compliance with regulations) and the administrative functionalities module (inventories and school management). SIGERD now incorporates early childhood indicators, since children can be registered as early as 45 days old at the National Institute for Comprehensive Early Childhood Care (Instituto Nacional de Atención Integral a la Primera Infancia, INAIPI). • Finally, a diagnosis of information system performance to analyze inefficiencies and improve the operation of SIGERD was completed. An evaluation of the source code and system resources was carried out to enhance the performance of the platform, making its large-scale implementation feasible. The recommendations for improving the system are currently being implemented by MINERD. 25. This outcome was fully achieved, and its efficacy rating is High. Prior to the Project, the integration of the data system was a critical gap in data collection and this small grant filled this gap. The EMIS system was improved to consolidate different statistical data from the Ministry and new modules- student and teacher, and administrative- for data collection were developed and incorporated, allowing for new functionalities in the system. The strengthening of SIGERD and the strengthened technical capacity of MINERD staff will allow the Ministry to respond to future data requirements. Such was the case under the current COVID context, which required MINERD to remotely monitor their students. MINERD was able to respond quickly to this data requirement by creating two new functionalities in SIGERD: the first was for recording if students received the printed booklets, if the student handed in the activities from the booklet and the quality of their work; the second functionality was to monitor students’ progress for each curricular area (initiated, in progress or attained). At the beginning of the Project, there were few technicians capable of incorporating these kinds of changes to SIGERD. Currently, MINERD has several trained staff and consultants who can make these types of adjustments to the system. Furthermore, the continuation of the Project during three different administrations shows the Government’s commitment to the activities supported by the Project, serving as an indicator for its sustainability. Objective 3: Strengthen the capacity of MINERD to use data to inform stakeholders in the education sector 26. The Project’s third objective was to strengthen the capacity of MINERD to use data to inform stakeholders in the education sector. Achievement of this objective was measured by PDO indicator 3: “Open data portal is available to allow stakeholders to access timely and relevant education data and analytics, including user-friendly reports�. The baseline was “No�, and the target was “Yes�. This target was achieved. 27. Objective 3 was supported by Component 2. A summary of the key activities completed that substantiate the achievement of this objective are below: • The design of an open data website to access information in an open and dynamic way was Page 12 of 31 The World Bank Strengthening the Capacity to Produce and Use Quality Education Statistics (P163049) completed. Dynamic dashboards were developed to display information for internal and external users. The open data portal is now operational, allowing all education stakeholders to access timely and relevant education data and analytics. Prior to the Project, MINERD received specific data requests (on paper), which were very slow, time-consuming, and inefficient to process. Currently, the portal allows generating customized user-friendly reports directly from the website in a timely manner for all users (internal and external). • A user intelligence network called Red de Inteligencia was established. The network includes people who capture/collect the data, experts who analyze the information, decision-makers and advisers who use it, and the communication channels among all of them. Currently, this network includes more than 600 users. The creation of this network gives sustainability to the information system. A total of 20 workshops were held to integrate the network, to train its 600 members (technicians and network focal points), and to disseminate the open data website. • A training program on data entry and use for school principals and MINERD staff was implemented. Fifty-two workshops were held for 2,300 principals and technicians. This training included the dissemination of student’s national learning assessments results. Before the Project, student’s national learning assessment results were physically sent out to schools. Currently, results have been incorporated into the SIGERD and can be accessed in a dynamic manner. • An Education Quality Index based on EMIS data was developed. The Educational Quality Index was created to measure institutional performance, and management and supervision at the school, district, regional and central levels. The index evaluates student learning, teacher quality, and school management and allows to establish indicators and goals for schools. • New internal regulations for uploading, processing, and using EMIS data and the Strategic Intelligence System were drafted. The regulation has now filled a legal gap because SIGERD had never had its own regulation before. The draft regulation was concluded at the end of the previous administration’s term in 2020. The regulation was presented to the new authorities and is expected to be approved shortly by the minister of education, but has experienced delays due to other COVID-19 political priorities. • Knowledge exchange initiatives with other countries to develop a strategy for education data dissemination and analysis were organized. During the implementation of the Project, meetings were held with teams from Brazil, Colombia, Chile, Panama, and Honduras to exchange information and experiences. 28. This outcome was fully achieved and its efficacy rating is High. An open data portal was created to allow stakeholders to access timely and relevant education data and analytics. This activity increased efficiency by reducing data requests (on paper) to MINERD because stakeholders can easily generate their own data, optimizing MINERD technicians’ time. This outcome is considered to have surpassed expectations because of the establishment of the user intelligence network with more than 600 users. Page 13 of 31 The World Bank Strengthening the Capacity to Produce and Use Quality Education Statistics (P163049) MINERD now utilizes user statistics to measure if the open data portal is being used. The latest statistics confirm that the portal is being used, with around four thousand views per day. Having trained users generates information demand and can lead to increased social accountability and transparency. Executed Consultancies 29. The results framework included one Intermediate Results Indicator (IRI): “Number of consultancies executed and/or being executed�. The baseline was 0 and the target was 11. This target was surpassed.12 A total of 14 consultancies were executed through two agencies that were hired: the Organization of Ibero-American States (Organización de Estados Iberoamericanos, OEI) and the Statistical Service Group (SSGRD). The consultancies contributed to the attainment of all three main objectives of the Project’s PDO. Of the 14 consultancies, 2 contributed to the achievement of Objective 1, 8 contributed to the achievement of Objective 2, and 4 contributed to the achievement of Objective 3. 30. The table below lists all the consultancies that were hired and their costs, accounting for 71 percent of the Project proceeds. The remaining 29 percent of the Project’s cost was used to finance MINERD trainings, the payment of financial audits, and other minor project costs. Objective Activity/ Consultancy Cost (US$) Implementer SIGERD Functionalities 1. Consultancy - Software design for the strengthening of OEI 46,206.24 MINERD's EMIS system (SIGERD) 2. Consultancy - Software Implementation, testing and OEI documenting for the strengthening of MINERD's EMIS system 20,548.50 (SIGERD)- New pedagogical (student and teacher module) 3. Consultancy - Software Implementation, testing and OEI documenting for the strengthening of MINERD's EMIS system 20,548.50 (SIGERD)- New administrative 4. Consultancy - Software Implementation, testing and documenting for the strengthening of MINERD's EMIS system 20,548.50 OEI (SIGERD)- Optimization of pedagogical (student and teacher) 5. Consultancy - Software Implementation, testing and documenting for the strengthening of MINERD's EMIS system 20,548.50 OEI (SIGERD)- Optimization of administrative System performance 6. Consultancy - Systems assessments and databases cleaning OEI diagnostics 19,323.18 based on integration plan (including gender aspects) Datasets / Information 7. Consultancy - Database Administrator for the design of datasets OEI 17,856.23 integration / Portal for the MINERD's Systems of Business Intelligence 8. Consultancy - Database Administrator Assistant for the design of OEI 15,286.14 datasets for the MINERD's Systems of Business Intelligence 9. Consultancy - Graphic design of dashboards and portal of the OEI 14,652.32 MINERD's System of Business Intelligence Education Quality Index 10. Consultancy - Design and implementation of an Education OEI 15,789.24 Quality Index for schools, districts and regional offices Development of 11. Consultancy - Development of training modules, including OEI 39,293.75 training modules videos Design of dissemination 12. Consultancy - Design of dissemination materials for training 27,589.12 OEI 12The last ISR indicated that 11 consultancies had been executed. However, final information provided by MINERD confirmed that a total of 14 consultancies were executed. Page 14 of 31 The World Bank Strengthening the Capacity to Produce and Use Quality Education Statistics (P163049) and training materials Data quality audit 13. Consultancy - Audit of quality of information in the EMIS (based on sample) and design of a predictive model of educational 38,652.24 SSGRD indicators Protocols 14. Consultancy - Development of protocols for indicators, data SSGRD 40,496.63 collection, and quality assurance OCI Trainings OCI Workshops (food, transport and per diem) 21,266.65 Reimbursement for payments made for training 67,539.15 OCI Project Costs Publications 2,006.15 Commission payments, withholdings, others 23,412.04 Audit KPMG Payment 1 (2019) 13,495.40 Dominicana KPMG Payment 2 (2020) 14,941.52 Dominicana Total 500,000.00 Overall Outcome Rating Rating: Highly Satisfactory 31. The simplified Implementation Completion and Results (ICR) Report template for small grants does not require a discussion on relevance and/or efficiency. However, a brief description is included in this section of all three dimensions for completion. 32. Relevance (High). The Project’s objectives and outcomes were highly relevant to addressing the Government’s capacity gaps to produce high-quality data and use them for decision making in the education sector at appraisal, and continue to be consistent with current education sector and Bank priorities. The Project was aligned with the Country Partnership Strategy FY15-FY18. Indeed, a highly functional SIGERD supported the sector during the recent school closure stage, as it was adapted to monitor remote learning. The strengthened information system can now be expanded and built upon going forward to adapt to MINERD’s new priorities and can be an important tool to improve sector management and budget allocation. 33. Efficacy (High). The Project was successful in increasing the capacity of the MINERD to produce high-quality data, consolidate data from different data systems, and use that data to inform stakeholders in the education sector, as measured by PDO Indicators 1, 2, 3. All three PDO indicators were fully achieved, and the IRI was also accomplished. All activities are considered sustainable in the future for MINERD. The efficacy rating is obtained through a simple average of each PDO part’s assessment, as per the analysis undertaken above and summarized in the table below. Objective Rating To strengthen the capacity to produce high-quality data High To strengthen capacity to consolidate data from different data systems High To strengthen capacity to use data to inform stakeholders High Efficacy Rating High Page 15 of 31 The World Bank Strengthening the Capacity to Produce and Use Quality Education Statistics (P163049) 34. Efficiency (Substantial). Despite being a small grant (US$0.5M) the Project allowed existing investments in the institution’s information system to be maintained. At appraisal, the Government was considering creating a new information system, however, because of the Project, MINERD was able to build upon the existing system to make it more efficient and integrate it with other information systems. In this sense, the Project ended up pursuing a more cost-efficient solution than the original idea to start a completely new information system. No metric was developed to measure the benefits, but to the extent that the strengthening of the EMIS has far-reaching implications for the student population and education authorities in the country, benefits from the Project seem to clearly outweigh costs. Although the Project was very cost-efficient, it is rated as substantial considering the many transaction costs it entailed, a typical feature of small grants. 35. The overall outcome rating for the Project is Highly Satisfactory and supported by ratings on relevance, efficacy, and efficiency. The Project’s relevance and efficacy are rated High and efficiency is rated Substantial. Since it is rated high in two criteria (one of which is efficacy) and substantial on the third, the overall rating is Highly Satisfactory. This rating factors in the fact that the Project faced issues, like the initial delays in effectiveness, but there were almost no shortcomings in achieving the PDO once implementation kicked off. 13 Other Outcomes and Impacts 36. A positive result of the Project was the quick response of the technical MINERD staff to adapt SIGERD to the COVID-19 context and distance learning needs. During the closure of schools due to the pandemic, the SIGERD was key to support some of the initiatives designed for monitoring remote learning. Printed booklets were given to students and learning content and activities, linked to these printed booklets to support learning from home, were broadcast on television and radio. Th ose teachers that went to schools were able to monitor progress of their students with support from technological equipment. The strengthening of SIGERD (Objective 2 of this Project) and the strengthened technical capacity of MINERD staff allowed the system to be adapted quickly and effectively to be able to remotely monitor students. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME 37. The processing time to open the designated account and to authorize the Project signatures were longer than expected at the appraisal stage. For this reason, the Project began without having a capital flow. MINERD agreed to use approximately US$67,000 of its own funds to begin implementation, which were later reimbursed with Project funds. The initial delay in starting execution was offset by a restructuring that extended the execution period by one year. 38. The COVID-19 pandemic caused unexpected implementation delays in planned activities, which were affected by the social distancing measures adopted in the country to control the spread of the virus. A timely second restructuring extended the closing date of the Project by about 6 months, allowing all 13The rating at the time of this Implementation Completion and Results Report (ICR) of Highly Satisfactory is higher than the rating of Satisfactory at the time of the latest Implementation Status & Results Report (ISR). At the time of the ISR (September 2020), there was uncertainty about the sustainability of the Project due to recent changes in authorities. However, at the time of the ICR, this Project is considered highly sustainable. Page 16 of 31 The World Bank Strengthening the Capacity to Produce and Use Quality Education Statistics (P163049) activities to be carried out as planned and all Project objectives to be achieved. 39. During the implementation period (April 2017 to September 2020) there were three different ministers of education (Andrés Navarro, Antonio Peña Mirabal, and Roberto Fulcar). Despite the changes in MINERD authorities, the permanence of key authorities (especially directors) and key technical staff in MINERD allowed for the continuity of implementation. 40. It was essential to have a designated project coordinator (within the Office of International Cooperation, OCI) to follow up on the overall execution of the Project and ensure the coordination of the three units involved in the implementation of this Project -- OCI, Planning and Statistics Office, and Communication Technology Office. The collaboration of these three offices was also crucial for the successful implementation of the Project. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME 41. The Bank team ensured quality at entry. The objectives of the Project were clear and relevant at appraisal and continue to be relevant at closing. The objectives were in line with the Government’s priorities and relevant to sector and development priorities. Implementation arrangements were adequate and included MINERD, which was responsible for the management and coordination of Project activities. The results framework design was simple and consistent with the PDO, and the monitoring and evaluation arrangements were suitable for a small Trust Fund. The PDO was directly linked to the main outcomes’ indicators. Outcomes’ indicators could have attempted to better measure the “use� of the new tools and improvements, but the many areas for improvement combined with the limited implementation time of the grant would have made capturing this dimension more difficult. 42. Bank supervision. The team timely addressed the challenges to the achievement of the development outcomes during implementation, including through the processing of two restructurings (described in section I) that allowed for all the objectives and indicators to be fully achieved. There was adequate supervision during the implementation of the Project, with periodic missions every six months. The Bank team worked closely with the implementation team in MINERD and OCI. The Project had three different TTLs and they ensured continuity of implementation while transitioning. Procurement and financial management were also supervised by the Bank’s specialists. MINERD had delays in sending the financial reports, but the Bank team ensured that all flags were cleared at the time of this ICR. This has been discussed with the MINERD team as something that must be strengthened in future operations. 43. This Project is highly sustainable. MINERD remains committed to strengthening data systems, producing high-quality data, and using that data to inform stakeholders. Several factors support sustainability going forward: (i) MINERD has trained staff that can adjust the data systems and build upon the progress that was made during the Project; (ii) the protocols that were created will ensure periodic revisions for data quality; (iii) the creation of the user intelligence network (600 users) will promote data use by stakeholders for informed decision making. Last but not least, the Project was able to implement consistently notwithstanding the changes in administration, showing the Government’s strong commitment with the activities supported under this Project. Page 17 of 31 The World Bank Strengthening the Capacity to Produce and Use Quality Education Statistics (P163049) V. LESSONS LEARNED AND RECOMMENDATIONS 44. There are four main lessons learned from the implementation of the Project: 45. If a Project includes regulation changes it is crucial to take the political timeframe into account. As part of the Project, a new internal regulation was drafted for uploading, processing, and using EMIS data and for the Strategic Intelligence System. This regulation also included a proposal for institutional performance. Currently, SIGERD does not have yet its own regulation approved. This activity was completed at the end of the previous government period and was then transferred to the new MINERD authorities; however, it is still pending to be approved by the minister of education. Although having a draft of the regulation is a step in the right direction, the approval of the internal regulation would guarantee the administrative sustainability of the improvements made to the information system. Approval of regulation seems more feasible if it is scheduled at the beginning or in the middle of a given administration’s term. 46. When hiring highly specialized consultancies, working with larger agencies could make it easier to find trained staff in a short timeframe. At the beginning of the Project, the goal was to hire several specialized consultants, but the process ended up being too complicated for MINERD to implement. When the consultants’ profiles for the selection process were published, none of the applicants met the requirements, which resulted in the cancellation of the procurement process. After approximately 6 months of delay, MINERD decided to hire two large agencies (the OEI and the SSGRD Group), who were responsible for the deliverables. These larger agencies were able to hire the required experts in a relatively short period of time. Processing these consultancies through these agencies facilitated the implementation and allowed the Project to find consultants with the required background. Throughout all these processes, MINERD supported the agencies to evaluate the technical competencies of the selected consultants. 47. The medium- and long-term sustainability of technological changes is strengthened by having staff and users trained on the new technology. Creating a network of trained users (User Intelligence Network or Red de Inteligencia) increased substantially the sustainability of the activities supported through this Project. The network is currently integrated by more than 600 users who have developed a good understanding of the information available in the system and open data website. MINERD measured user statistics to determine if the platforms were being used or not and confirmed the continuing use of the technological tools by users. Moreover, having trained staff also allowed MINERD to respond to the necessities that resulted due to the COVID-19 pandemic. MINERD’s technical staff quickly adapted SIGERD to distance learning necessities. Having trained staff makes it easier to build on a good system. 48. Small grants have high transactions costs, which could be offset when implemented with larger projects. Many local processes needed for recipient-executed funds are the same for small grants than for large projects, which tends to delay implementation. Such delays could affect the achievement of targets. This could be partially offset by allowing the Bank to execute funds or part of the funds, instead of having funds be completely recipient-executed. In addition, World Bank staff time is not covered. In the case of this grant, the team combined missions with another Bank Project to lower implementation costs. Having a larger operation allows budget-sharing to cover part of the costs. For small grants, high transactions costs should be identified during project appraisal stage as a design risk. Page 18 of 31 The World Bank Strengthening the Capacity to Produce and Use Quality Education Statistics (P163049) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: Strengthen the capacity of MINERD to produce high-quality data. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Protocols for data collection Yes/No N Y Y Y and data quality verification are available, disseminated and 31-Jan-2017 14-Jun-2019 30-Sep-2020 30-Sep-2020 piloted. Comments (achievements against targets): Objective/Outcome: Strengthen the capacity of MINERD to consolidate data from different data systems. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion EMIS systems is improved to Yes/No N Y Y Y consolidate different statistical data from the Ministry, and 31-Jan-2017 14-Jun-2019 30-Sep-2020 30-Sep-2020 new modules for data collections are developed and incorporated. Page 19 of 31 The World Bank Strengthening the Capacity to Produce and Use Quality Education Statistics (P163049) Comments (achievements against targets): Objective/Outcome: Strengthen the capacity of MINERD to use data to inform stakeholders in the education sector. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Open data portal is available to Yes/No N Y Y Y allow all stakeholders to access timely and relevant education 31-Jan-2017 14-Jun-2019 30-Sep-2020 30-Sep-2020 data and analytics, including user-friendly reports. Comments (achievements against targets): A.2 Intermediate Results Indicators Component: Component 1. Integration and improvement of MINERD's information systems Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Number of consultancies Number 0.00 18.00 11.00 14.00 executed and/or being executed 24-Mar-2017 14-Jun-2019 30-Sep-2020 30-Sep-2020 Comments (achievements against targets): Page 20 of 31 The World Bank Strengthening the Capacity to Produce and Use Quality Education Statistics (P163049) Of the 14 consultancies, 10 are linked to Component 1 and 4 are linked to Component 2. Component: Component 2. Training and dissemination of information Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Number of consultancies Number 0.00 18.00 11.00 14.00 executed and/or being executed 24-Mar-2017 14-Jun-2019 30-Sep-2020 30-Sep-2020 Comments (achievements against targets): Of the 14 consultancies, 10 are linked to Component 1 and 4 are linked to Component 2. Page 21 of 31 The World Bank Strengthening the Capacity to Produce and Use Quality Education Statistics (P163049) B. ORGANIZATION OF THE ASSESSMENT OF THE PDO Objective/Outcome 1 1. New protocols for data collection and data quality verification are Outcome Indicators available, disseminated, and piloted (PDO indicator 1) 1.Number of consultancies executed and/or being executed (IRI 1). Intermediate Results Indicators 2 of the 14 consultancies correspond to this objective. 1. Number of school audits to verify data registry quality (298) 2. New protocols developed for data collection and quality verification Key Outputs by Component 3. Number of instruments developed to implement the new protocols (linked to the achievement of the Objective/Outcome 1) (3) 4. Number of MINERD technicians trained on the implementation of the new guidelines and use of the instruments (350). Objective/Outcome 2 1. EMIS systems is improved to consolidate different statistical data Outcome Indicators from the Ministry, and new modules for data collection are developed and incorporated (PDO indicator 2) 1.Number of consultancies executed and/or being executed (IRI 1). Intermediate Results Indicators 8 of the 14 consultancies correspond to this objective. 1. Datasets created for different information systems- human resources and quality assessments- to allow their integration with Key Outputs by Component SIGERD. (linked to the achievement of the Objective/Outcome 2) 2. Develop two new modules for better data use: pedagogical functionalities (student and teacher module) and administrative functionalities. Page 22 of 31 The World Bank Strengthening the Capacity to Produce and Use Quality Education Statistics (P163049) Objective/Outcome 3 1. Open data portal is available to allow all stakeholders to access Outcome Indicators timely and relevant education data and analytics, including user- friendly reports (PDO indicator 3) 1.Number of consultancies executed and/or being executed (IRI 1). Intermediate Results Indicators 4 of the 14 consultancies correspond to this objective. 1. Design an open data website to access information 2. Develop a plan for an Education Quality Index Key Outputs by Component 3. Draft new regulation for inputting, processing and using EMIS data (linked to the achievement of the Objective/Outcome 3) and the Strategic Intelligence System. 4. Trained stakeholders of the Red de Inteligencia (600 trained) Page 23 of 31 The World Bank Strengthening the Capacity to Produce and Use Quality Education Statistics (P163049) . ANNEX 2. PROJECT COST BY COMPONENT Amount at Approval Actual at Project Percentage of Approval Components (US$M) Closing (US$M) (US$M) Integration and improvement of 0.27 0.30 111 MINERD's information systems Training and dissemination of 0.18 0.15 83 information Operating Costs 0.05 0.05 100 Total 0.50 0.50 100 Page 24 of 31 The World Bank Strengthening the Capacity to Produce and Use Quality Education Statistics (P163049) ANNEX 3. RECIPIENT COMMENTS Context The ability to produce and use statistical data for decision-making in the education sector of the Dominican Republic (DR) continues to have several challenges, including: (i) integration of different data systems (school decentralization system, school management system, student learning evaluations, early childhood and educational systems); (ii) a quality control system for the production and collection of data that integrates protocols, responsibilities, training materials, quality audits of training data, and other procedures; and (iii) dissemination, analysis, and use of data, including open data initiatives that allow multi-stakeholder access to data in a user-friendly way. The results of this Project will help deepen changes in policies and decision-making through better availability of the use of information and statistical data in education. In addition, it will contribute to greater efficiency in the use of public resources through better informed decision-making and promote effective monitoring of educational trends. Project Description Objective. The main objective is to strengthen MINERD's capacity to produce high-quality data, consolidate data from different data systems and use that data to inform stakeholders in the education sector. Components. The Project consists of the following components: 1. Integration and improvement of MINERD’s information systems. Support for the improvement of SIGERD. 2. Training and dissemination of information. Strengthening MINERD's capacity to produce quality data, make it available to the public, and use it to inform decision-making and system authorities. 3. Project implementation. Support for Project administration, audits, procurement, financial management, and basic monitoring and evaluation activities to be carried out by MINERD. Relevance of Consultancies The Educational System of the Dominican Republic has suffered from the lack of integration of its main sources of information from each data production system, in addition to not having specialized information dissemination tools for different stakeholders. A challenge for any information system is to provide quality information. This implies that the different processes become more efficient: data collection, storage and processing, dissemination to all users of the educational system. Achievement of Objectives In this sense, consultancies were hired to strengthen the capacity to produce and use quality educational statistics. These consultancies contribute to meeting the three specific objectives of the Project. (i) Strengthen MINERD's capacity to produce high-quality data The consulting firm SSGRD Consultores was hired to carry out an audit of the Information System for Page 25 of 31 The World Bank Strengthening the Capacity to Produce and Use Quality Education Statistics (P163049) Educational Management of the Dominican Republic (Sistema de Información para Gestión Educativa de la República Dominicana, SIGERD) in order to verify the quality of the registered data and create a model that allows adjusting the existing sub-registry of information. The results of the audit provided evidence of the quality of the data recorded in the SIGERD, giving way to recommendations to strengthen the data collection process. Based on the recommendations for improvement, the consulting firm SSGRD consultores was hired once more to prepare a protocol to verify the quality of the data registry. The protocol was shared with the supervision area for its implementation with the instruments derived from each process. The instruments were tested and are available for implementation in the 2020-2021 school year. a. Instrument to validate the registration of enrolled students. b. Instrument to register new students in the education system. c. Instrument to validate the registration of teaching and administrative staff data. d. Instrument to validate the registration of services and infrastructure of the education center. It is important to highlight that, both in the process of auditing the data record in the SIGERD and in the creation of protocols for monitoring the quality of the recorded data, specialized technicians from the supervisory areas, private schools, were involved from the beginning. They were trained throughout the process. This allowed for the empowerment of all the key actors in the system, leaving capacity in each regional and educational district and ensuring the sustainability of the process. A total of 350 technicians were trained. (ii) Consolidate data from different systems The Information System for Educational Management of the Dominican Republic (SIGERD) has been improved by incorporating new functionalities and improving existing functionalities. The new functionalities and improvements to the SIGERD are divided into two fields: administrative and teaching field. Among the new administrative and pedagogical (student and teacher) functionalities: • Registration of final grades, will have an impact by reducing the administrative burden of the schools and by improving the quality of the service by reducing the waiting time of users, automating the process and consolidating the information. • Redesign of the student file will facilitate the registration of specific educational needs that students have so that schools can adjust the teaching-learning process. Applied at all levels and subsystems. • Administer schedules, will allow optimizing human management processes by having accurate information on the academic load of each teacher and verification of compliance with current regulations. • Administration of the adult education subsystem will allow an effective administration of the complexities of the new modular curriculum of the education of young people and adults. • Administration of decentralized resources will provide schools with an administrative accounting management tool to streamline the processes of execution accountability to the entire educational community. The improvements made to the existing functionalities of the SIGERD will provide schools with a set of tools that will support the management of administrative and student and teacher processes in order to make management more efficient by using quality information. Page 26 of 31 The World Bank Strengthening the Capacity to Produce and Use Quality Education Statistics (P163049) SIGERD evaluation An evaluation of the SIGERD database and source codes was carried out to optimize the system and estimate resource needs, making its large-scale implementation sustainable. The consultancy made recommendations for improvements to the system, which are being implemented by the Information and Communication Technology Office of the Ministry of Education. The educational system had different data sources that feed different systems that produce information and that were not integrated for its use and dissemination. The Project focused on the integration of information systems, the creation of data structures and the dissemination of information. With the consultancies carried out, the following was achieved: - Integration of information systems, with the creation of data structures of the different MINERD systems. A committee of users called "Institutional Intelligence Network" was formed, constituting the Integral System of Strategic Information. - Each user has access to the information they require using the Power BI application that connects to a data warehouse that is integrated with data structures. Information is classified into three access levels: strategic level for directors; tactical level for analysts who create reports in their operational scope; and operational level for all users who have access to public information. (iii) Use these data to inform stakeholders in the education sector Fifty-two workshops were held nationwide on information management through the Comprehensive Strategic Information System, which made it possible to provide around 2,300 managers and regional and district technicians with an efficient information tool. The purpose of the workshop was to train participants on the importance of producing and using quality information for effective decision-making, addressing the different sources of information and technological systems of the Ministry of Education. The creation of data structures (dataset) allowed the creation of more than six hundred information dashboards available to internal and external users, by classifying the information according to access roles within the operational scope of each user. The information dashboards allow the information to be viewed dynamically and broken down by variables of interest. The information can be exported to formats such as .pdf, .ppt, .csv, .xls. for better handling and manipulation of the data by users. Disclosure of information through specialized channels. Internal users have access to classified information through the Comprehensive Strategic Information System with the Power BI application. However, external users did not have such access, therefore, a strategic information web portal was designed and created. It is available at http://siie.minerd.gob.do . The portal is accessible to the public with dynamic information boards and different report formats available. The user experience was tested and well evaluated, complying with the normative indicator of the Basic Internal Control Standards (NOBACI). Around 600 technicians from the different MINERD areas were trained so that they could identify the data structure required in their work areas, as well as formalize the information that is produced so it could be shared in an integral way with the other areas. These trainings were carried out directly through the OCI. Page 27 of 31 The World Bank Strengthening the Capacity to Produce and Use Quality Education Statistics (P163049) Educational Quality Index An Educational Quality Index (�ndice de Calidad Educativa, ICE) has been created, which is informed by the information produced from the school and will allow evaluating the performance of the actors of the educational system through a set of indicators that are integrated into the ICE from the regional and educational districts. The ICE of the school is composed of three sub-indices, these are formulated with the widest source of data. The data involves all the actors that are part of student performance. The first and most important of the sub-indices assesses learning, access and progress of the student population of the school. The second evaluates the educational quality of the teachers of the school. And the third evaluates the administrative management of the school. Each sub-index represents a percentage within the ICE score. The main objective of the ICE is to establish a performance system from the school that allows evaluating the management not only of the school, but also the supervision and monitoring of the educational districts and the support of the regional directors. Creation of installed capacity at all levels of the MINERD structure Throughout the development of the Project, training was carried out for the management and technical teams of the different MINERD instances, the following table consolidates these conferences: Objective Topics Number of Workshops and Participants Implemented by Strengthen Perform an audit of the 10 workshops SSGRD capacity to data record in the SIGERD produce high- with the protocols for 350 information technicians and quality data monitoring the quality of educational supervision and private the registered data schools at all levels of the MINERD Use data to Produce and use quality 52 workshops OEI inform information for effective stakeholders decision-making, 2,300 managers and technicians at all in the addressing the different levels of the MINERD system. education sources of information sector and technological systems of MINERD Identify data structures for 20 workshops OCI use and / or to be shared in a comprehensive 600 technicians and focal points of the manner with other User Intelligence Network: central instances. headquarters, decentralized institutes, regionals and districts Page 28 of 31 The World Bank Strengthening the Capacity to Produce and Use Quality Education Statistics (P163049) World Bank Performance Rating General evaluation: Highly satisfactory Team Evaluation Justification World Bank Team Highly Since the beginning of the Project, the World Bank team, satisfactory both locally and from headquarters, accompanied us in putting together the Project with high quality standards and for it to be complementary to the loan Project that the country has to strengthen the education sector. The expertise of the Bank managers who were in charge was decisive in order to overcome the inconveniences that arose during the implementation of the Project. They helped us to find solutions, both with the regulatory authorities of our country, as well as to structure the Project in a way such that it could be feasible to implement and achievable. The World Bank team always followed-up on the execution of the Project and their efficient supervision of the Project allowed us to make the necessary adjustments to bring the Project to fruition. Ministry of Education Performance Rating General evaluation: Highly satisfactory Team Evaluation Justification Technical Planning Highly They made sure activities were concluded, meeting the and Statistics Team satisfactory indicators and added value with their experience, commitment and responsibility. The team adequately supervised the design, development and implementation of the SIGERD functionalities, the creation of the datasets, the web portal and all the processes in order to achieve a quality data. International Satisfactory The execution of the processes was good considering the Cooperation Team limitations that arose, which were overcome thanks to the (OCI) delivery of this team who gave adequate follow-up and proposed timely solutions to resolve obstacles in the process. MINERD Highly The involvement of the authorities in this Project was Authorities satisfactory undoubtedly the main factor for its success. It is not common for the authorities to be involved in all stages of the Project, however, the authorities were committed and promoted the achievement of the objectives of this Project. Page 29 of 31 The World Bank Strengthening the Capacity to Produce and Use Quality Education Statistics (P163049) Factors that affected implementation and results Positive factors • The involvement of both the authorities and the key actors of the educational system in each process was decisive in the success of the Project. • Having a Project leader helped a lot to follow-up with activities and with the fulfillment of the deliverables in the scheduled time. • Having the technological tools available facilitated the implementation of the institutional intelligence network. • Having good instructors facilitated the trainings and reduced the costs. Negative factors • Finding qualified personnel for the development of functionalities at SIGERD was a challenge to achieve the Project's objectives. • Timely availability of technological resources by the MINERD Technology Directorate was one of the processes could be improved. Page 30 of 31 The World Bank Strengthening the Capacity to Produce and Use Quality Education Statistics (P163049) ANNEX 4. SUPPORTING DOCUMENTS Statistical Service Group (2019); “Informe de Resultados del Estudio Auditoría de la calidad del dato en el Sistema de Información para la gestión de la educación (EMIS).� Organización de Estados Iberoamericanos OEI (2020); “Informe Técnico. Productos 4ta partida.� Organización de Estados Iberoamericanos OEI (2020); “Informe Técnico. �ndice de Calidad Educativa.� Organización de las Naciones Unidas para la Educación, Ciencia y la Cultura UNESCO (2016); “Informe de resultados Tercer Estudio Regional Comparativo y Explicativo. Logro de aprendizaje;� UNESCO Santiago, Oficina Regional de Educación para América Latina y el Caribe. World Bank (2020); “Proposed Project Restructuring;� Report No. RES41221. The World Bank, Latin America and Caribbean Region, World Bank. World Bank (2019); “Dominican Republic Learning Poverty Brief.� The World Bank. http://pubdocs.worldbank.org/en/212111571223351532/LAC-LCC3C-DOM-LPBRIEF.pdf World Bank (2019); “Proposed Project Restructuring;� Report No. RES35484. The World Bank, Latin America and Caribbean Region, World Bank. World Bank (2017-2020); “Implementation Status & Results Reports;� Seq No: 1-7. Washington, DC: World Bank. World Bank (2017); “Project Information Document (PID);� Report No. PIDC106339. Washington, DC: World Bank. World Bank (2017); “TFSCB Grant No. TF0A4773 Strengthening the Capacity to Produce and Use Quality Education Statistics Project Letter Agreement.� Washington, DC: World Bank. World Bank (2014); “Country Partnership Strategy for the Dominican Republic for the period FY15-FY18.� The World Bank, Latin America and Caribbean Region, World Bank. Page 31 of 31