RESULTS-BASED FINANCING RBF EDUCATION EVIDENCE COLOMBIA Can a Management and Information System Improve Education Quality? FEBRUARY 2019 REACH funded the development of an information system that foster improvement in the education system by providing data for the use of education practitioners and policymakers. The Results in Education for All Children (REACH) Trust Fund supports and disseminates research on the impact of results-based financing on learning outcomes. The EVIDENCE series highlights REACH grants around the world to provide empirical evidence and operational lessons helpful in the design and implementation of successful performance-based programs. Most countries have been more used by different schools, regions, successful in expanding access and programs. This makes it to education (getting children difficult for governments to design in school) than in improving the policies to address the most urgent quality of education (producing priorities and to help the schools better learning outcomes). While in greatest need and prevents x+y many factors contribute to learning coordination between the national, outcomes, it is difficult for schools regional, and local levels of education or governments to know which management. Furthermore, many By re-orienting the focus indicators of education quality to schools see M&E systems as towards results, well-designed focus on. To address this, many punitive rather than as a way to help monitoring systems can help countries have invested in monitoring them to improve. establish the preconditions for and evaluation (M&E) systems to implementing performance- collect and disseminate information These issues have limited how based incentives on the quality of education. However, useful these M&E systems have this information is often highly been in improving education quality. fragmented, with different indicators However, if well designed, an This note was adapted from “Developing an Effective Management and Information System for Education Quality in Colombia.” World Bank Group. Unpublished Manuscript. 2 RFB EDUCATION | EVIDENCE The Results in Education for All tool for stakeholders to understand Children (REACH) Trust Fund at the the strengths and weaknesses of World Bank funded the development the education system and take of a Management & Information appropriate action. System to monitor the quality of the education system in Colombia. This The scope of this monitoring system system builds on existing monitoring was defined using in the following tools, which focus on outcome three methods: (a) focus group measures such as test scores but discussions with national, regional, do not capture intermediate quality and local education stakeholders; indicators that can shed light on how (b) a review of the literature dealing learning outcomes are achieved. with the causal relationship between The overarching purpose of this various factors and education quality system is to foster improvement in other settings; and (c) quantitative education quality monitoring system in the education system by analysis to further investigate the can have many benefits, including informing the decision-making and relationship between these factors providing real-time information everyday activities of education and education quality in Colombia. on the current state of education practitioners and policymakers. This The results of these three exercises quality, improving managerial can be achieved by: (a) gathering were synthesized into questionnaires practices, identifying key areas detailed and relevant information aimed at principals, teachers, that need improving and how to about activities within schools students, and parents to conduct a do so, fostering coordination, and and (b) managing the information self-diagnostic assessment of each informing evidence-based policies efficiently and making it accessible school. The information in these and programs. By re-orienting to users to enable them to analyze, assessments was then posted on schools towards results instead of understand, and provide evidence- a digital dashboard that allows inputs, such a system can also help based recommendations on how principals and teachers to explore to establish the preconditions for to improve education quality. This an individual school’s results and implementing performance-based monitoring system is not intended to enables government staff to view incentives at the school, local, be an accountability mechanism for data aggregated to the local, regional, regional, or national level. schools but rather a management or national level. CONTEXT stable as a share of GDP since then.1 for students from the lowest income However, while these investments quintile was 385 compared to 461 have led to modest improvements for those from the highest income Colombia has recently undergone a in the performance of Colombian quintile.3 radical social transformation, with students on the Saber and PISA the historic Peace Agreement with standardized tests, Colombia remains While the national government FARC in 2017 that improved the internationally uncompetitive, with through the Ministry of National prospects for sustained peace, and 38.2 percent of students failing to Education (Ministerio de Educación several important macroeconomic, structural, and social reforms reach baseline proficiency in 2015.2 Nacional, or MEN) is in charge of all have also been implemented since There is also significant inequality public policy related to education, 2003. Investment in education has in scores between rural and urban regional authorities are responsible increased significantly, rising from areas and between high-income for the administration of educational 2.7 percent of GDP in 1990 to 4.5 and low-income households. For services, and schools themselves percent in 1999, and then remaining example, the average science score are directly in charge of education COLOMBIA 3 provision. Since 1991, the education diverse population, it makes it for their learning results but have system has been moving towards difficult to ensure the quality of little control over many of the a more decentralized structure, the curriculum and pedagogy in factors that contribute to the giving schools more decision- all of Colombia’s 57,000 schools. achievement of these results. Given making power over curricula and Meanwhile, schools have very little this situation, there is a critical pedagogy. While this approach autonomy in the management need for a coordinated system for was taken to ensure the provision of education resources (human, gathering information about schools of diverse pedagogies to meet physical, or monetary). As a result, to be used to design and target the varying needs of a widely schools are being held accountable interventions to improve learning. WHY WAS THE for data requests. There are 150 disconnected information systems schools and policymakers can be more effective in setting specific INTERVENTION within Colombia’s MEN, and most deal only with enrollment, access, objectives, in selecting which interventions to prioritize to achieve CHOSEN? and learning outcomes rather than the more detailed indicators that these objectives, and in deploying resources effectively to achieve the would shed light on what factors desired results. The Management System for contribute to education quality. As Education Quality (Sistema de The first step in the development of a result, while schools currently Gestión de la Calidad Educativa, or the SIGCE was to define its scope. have access to data on their own SIGCE) in Colombia was developed This involved defining: (a) the purpose learning outcomes, they have little to solve two related problems in the of the system as a management tool information about how to improve education system in Colombia— to inform everyday decision-making; them. Similarly, local, regional, information fragmentation and (b) the intended users (schools, local and national governments lack a a lack of coordination among governments, and MEN officials); system-wide view of what schools stakeholders—both of which limit (c) what information would be need or how to devise policies to the extent to which the education included; and (d) how best to address those needs. Furthermore, system can be improved. The way present this information to different even when data are readily available, in which education ministries and types of users. technical staff may lack the local governments are typically expertise or management tools to organized makes it difficult to analyze them effectively. coordinate programs that span across units of the government, To address these problems, the and they have few incentives to SIGCE was designed to provide coordinate with each other. Each coordinated information to all local government in Colombia has stakeholders within the education a different development plan with system to enable them to track different objectives, which may the system’s progress and results, or may not be aligned with the develop improvement plans at the objectives of the MEN. This leads school level, and design and prioritize to fragmented programs, uneven policy interventions at the national implementation, and a lack of policy level. A critical feature of the SIGCE effectiveness. These different units was that it was designed with the also typically get their information aim of focusing the entire education from different sources for different system on results. By using the tool topics, often with long lag times for planning as well as monitoring, 4 RFB EDUCATION | EVIDENCE HOW DID THE would be included in the monitoring system, and the extent of the assessments to construct values for each indicator for each school. INTERVENTION resources available to implement it. The third phase involved presenting the information captured in these WORK? The initial output of this three-part exercise was a comprehensive indicators to all relevant stakeholders in a digital dashboard, which was To determine what information to list of variables, categorized into designed to be comprehensive include in the system, “education 6 major dimensions, 25 areas, while at the same time user-friendly quality” was defined not just as and 39 sub-areas. The variables and easy to understand and to use learning outcomes but also as all were prioritized based on their for practical decision-making. The of the factors that cause learning demonstrated impact on education dashboard includes customized views outcomes. Three methods were quality, the feasibility of collecting for different types of users, giving an used to select which of these them, their ability to be easily overview of the Colombian education factors would be included in the understood by any actor, and their system across the six dimensions of monitoring system. First, focus lack of potential self-reported bias. education quality and more detailed group discussions were held Each variable was then mapped to information on each area, sub-area, with representative samples of a specific question to be posed to and indicator. The dashboard also principals, teachers, students, principals, teachers, students, and allows government staff to develop parents, government officials, and parents, for whom corresponding detailed analyses by education level, other education stakeholders to questionnaires were then developed. school schedule, urban or rural gather their input on which variables Furthermore, thresholds were set location, and municipality to inform should be included based on their for each indicator that identified the policymaking. relevance to the participants and on desired level for a school to achieve how feasible it would be to collect as well as the level that would trigger them. Second, a literature review an alert. The final output of this of recently published academic papers was conducted to gather the process was an indicator list with clearly defined logical links between WHAT WERE best available information on which variables already have been proven the various dimensions’ areas, sub-areas, variables, questions, and THE RESULTS? to have a causal relationship with indicators of education quality. The first key result of the education quality in other settings, focus group discussions was using standardized test scores as a After the scope of the monitoring the construction of six major proxy for quality. Third, quantitative system had been defined, the SIGCE dimensions to encompass all analysis was done to test hypotheses was implemented in three phases. aspects of educational quality. about which variables have a The first phase involved each school Through the focus group discussions statistically significant relationship developing a self-diagnosis using the held with principals, teachers, with student academic performance questionnaires developed from the students, and parents, the six in Colombia, using Saber and final indicator list. It was determined dimensions of education quality PISA standardized test scores that principals, teachers, students, were defined as: (a) Teachers, and Colombia’s Synthetic Index and parents would be the best which captures variables related to of Education Quality as outcome sources of relevant information for the professional development and measures. In defining the scope of the monitoring system, and, therefore, continuous improvement of teachers the monitoring system, it was also these stakeholders were tasked and school staff; (b) Academic necessary to identify which types of with completing the self-diagnostic and Pedagogical, which captures stakeholders and how many of each assessment of each school. The variables related to the process of type would be included in the focus second phase involved analyzing the knowledge acquisition at the student groups, the depth of information that information collected through these level; (c) School Environment and COLOMBIA 5 Wellbeing, which captures variables • All respondents believe that education quality across the six related to the emotional and physical complementary services for dimensions. See Table 1. environment of schools; (d) Family, school communities are lacking. School, and Community, which The quantitative analysis confirmed • All respondents believe that captures variables related to the that several of these variables school buildings are in relationship between the school and have a significant relationship with poor condition. its local community; (e) Management education quality in Colombia. and Administrative, which captures • Principals believe there is a Data from several databases variables related to the smooth significant lack of administrative within Colombia were analyzed functioning of the school and the personnel in schools. to measure the relationships continuity of the learning process; between education quality variables The literature review identified and student performance, while and (f) Infrastructure, which captures several types of interventions controlling for student and school variables related to the quality of that have been proven to improve characteristics that are known schools’ physical infrastructure. The focus group discussions also yielded some key findings regarding schools’ strengths and weaknesses: TABLE 1: Interventions that Improve Education Quality Dimension Intervention • Principals and teachers believe that the curriculum needs to be Teachers improved to strengthen schools’ Teacher profile academic performance. Teacher training • Students believe that their Learning communities teachers have strong Teacher assessment classroom skills. Academic and Pedagogical • Parents believe that the Educational project pedagogic and academic Curricular design dimension in schools is weak. Classroom practices • Teachers and principals consider Assessment themselves to be a strength in their Family, School, and Community schools, and a key element for Community participation on school activities achieving excellent school quality. Community participation on school government Articulation with external programs • Students believe that their School Environment and Welfare teachers’ profiles are the main Management for school climate cause of both the advantages Learning environments and difficulties that they face in school. Welfare promotion Additional services • Parents believe that schools Risks and emergency measures are mostly inclusive and Management and Administrative non-discriminatory. Human resources • Students believe that the school Expenses management climate is one of the most Infrastructure positive aspects of their learning General structure (location, infrastructural conditions, services, security) experience. Equipment (comfort, spaces, furniture) 6 RFB EDUCATION | EVIDENCE Table 2: Relationship Between Education Quality Variables and Student Performance to predict performance. The outcome measures used were Dimension Variable Outcome Coefficient the standardized test scores of Teachers Colombian students on Saber and PISA and Colombia’s Synthetic Index % Teachers have graduate degrees ISCE 0.049** of Education Quality (Índice sintético Academic and Pedagogical de calidad de la educación or ISCE). Teachers explain scientific ideas PISA 0.196*** The relationships listed in Table 2 Teachers discuss students’ questions PISA 0.273*** were identified, with statistically Teachers demonstrate an idea PISA 0.159*** insignificant results reported as not applicable (N/A). Family, School, and Community Parents participate in school PISA N/A It is important to mention that School Environment and Wellbeing other dimensions also affect the Students do not listen to teachers PISA -0.194*** quality of education, such as the There is noise and disorder in classrooms PISA -0.084*** sociodemographic characteristics Teachers wait for students to quiet down PISA -0.124*** of students and their families, the Students cannot work well PISA -0.235*** location and the surroundings of Students do not start working promptly PISA -0.142*** schools, and the academic and Saber (Math) -0.118*** professional experience of teachers Students are robbed and principals. However, because Saber (Reading) -0.178*** schools are not able to influence Saber (Math) -0.110*** Students are insulted by classmates these factors directly, principals Saber (Reading) -0.093* and teachers should consider these Saber (Math) -0.122*** Students are excluded from group dimensions simply as contextual Saber (Reading) -0.130*** information, but regional and Students are hit, slapped, kicked, shoved, Saber (Math) -0.107*** national policymakers could use or pinched Saber (Reading) -0.089*** them in to develop comprehensive Saber (Math) -0.110** Number of times teachers are absent public policies. Saber (Reading) -0.126** Saber (Math) -0.099** The final result of the focus groups, Number of times students are absent Saber (Reading) -0.118*** literature review, and quantitative Saber (Math) -0.025 analysis was an indicator list to Number of times teachers are late Saber (Reading) -0.155*** be included in the SIGCE. The final Saber (Math) -0.079* SIGCE indicator list consisted of 604 Number of times students are late Saber (Reading) -0.073* indicators that were then included Management and Administrative in questionnaires to be distributed to the principals, teachers, students, Student-teacher ratio ISCE, PISA N/A and parents who would conduct Infrastructure the self-diagnostic assessments of Percentage of students with electricity ISCE 0.056*** their schools. The list included 51 Percentage of students with water ISCE 0.084*** indicators in the Teachers dimension, Percentage of students with gas ISCE 0.054*** 163 in Academic and Pedagogical, Poor quality physical infrastructure PISA -0.141** 51 in Family, School, and Community, 208 in School Environment and Lack of educational material PISA -0.131** Wellbeing, 29 in Management Poor quality educational material PISA N/A and Administrative, and 102 in * p<0.1; **p<0.05; ***p<0.01 Infrastructure. COLOMBIA 7 WHAT WERE information contained in the SIGCE would prove to be valuable to both caused by the large number of information requirements from public THE LESSONS schools and governments. No matter how accurate and carefully and private institutions that schools must fulfil. If the new dashboard can LEARNED? measured the data were, they would not be used widely unless they were solve this problem, it will be a valuable management tool for them. While the process of developing the accessible and presented in an easily SIGCE was careful and rigorous, understood visual interface. Several One of the main conclusions of an information system alone is interfaces were developed to allow the focus group discussions with not enough to guarantee results— users to see the data presented principals, teachers, students, and tools must be used. Thus, it was in different ways, and easy visual parents was that schools should also crucial to actively promote alerts were designed to help users to use the SIGCE to identify their own the use of the system in planning interpret the data, such as red-yellow- strengths and weaknesses taking and implementation Furthermore, green stoplights to put a school’s into account the perceptions of since the SIGCE relies on schools to performance on each indicator in the students, parents, teachers, and conduct self-diagnostic assessments, appropriate context. principals. Therefore, SIGCE can be the entire system depends on used by schools for self-reflection schools taking this process seriously. Throughout all of the steps of the and evaluation, rather than looking It was critical to ensure that school development process, feedback was towards external entities to provide managers understood that the sought from all relevant stakeholders feedback and develop action items. system would help them to plan and on the indicator list, the logical flow from dimension to indicator, the Finally, it is important to highlight to elicit more effective responses design of the dashboard, and other that, during the development of from local and central governments features of the system to test and the SIGCE, teachers and principals to address any shortcomings that validate the monitoring system and expressed an interest in continuing were identified so they would see to ensure that the system’s users to use the SIGCE in the future and that the information that they entered were on board. in receiving technical assistance to into the SIGCE would produce useful generate action plans to improve results for them. Principals and teachers pointed education quality at the beginning of The design of the dashboard out that the SIGCE can also reduce the following school year. was also vital to ensure that the the administrative burden on them CONCLUSION yield many benefits. First, it will improve school management by Creation of the SIGCE This intervention has successfully helping schools to identify their own management system strengths and weaknesses and to created a management and develop school improvement plans. establishes the information system for the public management of education in Second, it constitutes a key source necessary precondition of information for evidence-based Colombia. The system—the policymaking, helping policymakers for including SIGCE—is capable of collecting large amounts of school-level at all levels of government to identify the factors that help to improve performance-based information at a relatively low cost and of aggregating and presenting learning outcomes and to establish incentives in Colombia. policies and programs that will help this information in a user-friendly schools to improve in those areas. way. This system is expected to Third, it will help policymakers at all levels of government to monitor both intermediate indicators of incentives in Colombia, for example, the progress of these policies and education quality and final learning through the provision of incentives programs and to assess how well outcomes. Sixth, it could potentially to local governments based on the they are meeting schools’ needs. promote transparency in the public performance of schools in their Fourth, it will increase coordination management of the education municipalities. This performance- between schools and the local, system by making information based approach is likely to result in regional, and national education available to the public, including the more efficient use of resources authorities in managing and parents and other members of by making it possible to target those improving education quality. Fifth, school communities. Finally, the schools and interventions that have it will lay the foundation for the creation of the SIGCE establishes the greatest potential for improving results-based evaluation of education the necessary precondition for education quality. policies based on their impact on introducing performance-based 1 Piñeros (forthcoming) in Low and Middle-income Countries: a Systematic 24 Snilstveit et al (2015). 2 PISA 2015 (OECD). Review.” 3ie systematic review 24. International 25 Krishnaratne, S., H. White, and E. Carpenter (2013). Initiative for Impact Evaluation (3ie), London. “Quality education for all children? What works in 3 PISA 2015 (OECD). 10 Murnane and Ganimian (2016). education in developing countries.” Working Paper 20. 4 Evans, D. and A. Popova (2015). “What Really Works 11 Glewwe and Muralidharan (2016). International Initiative for Impact Evaluation (3ie), New to Improve Learning in Developing Countries? Delhi. An Analysis of Divergent Findings in Systematic 12 Evans and Popova (2015). Reviews.” World Bank Group Policy Research 26 McEwan (2015). 13 Glewwe and Muralidharan (2016). Working Paper 7203. 27 Murnane and Ganimian (2016). 14 Snilstveit et al (2015). 5 Murnane, R. J. and A.J. Ganimian (2016). “Improving 28 Snilstveit et al (2015). 15 Asim, S., R.S. Chase, A. Dar, and A.D. Schmillen educational outcomes in developing countries: Lessons 29 Asim et al (2015). (2015). “Improving education outcomes in South from rigorous evaluations.” Review of Educational Asia: findings from a decade of impact evaluations.” 30 Krishnaratne et al (2013). Research, 86, 719–755. Policy Research Working Paper No. WPS 7362, 31 Glewwe et al (2011). 6 Evans, and Popova (2015). Impact Evaluation series, World Bank, Washington 32 Glewwe and Muralidharan (2016). 7 Glewwe, P. and K. Muralidharan (2016). “Improving D.C. Education Outcomes in Developing Countries: 33 Snilstveit et al (2015). 16 Snilstveit et al (2015). Evidence, Knowledge Gaps, and Policy Implications” 34 Murnane and Ganimian (2016). 17 Glewwe, P., E. Hanushek, S. Humpage, and R. Ravina in E. Hanushek, S. Machin, and L. Woessman 35 Glewwe, P. and Muralidharan(2016). (2011). School Resources and Educational Outcomes (eds.). Handbook of the Economics of Education, pp in Developing Countries: A Review of the Literature 36 Snilstveit et al (2015). 653–743. Elsevier, Amsterdam. from 1990 to 2010. Cambridge, Mass: National 37 McEwan (2015). 8 McEwan, P. J. (2015). “Improving Learning in Primary Bureau of Economic Research. Schools of Developing Countries: A Meta-Analysis 38 Murnane and Ganimian (2016). 18 Glewwe Muralidharan (2016). of Randomized Experiments.” Review of Educational 39 Glewwe and Muralidharan (2016). Research, 85, 3, 19 Asim et al (2015). 40 Murnane and Ganimian (2016). 353–394. 20 Snilstveit et al (2015). 41 Glewwe et al (2011). 9 Snilstveit, B., J. Stevenson, D. Phillips, M. Vojtkova, 21 Murnane and Ganimian (2016). E. Gallagher, T. Schmidt, H. Jobse, M. Geelen, M.G. 42 Krishnaratne et al (2013). 22 McEwan (2015). Pastorello, and J. Eyers, J. (2015). “Interventions for Improving Learning Outcomes and Access to Education 23 Glewwe and Muralidharan (2016). PHOTO CREDITS: Cover: “Students in a technical education prsogram” by World Bank/Charlotte Kesl, license: CC BY-NC-ND 2.0 Page 2: “A rural secondary school in La Ceja del Tambo” by World Bank/Charlotte Kesl, license: CC BY-NC-ND 2.0 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) worldbank.org/reach REACH is funded by the Government of Norway through NORAD, the Government of the United States of America through USAID, and the Government of Germany reach@worldbank.org through the Federal Ministry for Economic Cooperation and Development.