Operational Note 1 Keeping it Simple: Supporting Government to Use Evidence to Understand Problems What matters in improving the quality of primary education? The government team working to answer this question in Belu, East Nusa Tenggara, thought that they knew how to improve education quality based on national government programming. However, a simple data analysis challenged their beliefs and pushed them to look for new answers and new approaches in improving education in their district. This operational note documents the problem-solving approach applied in Belu and highlights the importance of two factors in successful problem solving: (1) using evidence to understand the root causes of a problem, and (2) supporting government actors to do their own analysis. This operational notes series aims to share experiences and practical lessons from MELAYANI – Untangling Problems in Improving Basic Services (Menguraikan Permasalahan Perbaikan Layanan Dasar di Indonesia). MELAYANI is a program that builds local government capacity to address service delivery problems at the district level. It does so through helping district governments identify meaningful problems, break them down, analyze their parts, and develop and refine solutions. The methodology for problem solving builds on the problem-driven iterative adaptation (PDIA) methodology developed by a team at Harvard University. It focuses on building team ownership of problems and solutions, empowering local staff to innovate and experiment, using data to understand problems and their causes, and iterating to sustainable solutions. The program emphasizes that staff themselves must do the work to understand the problem and identify and implement solutions. MELAYANI provides tools to support the process, which is guided by a trained coach, who is supported by a mentor with expertise in the PDIA methodology. MELAYANI receives support from Australia’s Department of Foreign Affairs and Trade and is managed by the World Bank. Picking a problem that they are in fifth or sixth grade) and even in some matters cases struggled to sign their name on church documents. Belu, a poor, largely rural district in East Nusa Belu’s results on the National Exam (Ujian Tenggara Province on the border with Timor Nasional) are consistent with the stories told Leste,1 was one of three districts where the by local leaders. Only 32.8% of the district’s MELAYANI Program supported District staff to sixth graders were able to pass the test. At the identify and solve a service delivery problem. To same time, the district has very high rates of experiment and learn how to solve problems, graduation, indicating that students are passing the MELAYANI program encouraged districts without challenge from one year to the next. to select problems that were important to These numbers are of serious concern to the them, provided they were in line with national Belu education department, which sees them as goals. This allowed districts to think about their evidence that children are moving through the priorities as the first step of the approach. system without learning. In addition to the problem of education quality, government leadership, including the vice- regent and heads of key departments, identified high levels of maternal and neo-natal health as an additional challenge in Belu. To decide the priority issue that the program would work on, MELAYANI proposed a process in which several departments could put Belu is located in Indonesia’s poorer eastern region, on forward proposals based on an explanation of the border with Timor Leste the problems that they faced. These would be During initial meetings to determine Belu’s focus, discussed and ranked in small groups. Both the the elected head of Belu (Regent or Bupati) health and education departments presented expressed his desire to work on improving the problems that they wanted to address. Those quality of education. He had long felt strongly presented by health had considerably more data about the issue, including it as a key issue during and while education could identify their problem, his election campaign. He shared that religious they struggled to quantify it. In the discussion, leaders had told him stories of children who some individuals from the health department could not read or write well enough to prepare took this as an indication that education was for communion, (which is usually held when “not ready” to solve their problem. 1 Belu has a population of just over 350,000 people, of whom 80% are living in rural areas, based on population projections from 2017. It has a poverty level of 15.7%, based on World Bank staff calculations using BPS data from 2018. 2 The health department joined the meeting with far more people (they have “representation” in other departments, such as family planning) so were able to out-vote the education proposal. However, the Bupati intervened to request a focus on educational quality, since he felt it was more important. While the approach was designed to force a discussion, it did not take into account the imbalance in numbers of people. Given the strength of the arguments for the education MELAYANI coach facilitating analysis of obstacles to proposal, and clear demand from the Bupati improving education quality (who is critical to the authorizing environment), MELAYANI ultimately supported education. considered other elements of education, such as community spirit, religion and sport, but decided Defining the problem more to stick to the core set of skills as measured by carefully the National Exam. This decision was driven in part by the data that To address the problem of poor quality of they had available to them. There is limited education, MELAYANI supported the Belu district data available at district level to measure government to establish a team to work on the education quality. They saw the National Exam issue. Initially this was composed of a broad as comparatively unbiased, and thanks to recent selection of staff working on primary education, changes in test administration, less prone to including school monitors (pengawas) employed cheating. It focuses primarily on core educational by the district.2 The first question that they were outcomes (e.g., the three Ms). The alternative asked by the MELAYANI coach was what exactly was the yearly tests administered by the school. they meant by quality of education, and how These provide more information about the they might see or measure it. students but are considered unreliable; there is both potential and incentives for teachers to A facilitated discussion revealed that the team manipulate the exam or modify results (as they wanted education to deliver the “three Rs”: do not want to have failing students) and there Reading, (W)riting, and (A)rithmatic (or, more is greater opportunity for children to pass on accurately in Indonesian, the “three Ms”: a single strength (e.g., being good at sport but Membaca, Menulis dan Menghitung). The team unable to read). 2 District governments are responsible for grades 1-9, covering primary and lower middle schools (sekolah dasar (SD) and sekolah menengah pertama (SMP), respectively). Based on Law 23/20014, grades 10-12, or upper middle school (sekolah menengah atas (SMA)) are the responsibility of the province. 3 Understanding the problem: base pay. The proposed certification process was testing assumptions significantly watered down in the negotiation of the law and subsequent research has shown that the additional allowance has not contributed to Once the Belu team determined what they improved educational quality.4 However, the idea meant by quality education, the MELAYANI coach that teacher certification improves educational facilitated the group to use a fishbone diagram3 quality remains strong. While the Ministry of to further break down the issue and think about Education does not provide any direct incentives what factors they believed were contributing to to districts to increase certification, it does poor education quality. Many team members encourage it. Teachers themselves are highly believed that low levels of teacher certification motivated to push for certification, as it results and qualification (e.g., a university degree) were in a significant increase in their salaries. a major part of the problem. To start exploring the obstacles to improved quality further, the MELAYANI coach asked the team to look more carefully at the data that they had, to see what analysis would reveal. As of December 2017, they had one year of national exam results (2017), as well as basic data on teacher certification, qualification, civil servant status and school location. A member of the team from the data section of the Belu education department put all the data together and then ranked the schools from best to worst based on their National Exam scores. This Fishbone analysis of teacher quality analysis was undertaken with little support from This perception was driven in part by the fact that the MELAYANI coach. A staff member from the the national government’s predominant response data section was interested and wanted to work to the same problem of poor educational quality with the data but did not have the opportunity was the passage of the “Teacher Law” in 2005. within the scope of his job. The law rewarded teachers meeting certain criteria and certification with a “professional” (or Even the most basic data can shed light on certification) allowance equal to 100% of their the causes of problems, starting with basic correlations. In addition to correlating school 3 The fishbone diagram, also called an Ishakawa diagram, is a tool for brainstorming the cause and effect of problems. It facilitates sorting problems into useful categories. See https://asq.org/quality-resources/fishbone for more information. 4 See World Bank (2018) World Development Report 2018: Learning to Realize Education’s Promise. Available at http://www. worldbank.org/en/publication/wdr2018 This is consistent with research in Indonesia that shows increased teacher salaries (stemming from certification) has had no impact on teaching quality. See de Ree, Joppe et al (2017) Double for Nothing? Experimental evidence on the Impact of an Unconditional Salary Increase on Student Performance in Indonesia NBER Working Paper 21806 Available at http://www.nber.org/papers/w21806 4 performance with location within the district, education department was very focused on its the data team also looked at the proportion of overall district pass percentage on the National teachers in each school with certification. Exam, they had never looked at results at the school level in detail before. Since they had The findings from the data analysis shocked not taken the step of ranking individual schools the Belu education team. To begin with, based on performance, they did not know which conventional assumptions about the poorest schools were performing poorly or well. Ranking performance being in remote locations were schools can be politically fraught, particularly not borne out. The best-ranked secondary when there are recognized challenges with the school was in a location far from the city with National Exam (cheating was widespread, but no electricity, while worst-performing primary many feel it has been reduced). However, not school was in the middle of town. The findings ranking schools had left the Belu education also showed that teacher certification and department management in the dark as to what qualification were not correlated to the quality determines education quality. As a result, they had relied largely on assumptions, for example, that urban schools always perform better— because they have greater access to resources (teachers, money) and because urban-dwelling parents are more likely to be better off. Beyond just ranking the schools, looking at the correlation between certification and performance took Belu’s analysis of the causes Raimanuk SMPN in the remote interior of Belu district of education quality to the next level. It was not recorded the highest scores for Grade 9 on the national rigorous, relying simply on an examination of the exam numbers of certified teachers per school in the of student results on the test. In the words of highest and lowest ranked schools. However, it Pak Luhut, who conducted the analysis “the did make the team seriously re-think how they real issue is not access or infrastructure but saw the causes of the problem. While they know about the competence of teachers, which is not that pressure remains to continue certifying necessarily related to whether they have S1 (an teachers, they started looking elsewhere for key undergraduate degree).” drivers of educational quality. The explanatory power of While this analysis was successful, it is important basic analysis: using what is to sound a note of caution. First, the team’s available ability to rely on the data is due in part to improvements in quality that have taken place in recent years. While the team is aware that it There were several genuinely new things about is still imperfect, it can guide them to outliers. the analysis the Belu education team did as part Second, while the team conducted initial analysis of their problem-solving process. While the largely on their own, both they and the coach 5 needed some support to find the boundaries of encouraged them to learn more from their own use for their data. Appetites whetted by their schools. initial correlations, they were keen to carry on examining indicators, including some that were Simple analysis can be effective in helping to only tangentially related to their questions. It define problems. While the analysis that the Belu was important to help them stay focused on the team undertook was quite straightforward, it questions that they wanted to answer. showed that at the very least, they needed more information to understand factors influencing The next operational note will look at how the quality of education. It also showed them the team looked deeper into factors driving that some of their assumptions were incorrect. educational quality in Belu. From an operational perspective, simplicity is important. However, there is a fine line between What have we learned? “good enough” analysis and weak analysis, and oversight may be necessary. Poor, remote areas of Indonesia are just as A lack of data is not necessarily a sign that a capable of generating innovation as richer, department is not “ready” to solve problems. more capable locations. It can be easy to Indeed, it may be a sign that they need to turn stereotype areas like Belu as being low capacity their attention to identifying information that poor performers, but the achievements made by will help understand problems in a meaningful the Belu team demonstrate that with the right way. motivation they can make substantial progress. There are committed individuals in the Letting the Belu team do the analysis for bureaucracy, and the problem-solving process themselves was critical for both their can help them find a way to work in a more acceptance of results and their understanding meaningful way. The staff member from the data of the situation in their district. There is already section who led the data analysis, had previously evidence that teacher certification does not work been a high school teacher, but took a job with well in general and is particularly ineffective in the education department in the hopes he could Indonesia.5 However, it was important for the have a broader impact. He had been educated Belu team to discover this on their own, after a abroad in curriculum development and felt that careful consideration of what data they would he had something to offer. He had already been trust. Looking in detail at data about the schools looking through the National Exam data but had in their district also helped them learn more not found a way to present this to his superiors about their district, breaking down some of the without internal demand. The problem-solving stereotypes that they held about themselves, process gave him the opportunity to put his skills such as the belief that the children in town were to work and a place to share results. getting a better education than those in more remote areas. This change in perception also 5 Ibid 6 Leadership matters in setting priority issues. recognizing that the value of data may MELAYANI designed a more democratic be to rule out expected explanations as approach to selecting a priority issue. However, to why performance is not improving); the reality was that the chosen issue had to have 6. Identifying alternative (qualitative) leadership support. It is important to remember approaches to explore the obstacles to this in program development and balance it quality improvement, including focus with empowerment and engagement of teams group discussions and interviews with who will have to do the work going forward. frontline staff. Bringing these lessons to Central government can support quality scale improvement by local governments in the following ways: The sequence of steps followed by the Belu team Central government program managers can could be converted into a tool other districts support local government staff to explore their could use to undertake a similar analysis. The own solutions to challenges of service delivery tool would lead the government team through quality. While the national government provides the process of: the space for local governments to adapt national programs to local conditions, it is not 1. Defining their quality improvement always clear what that looks like as a process. objective and the indicator(s) to measure it; Central government program managers can 2. Using techniques like the fishbone put more emphasis on the collection of data diagram and five whys to unpack root for local management decision-making, as a causes of obstacles to improvement; critical first step in developing a more detailed 3. Assessing available data to determine understanding of local conditions. its suitability for enhancing understanding of the chosen problem Central government can create opportunities and its reliability in general; for staff from Belu to share their experience 4. Exploring simple ways to analyze data with other districts. This might include video including ranking and triangulating two interviews that can be shared on YouTube, data sources to look for correlations; it might also include national fora where the 5. Understanding the difference between staff from Belu are given space to present their correlation and causation (and experience. This operational note was written by Karrie McLaughlin with input from Kathy Whimp. Thanks to reviewers Rachel Lemay Ort, Jumana Qamruddin, Michael Woolcock and Noah Yarrow for feedback, as well as for the support and insight of World Bank Melayani team members Ahmad Zaki Fahmi and Noriko Toyoda. Acknowledgement and appreciation is extended to the Melayani coach in Belu Mikhael Leuape and the team in Belu who are working to improve the quality of education in their district. 7