Operational Note 2 Digging Deeper: Iterating Understanding of a Problem Thrown off by their discovery that teacher certification was not as well correlated to student performance on the National Exam as they initially believed (see Operational Note 1, “Keeping it Simple: Supporting Government to Use Evidence to Understand Problems”), the government MELAYANI team in Belu district set out to better understand what was impacting educational quality in their district. This operational note documents their journey as they worked to first reorient their thinking on key challenges and then dig deeper into determinants of quality education. The note highlights key challenges facing local government actors not only in defining and understanding the problems that they face, but in articulating them in a way that helps them continue to keep sight of their broader goals. It also highlights the seductive pull of practiced ways of thinking along program or hierarchical tracks, and the need to consciously exercise problem-oriented ways of thinking. This operational notes series aims to share experiences and practical lessons from MELAYANI – Untangling Problems in Improving Basic Services (Menguraikan Permasalahan Perbaikan Layanan Dasar di Indonesia). MELAYANI is a program that builds local government capacity to address service delivery problems at the district level. It does so through helping district governments identify meaningful problems, break them down, analyze their parts, and develop and refine solutions. The methodology for problem solving builds on the problem-driven iterative adaptation (PDIA) methodology developed by a team at Harvard University. It focuses on building team ownership of problems and solutions, empowering local staff to innovate and experiment, using data to understand problems and their causes, and iterating to sustainable solutions. The program emphasizes that staff themselves must do the work to understand the problem and identify and implement solutions. MELAYANI provides tools to support the process, which is guided by a trained coach, who is supported by a mentor with expertise in the PDIA methodology. MELAYANI receives support from Australia’s Department of Foreign Affairs and Trade and is managed by the World Bank. 1 Where to start? the district government staff to visit schools just to learn from their experience. Usually they visit schools with specific program-related questions While the government team in Belu district or monitoring objectives. Indeed, though there initially thought that insufficient teacher seemed to be agreement in principle that the certification was a key cause of poor learning team wanted to try to understand what was outcomes, they were willing to question their working to support student learning, group assumptions when faced with evidence that members struggled to articulate how they would there was little correlation between the presence approach the issue at a school, in part because of certified teachers and higher student test it was so different to their normal “monitoring” scores.1 approach. As a result, each member tried to make the goal more programmatically specific, In addition, an examination of test scores resulting in very different ideas about what the indicated that some rural schools were doing objectives of the visit should be. better than expected and several urban schools were performing surprisingly poorly. Indeed, the Ultimately, with some facilitation, the team secondary school with the highest scores was in decided to keep it very simple, with only two a rural area with no electricity and the lowest- questions that would allow teachers and scoring primary school was in the middle of headmasters to share their experiences: town, undermining many previous assumptions. This information made the group question their • What factors support you / your teachers understanding of the issue, and they decided to in teaching students well? (Faktor-faktor go to the schools to gather more information. apa saja yang mendukung keberhasilan guru dalam mengajar anak sekolah?) Though visiting a school to see what is or is not working might sound obvious, it is unusual for Where is Belu? Belu is located in East Nusa Tenggara Province, and sits on the border of Timor Leste. It has a population of just over 350,000 people, of whom 80% are living in rural areas, based on population projections from 2017. It has a poverty level of 15.7%, based on World Bank staff calculations using BPS data from 2018. Following local conflict, the district also has a population of displaced people. 1 See “Keeping it Simple: Supporting Government to Use Evidence to Solve Problems” in this operational notes series. 2 • What obstacles do you face in teaching/ In the low-performing schools, teachers were in supporting teachers to do well? less willing to talk, and when they did, they (Hambatan/ tantangan apa guru focused less on describing what they were mengalami selama mendorong proses doing in the classroom than the fact that they belajar mengajar?) felt demotivated and overwhelmed. There were often no staff meetings or supervision, and Visiting Schools where they were happening, they were focused purely on administrative issues. Teachers felt on their own in interacting with students. The team member that visited the best- and worst-performing schools was a staff member A new headmaster had recently been appointed from the education department’s data to one high-performing school, and teachers section, supported by the MELAYANI coach were able to compare between previous and the mentor. The results were illuminating, leadership, which had been supportive and highlighting both the importance of leadership transparent and the new headmaster, who had by headmasters as well as support systems for yet to meet with teachers several months into teachers. her job, or to share the school budget with them. The teachers in that school explained the The difference between the high- and low- (negative) impact of the change on their own performing schools was tangible. In better- performance and desire to work and learn. performing schools, teachers were keen to share their experiences and the new things that they Sharing and iterating the were trying, as well as being more comfortable problem – back to the discussing the challenges that they faced. They fishbone described routine staff meetings, as well as supervision and support from the headmaster which sometimes involved other teachers to The small team which had conducted the school discuss or problem-solve as a group. This helped visits shared findings from the visit both with teachers learn from each other, find and share the larger education team and in a whole-of- solutions. One teacher in a location without department meeting that included headmasters electricity noted, “if I go to Atambua [the district and teacher representatives. capital city], I try to browse (the internet) looking for new things I can do in the classroom. Then Within the team, the MELAYANI coach facilitated I share with the other teachers here.” One a more focused discussion on some of the issues high-performing school described their school that they had found in the field visit. In particular, monitor2 as being reasonably engaged, but this the group talked in more detail about what they was not the case in other schools. termed “teacher competency.” This idea covered 2 School monitors are employed by the district education department to provide technical support and oversight to groups of schools. In theory, they bring expertise both to headmasters and teachers and provide assistance through school visits and facilitating discussions across schools. 3 World Bank mentoring helped the MELAYANI coach to re-organize the fishbone diagram a number of concepts, ranging from individual education team decided that they wanted to motivation to teach, to school management and dig deeper into three issues they thought were leadership to testing practices. In all, the group affecting the quality of learning. had developed a list of 13 general concepts under the heading of teacher competency. The first issue they termed “studying and learning activities,” covering in-classroom activities. This At this point, the coach and team struggled a bit was the most exploratory of the three, as the with the process. Though the team had done group was intrigued by the findings uncovered some good thinking about different aspects by the open questions in the previous round of of the problem, they were having trouble school visits. They thought that there was likely seeing where to go next. Part of the problem more to learn. seemed to be grouping some of their ideas in order to interrogate them more clearly, as well The other two issues that they wanted to as identifying where they might have some explore were (i) the effectiveness of the overlapping underlying issues. The World Bank school committee and (ii) understanding provided mentoring support at this point, how administrative burdens (particularly the helping the coach create clearer clusters to take administration of school operational assistance, back to the team for discussion and to carry or Bantuan Operasional Sekolah (BOS))3 impact forward. Sub-themes were written on post-it on teaching. notes so they could easily be moved. In the case of school committees, there Digging deeper by engaging was a broad agreed understanding that the with schools committees were not functioning. While there seemed to be a general understanding as to why, the local planning department asked that the As part of the discussion around the restructuring team talk to school committees and document of their fishbone diagram, the larger Belu challenges so that they could more easily include 3 BOS is a capitation-based school transfer paid by central government to schools, to supplement funding from the district government. From a funds-flow perspective, BOS is transferred from central government to provincial governments, which then disburse BOS to the districts. BOS expenditures and detailed reporting of spending are regulated by a Ministry of Education regulation and there is parallel reporting on absorption by districts to the Ministry of Finance. 4 responses in their planning documents. This deeper problem of poor teacher attendance decision immediately raised an interesting (and which is unrelated to administrative challenges. illuminating) dilemma: how to talk to school committees? With these topics in mind, the team worked through the design of this next phase of their Over the past years, the Ministry of Education investigation of the problem, including: (i) the has stressed school committee independence, development of a basic set of questions that they as there were concerns that they were being wanted to ask, (ii) factors in choosing schools to used to collect funds for schools following the visit (including test scores, internally displaced abolition of school fees. While primarily aimed person (IDP) populations, access, and parent at schools, this has resulted in a wariness on the engagement), and (iii) how they would present part of district education departments to engage the work that they were undertaking to school with school committees on any topic, out of fear staff and parents. This discussion evolved quickly that they would be seen as out of line. In Belu, into an excellent consideration by the group on the education department had not formally how they might manage power dynamics (as spoken to a school committee, other than in education department staff coming to schools), cases of serious school problems, for years, if remember to listen (rather than give instruction) ever. Ultimately the issue of how to engage the and generally communicate effectively. school committee was addressed with a letter to the school and school committee explaining the Formalizing problem goals of the discussion with the team, but norms statements had to shift to even begin the conversation. Finally, the issue of BOS administration was The design of the MELAYANI approach drew pursued with a similar goal of documentation, on previous World Bank experience in working though early findings showed that the issue with local governments. That experience was overstated. The initial problem breakdown indicated that local government actors were had indicated that this was a challenge for all often interested in learning new things, but teachers, whereas initial school visits showed struggled to set out what they knew, articulate an impact only on the single teacher responsible clear questions, and link those questions to the for reporting. It became clear over time that problem that they were trying to address. this issue was both political and somewhat diversionary. It is political in that the Bupati While the PDIA process used in MELAYANI has a proposal to put an administrator in each provides clear steps to address these issues, first school, creating a new civil service position, and in asking the group to clarify their overarching so there is an implicit expectation that staff need problem and in subsequent use of the fishbone to identify this as a problem in order to justify diagram to break it down, the MELAYANI design their leader’s desired solution. It is diversionary team believed that a more detailed problem in that while there certainly are administrative statement would be necessary. In response, any burdens that pull teachers out of class (BOS team undertaking additional analytical work was administration being one of them), there is a asked to develop a “concept note.” To stimulate 5 the team to arrive at a clear and focused plan for those issues in particular and what they wanted their work, to the concept note format set out a to know. In particular, articulating how the sub- series of questions that included: problems linked to the overarching issue that they were exploring proved challenging. • What do you want to learn about? • Why is it important to this district? (e.g. One possible reason why articulating a clear how does it link to the “big problem” you narrative to link the sub-problems to the over- are trying to solve?) arching goal of improving education quality • What do you know already about this was that the team had chosen to look at three issue? issues, rather than just one. In the minds of • What are your key research questions? both the team and the coach, the issues became linked to the actors in the education system The concept note also included more detail to which they related. Teaching and learning on the methods the team would use to was about teachers, functionality of school gather information, the data they would use committees was about school committees and (including how they might choose respondents excessive demands of administration was about if necessary), who would undertake analysis and headmasters. This impacted how the team how, what participants would be told about the approached their questions: asking teachers work and any ethical considerations. In addition, about teaching, committees about their role, it asked the team to set out a timeline and detail and administrators about theirs. any resources that they might need. This focus on actors led to a tendency to list all While not part of the “pure” PDIA process, the of the potential problems related to each actor, concept note helped focus the team on what rather than stick to the tighter issues that they they were trying to achieve. The idea was that had agreed upon. The tendency to “name” the team could flesh out the concept note out as problems in a shorthand way by referring to the they went along, and it would help them clarify actors at the center of the sub-problem allowed the “story” behind their problem, while serving for a wide variety of interpretations of the as a tool to help them be clear about what they problem, what we might call “problem-creep”. wanted, and assist in communicating what they were doing. An example of problem-creep occurred when the team began to work through the issues A moment of confusion related to why the school committees are not about problem definition working effectively. When the team had originally discussed Interestingly, while the group was having good problems related to the school committee, discussions about detailed aspects of the sub- they were interested in how the committee problems that they had identified, both the supported a better school environment. This coach and the team struggled to set out a included helping ensure teacher attendance, succinct story about why they were pursuing and student discipline and working together 6 with the headmaster to make decisions in the There are several potential reasons why the team interest of the school by overseeing the use struggled as it did. The first is the dominance of of BOS funding (part of their formal role). In compliance orientation in public management particular, many schools in Belu hire teachers in Indonesia. Good performance is defined using BOS funds, a process which should involve as meeting regulations, and shortcomings in the school committee, but often does not. performance are assumed to derive from failure to comply fully with regulations. However, with the sub-problem now captured by the heading “school committee”, the focus Articulating specific problems of whole-of- shifted to whether the committee was doing system performance, rather than focusing on its job per the regulations, rather than on the programs to manage system components, effective function that it could perform, or what was not something that came easily to the was stopping it from being effective. Looking Belu education team. They were not used to back at the fishbone and remembering that articulating issues as problems and tended the lack of a school committee was not a stand- to slide back into programmatic thinking that alone problem, but a factor contributing to tended to be focused on a single part of the worsen or improve other problems helped the system. This is linked to the fact that from team sharpen their questions to focus better on the national to the local level, Ministries and school committee effectiveness. departments are highly siloed. Similar challenges emerged around “teaching Related to this, a considerable mental shift and learning.” As an exploratory issue, the topic was required for the team to think about these of teaching and learning had potential to link to system actors and the actions for which they many aspects of the classroom environment, so are responsible jointly contributed (or not) to a the group decided to tighten the definition of desired objective, such as supporting teachers to the sub-problem by focusing on what supported be effective in the classroom. Finally, there was a teachers to teach better. Initial interviews tendency to quickly “name” a problem in a way with teachers and headmasters showed that that made it legible within the bureaucracy, which important elements included support from was useful in terms of internal communication, headmasters and other teachers, as well as but risked losing sight of the problem the team potential for a larger role of school monitors. had originally identified. However, when the focus of the sub-problem was defined by the actor it related to—teachers— Bringing challenges to the other dimensions of the role of other actors life and understanding in supporting teachers to perform was lost, and “problems” issues such as certification and use of required tools such as planning forms (rather than the usefulness of existing tools), re-surfaced as the Though still struggling slightly with their problem focus of problem-definition. statements, the coach and team needed to 7 continue with the school visits that they had able to then follow up on emerging themes such decided upon. as supervision and feedback by headmasters, support from school monitors etc. On the Working with the World Bank mentor, the team third issue of administrative burden, the team undertook a very iterative approach to designing worked across each school visit to build a map the question guide for their visits. The group of the administrative processes by sharpening developed an initial question guide, tested it in questions around each step in the administrative two schools, spent a day looking at initial lessons process and develop an accurate timeline of and refining questions, and then repeated the how much time teachers were spending on process. This turned out to be the watershed administrative procedures. moment for them in understanding their own goals as well as thinking more clearly about The first round of discussion from the school how they would use the information that they visits served to clarify the team’s goals and gathered. questions, and by the second round two days later, the team was already considering the best The key element in the process was the debrief way to capture and present information so that of initial findings, particularly the question it could be best used to drive change. With this “Does the information we have gathered help us work done, the team used the tools in six more respond to our problems?” Having just gathered schools. considerable information from schools, the team had to really consider what they wanted What have we learned? to know. This was very new for team members, who were used to collecting information without much regard to how it would be used—including Breaking down problems is a skill, as is problem- writing down interview responses and then oriented thinking. Even where the group “giving them in” without thinking about whether understands key elements of a problem, finding the information was useful or not. a way through the details can be overwhelming. Grouping similar issues (for example, different The process of conducting school visits was aspects of teacher capacity development) or slightly different for the various issues. Following similar aspects of different issues (administrative each day of school visits, the team sat together challenges across the sector) can be challenging. and wrote onto sticky notes the key lessons or MELAYANI experience was that this skill proved new ideas about two of their issues: “teaching challenging for coaches too, and that coaches are and learning” and the role of school committees. likely to continue to need support and mentoring Following the first school visits, the questions on this front as they develop this skill. The use of, for school committees were revised to focus and continuous return to, the fishbone diagram more on understanding school-committee helps the government team to remember how collaboration. The questions around support for the details that they are focusing on link back up teachers in the classroom were not revised, but to their larger goals. the team started to group key issues, and was 8 Asking for a clear problem statement is Supporting the immediate use of data/ useful as a measure of how well a problem is information builds understanding and support. understood… The program request for a mini The approach of reviewing findings from the concept note illuminated challenges within field visits in developing the question guide was the team. This type of tool can be useful as a originally used because the team was so unsure measure of understanding without creating a of its direction. It ended up being an incredibly large administrative burden on a government useful approach, as it got team members team, assuming that the discussions that they immediately thinking about the implications of are having are sufficiently focused. It can serve as their work, rather than leaving it to the end of a useful red flag if the team is lost in the details. the “study,” as would normally be the case. The use of sticky notes of key findings was an easy … but problem statements should be dynamic way to visualize findings and start basic analysis and continue to be refined over time. As this by clustering. case demonstrates, developing a “perfect” problem statement should not be a goal in itself, There can be surprising value in the nor should it hold up the team from continuing documentation of problems. A potential criticism work, which may help them to tighten their of the Belu team’s process is that a lot of time and thinking. However, the process of refining a effort was spent documenting problems rather problem statement requires explicit discussions than working on solutions. Indeed, initially the with the team that help them continue to link mentors tried to encourage the team not to hold the details that they are finding/ working on to discussions about school committees but rather the broader issue that they are addressing. to move forward to find solutions for their lack of involvement. Interestingly, the request by How problems are described matters. Coaches the local planning body to document challenges and team members should try to balance being with school committees was itself instrumental specific and succinct as they talk about their in revealing the lack of engagement between the problems to reduce misunderstanding. There Belu education department and the committees is always a tendency to simplify complicated themselves. This proved to be a very important issues by the use of shorthand terms (e.g. finding which might not otherwise have teacher competency), but these shorthand surfaced. In terms of developing a solution, terms also potentially create confusion. In the documentation process became part of a addition, there is also a tendency to use words solution in the sense that it stimulated the first and descriptions familiar to the bureaucracy, step in establishing a relationship between the many of which frame actors (school committees, education department and school committees, teachers, headmasters) rather than system as well as stimulating development of support characteristics (quality of classroom learning) within the department for engaging the school as important. While using these concepts can committees in any way at all. be useful, generalizing problem statements permits co-existence of quite different concepts of a problem, both within the team and within government more generally. 9 How to scale up assistance as they help teams work through a problem. An external view can often be helpful in seeing the “red thread” The sequence of steps followed by the Belu team that ties issues together or issues that lack could be converted into a tool other districts clarity. could use if they are undertaking more detailed analysis of a problem. This tool, modeled on the Central government can support quality “concept note” designed for MELAYANI, would improvement by local governments in the help local governments sharpen their thinking following ways: and ensure that the analysis that they undertake is fit for purpose based on their own needs. Central government program managers can encourage local government staff to dig into Programs supporting local (or national) priority service delivery challenges. The role of governments through the use of coaches or central government program managers should facilitators can learn from the experience in be focused on supporting local governments to Belu. In particular, this case suggests: better understand their own local context and challenges in order to ensure that nationally • A need to provide coaches with training on mandated services can be delivered effectively. not only supporting the formulation of an overarching problem, but basic analytical Central government program managers can skills such as grouping information to help look for ways to explicitly fund the gathering of the team navigate through the details of information on key service delivery challenges. any particular problem. Looking more closely at local challenges takes staff time and operational money. This type of • The importance of ongoing, potentially work is currently difficult to budget and therefore on-demand, assistance to coaches for rarely undertaken. This operational note was written by Karrie McLaughlin with input from Kathy Whimp. Thanks to reviewers Rachel Lemay Ort, Jumana Qamruddin, Michael Woolcock and Noah Yarrow for feedback, as well as for the support and insight of World Bank Melayani team members Ahmad Zaki Fahmi and Noriko Toyoda. Acknowledgement and appreciation is extended to the Melayani coach in Belu Mikhael Leuape and the team in Belu who are working to improve the quality of education in their district. 10