34222 Applying Mixed Methods Research to Participatory Development Projects and Local Conflict Mediation: A Case Study from Indonesia1 Patrick Barron, World Bank Rachael Diprose, University of Oxford Claire Q. Smith, London School of Economics Katherine Whiteside, Brown University Michael Woolcock, University of Manchester 21 January 2008 Abstract An enduring lesson of social theory is that periods of institutional and political change are often associated with conflict. It is less well understood, however, whether and how development projects help, hinder or are incidental to these processes of change. This paper summarizes the methodological strategies underpinning a study designed to assess the ways in which a large participatory development project in rural Indonesia influenced trajectories of local conflict. Drawing upon a range of coherently integrated qualitative and quantitative research methods, it shows how, where and in what order these methods were combined to yield an innovative array of empirical data on which to base assessments regarding the nature and extent of the project's impact on prevailing local conflicts. We conclude by offering some lessons from our study, and suggestions for others contemplating large-scale mixed methods research on complex issues. 1The views expressed in this paper are those of the authors alone, and should not be attributed to the various organizations with which they are affiliated. We are grateful to the many individuals and funding agencies that have supported this work and provided detailed feedback on it, but our particular thanks to Scott Guggenheim for giving us the opportunity to undertake the study. Email address for correspondence: pbarron@worldbank.org 1 I. Introduction This paper summarizes the methodological strategies and choices employed in a study examining the nexus between local conflict and development projects in rural Indonesia. The study sought to: (a) understand the trajectories that local level conflicts take, and the factors that lead to resolution, stalemate, or escalation; (b) examine the impact of the Kecamatan Development Project (KDP), a massive (US$1 billion) World Bank-financed community-driven development project,2 on the ability of communities to manage conflict; and on that basis (c) make recommendations to those designing and implementing related participatory development and conflict mediation projects in Indonesia (and, where relevant, elsewhere).3 Because the aim of the study was to uncover program impacts, it grappled with the difficult empirical task of generating evidence to support causal inference. Thus beyond measuring any changes in conflict management capacity, the evaluation sought to directly attribute those changes to their causal sources, disentangling direct and indirect KDP program effects from other forces. The fundamental technical challenge of impact evaluation involves estimating the counterfactual state of participants, or what would have happened to participants had they not benefited from the project. For this study, participants include individuals, villages and sub-districts, generating multiple units of analysis. Since the counterfactual state is by definition unobservable, researchers use a variety of methodological approaches and the investigation of similar non- beneficiary areas to approximate it. While the most empirically rigorous approach involves random assignment to participant and control groups,4 KDP and many other development initiatives do not assign program participation in this way. Thus the first research challenge was to develop a methodology that could isolate KDP impacts when participation had already been predetermined. In addition, since the research began after the program started, the study had to compensate for the lack of baseline data (for example, through building historical narratives and asking recall questions). Beyond the basic empirical challenges, focusing on the subject of conflict added further layers of complexity. Conflict is a sensitive topic, one people frequently feel uncomfortable discussing. Citizens and authority figures may face strong and multiple incentives to remain silent, leading to underreporting problems. Violence raises further practical and ethical concerns, including protecting the safety of field staff and ensuring that the research process does not reignite or exacerbate existing disputes. Conflict is also a messy topic, enmeshed in a range of contributing factors. In Indonesia, the dynamics of transition at the macro level--from an authoritarian state to the foundations of a democratic one--provide the broader context to many of these conflicts. During the thirty years of New Order government, a `vertical' institutional state structure forcefully repressed sources of conflict, but at the tragic expense of the human rights of Indonesians. When the regime--with its system of control and intimidation--fell, long repressed tensions rushed to the surface. At the 2For a summary of KDP, and the thinking behind it, see Guggenheim (2006). 3The full methodological details are outlined in Barron et. al. (2004). Results from the study are summarized in Barron, Diprose and Woolcock (2007). Other papers from the study can be found online at www.conflictanddevelopment.org 4On the limits of randomization as an evaluation strategy and tool, see Ravallion (2005). 2 time, the police, military, judicial and legislative systems were either too weak or too compromised to contain the ensuing violence. Recent decentralization and regional autonomy laws, accompanied by ongoing democratization, have set the country on a path that has the potential to dramatically change the nature of Indonesian society (Friend, 2003). Yet a fully functioning democracy has still to take its place, and a conflict management vacuum is still to be filled. At the local level, ethnic, linguistic, religious, and other diverse group identities, economic inequality, and disagreements over resources and values contribute to social tensions. Without strong local level capacity to manage conflicts, social tensions have the potential to erupt into violent conflict. In this complex setting, we wanted to understand the specific forces that instigated or mitigated conflict, and then to isolate the impacts of KDP programs and processes from those other factors. As a relatively new research field, there were few existing models to draw upon to help us in our task. Previous research had tended to focus on large-scale conflicts (e.g., civil wars), and to deploy either single case study approaches (making broader generalizations problematic) or national-level, cross-sectional quantitative data (which deal inadequately with issues pertaining to sub-national variation, context, and process). Investigating causal relationships regarding a sensitive, multifaceted phenomenon logically lent itself to an integrated, mixed method research design. Thus this study used a quantitative sampling frame to permit generalizations, isolate program effects, and support causal assertions. The fieldwork combined qualitative methods (that explored issues in depth, built trust to facilitate sensitive discussion, and that delved into processes) with quantitative fieldwork and secondary data analysis (which collected general survey data that documents broader trends, enabling cross-village comparisons, and allowing us to test the generality of hypotheses). Integrating the two methodological strands generated a more thorough, valid and reliable understanding of conflict and KDP's relationship to it than either approach could have done alone. While the focus of the study was to evaluate the evolution of conflict trajectories at the local level, and KDP's impact on conflict management in rural Indonesia, a complementary goal was to develop and share innovative research methods and tools with other interested researchers and development practitioners. However, the research we describe cannot be vacuum-packed and parachuted into new locations. The endeavor evolved over several years and relied on the strengths, resources, and experience of many contributors. We cannot overemphasize the importance of serious, extended consideration of the local context, and a deep commitment to training and engaging all levels of the research team.5 Thus while much of the methodological framing and training materials can (we hope) be applied to other contexts, successful replication--in other parts of Indonesia, or elsewhere--will require fundamental adaptation of the research hypotheses, questions and field tools to local situations. II. Research Questions The research strategy aimed to let the questions drive the methods (Mills 1959). As such, we first identified important, interesting and researchable questions; generated specific hypotheses 5The full set of training materials used to prepare the quantitative research teams (see Barron, Diprose and Smith 2004) is available online to those considering or already engaged in similar studies elsewhere. 3 to test; and then selected the most appropriate methods to yield fruitful answers (see Rao and Woolcock 2003). Here we sought to understand the mechanisms by which participatory development projects help villagers manage local conflicts. Specifically, we wanted to know whether, how and under what conditions KDP impacted local conflict management capacity. Programmatically, we predicted that KDP may influence local conflict management because it requires communities to submit proposals for grants to a committee of their peers, who evaluate them on their technical merit, likely impact and sustainability, and prioritize them by local needs. The process incorporates previously marginalized groups in the decision-making processes. KDP thus establishes "new (and, by design, inclusive) community forums for decision making on key issues" (Wetterberg and Guggenheim 2003). Yet KDP is likely only one piece of a large and complicated puzzle that influences local conflicts and their impact, duration, and management. Thus in seeking a more precise understanding of the influence of KDP, we also had to investigate and understand a number of other localized factors. Specifically, the study methods and design responded to the following three research areas; each had its corresponding research questions and specific hypotheses to test.6 1. Localized factors that influence levels of violent conflict The first research question was: What are the main factors that affect local level capacity to manage conflict? We hypothesized these would include: (a) economic and structural factors that we predicted may determine the parties involved in conflict; (b) psychological and cultural factors, which may explain both the types of actors involved and the intensity of conflict; and (c) social and political institutional factors that potentially dictate the extent to which the first two sets of factors develop in ways that either promote or prevent violent conflict. 2. Forms of civic interaction that help resolve or manage conflict The next group of research questions included: How important is the nature and extent of interaction between different groups, and between those groups and the state, to local conflict mediation? How are boundaries between different groups constructed and sustained? Here we tested hypotheses regarding: (a) the potential role of inclusive, inter-ethnic group institutions in reducing conflict; (b) the relative importance of formal versus informal associational activity across ethnic groups in influencing level of conflict; (c) whether the lack of bridging forums within multi-ethnic communities means inter-ethnic group conflict tends to trigger violent conflict; and (d) whether traditional institutions restrict conflict management capacity when they do not include other ethnic groups in decision-making.7 3. The relationship between KDP and community conflict management The last set of research questions and hypotheses turned specifically to the role of KDP, asking: Does KDP help communities manage conflict more constructively? More generally, can external agents help establish more inclusive, transparent, and accountable local level institutions 6These hypotheses were generated at the beginning of the study on the basis of our reading of the literature, prior field research over the previous year in four provinces (Lampung, Central Kalimantan, East Java, and NTT), impressions from field staff regarding the nature and extent of KDP's impact, and our sense of the specific local conflict mediation impacts that KDP could, at best, be expected to generate. They were refined as the study unfolded. 7While our original hypotheses focused on inter-ethnic group interaction and conflict, the research found that inter- group interaction in a wider sense was equally important. 4 for mediating conflict? If so, for what types of conflict cases, and under what conditions? More specifically, for what types of cases does KDP have a positive impact? In what cases does KDP itself lead to conflict or worsen existing conflict? What external conditions need to be met for KDP to have a positive impact? If KDP does have an impact, which elements of the KDP program appear to be most influential, and what are the specific mechanisms through which it effects change? III. Using Mixed Methods to Answer the Research Questions To answer the range of research questions and investigate the causal chain of events, we designed a mixed method evaluation that incorporated qualitative and quantitative components in a complementary and mutually reinforcing way. Contributions of Qualitative Methods The contributions of qualitative methods are rooted in the subject matter. Qualitative approaches are better able to assess the dynamics and trajectories of local level conflict, as these phenomena are difficult to quantify or reduce to numbers, and are intricately tied to local context--areas quantitative approaches have difficulty adequately accounting for. Also, since the dynamics of local conflict have not been exhaustively researched, qualitative methods helped the research team remain open to unexpected findings. The study was further concerned with issues of process, another area better addressed through qualitative exploration. First, qualitative approaches are needed to identify the key mechanisms that trigger, sustain, or resolve conflict. To this end, the research team conducted seven months of qualitative quasi-anthropological fieldwork to develop case studies of how different actors-- villagers, facilitators, local leaders--together negotiate (or fail to negotiate) different types of conflicts in different settings. Using a modified version of the process tracing method (George and Bennett 2005; Varshney, 2002), researchers investigated `conflict pathways', seeking to understand the discrete stages in the evolution of conflict. By doing this, researchers were better able to identify the factors that transform underlying social tensions into different outcomes (violence or peace). Second, as an evaluation of KDP, the study had to dissect the various components and processes within KDP itself to understand how each one interacts with conflict and its management. The identification of positive and negative areas of KDP influence is information that the project managers can use to improve the program's effectiveness. For both areas, qualitative methods were critical to understand the mechanisms by which particular variables become salient and the importance of local contexts.8 Qualitative Instruments and Types of Data Collected Researchers used three qualitative instruments--in-depth interviews, focus group discussions, and informal interviews/participant observation--to collect two types of data: case-based and general data. The primary research tool for the qualitative research was the in-depth interview, accounting for 8Gibson and Woolcock (forthcoming) provide a detailed qualitative analysis of selected cases from this study as they pertain to questions of `empowerment' (i.e., whether KDP helps otherwise marginalized groups, as it claims to do, enhance their bargaining power in deliberative contests with elites). 5 the majority of researchers' field time and the greater part of the data we will analyze. In all, we conducted almost 800 in-depth interviews. Researchers used an open approach: they were given a range of topics to cover, but were free to choose which to address in a particular interview; and they were given a list of guiding questions, but were free to adapt them in order to get at the information needed. Importantly, all field staff were given extensive training on how to conduct these interviews and write them up afterwards (see Barron, Diprose and Smith 2004). Researchers also used a secondary research tool, the Focus Group Discussion (FGD), for three main purposes. First, FGDs were used to gather information on specific conflict cases from groups that might be hard to access in other settings. These groups included the `marginalized' (e.g., poor women), as well as those that one might have difficulty locating or accessing individually (e.g., victims of domestic abuse). Second, they used FGDs to collect more general background information on the villages being studied, including data on community life, groups and networks that existed in the village, economic conditions, etc. Third, FGDs were used to gather diverse perspectives on conflict and, more broadly, security. Women in particular often had different perspectives than authority figures on what constitutes the major problems and issues in a village. For both in-depth interviews and FGDs, researchers followed specific guidelines for the sampling of respondents. In effect, they covered a cross section of the population, balancing authority and non-authority figures, and men and women. They interviewed members of a variety of institutions, professions and village groups. For the conflict cases, researchers used snowball sampling to identify "experts" on a particular case--those involved and key observers. The third research tool we used really consists of two different methods: the informal interview and participant observation. The study used these `informal' methods for two main purposes: first, to gather key unspoken information from the way people act, their relationships and so on; and, second, to gather spoken information people give in informal environments. These techniques further helped establish relationships, build trust, and paved the way for formal in- depth interviews, all of which takes substantial time. Unlike the other tools, there was no formal sampling of respondents. Rather anyone the researchers met, and everyone they saw, was a potential source of information. However, these techniques were especially useful at getting information from `marginalized' and `silenced' groups. In some cases, women were uncomfortable being formally interviewed, but were happy to talk when the researchers were helping them in the house (the researchers lived with informants in the villages they were studying). Throughout, in addition to following up specific cases, the researchers collected general data on conflict and village and sub-district demographics to establish a broader picture of conflict management capacity. The general data helped illuminate the cases we followed, gave us a broader picture of the `conflict map' of the research sites, and allowed us to test the generality of the hypotheses that emerged from the conflict pathway case studies. Contributions of Quantitative Methods The main contributions of quantitative methods revolve around establishing generality and causality--a task difficult for the qualitative work because of its small samples and non-random 6 selection of research sites and respondents. At the broadest level, the research design used a quantitative sampling frame to capture the major dimensions of heterogeneity within the population and increase the reliability of results. Because we were looking for common patterns of project impact, our findings would be strengthened if they hold up in a variety of different settings. Thus we chose two very different provinces and both high and low capacity districts for fieldwork, and then tracked similar conflict cases in matched KDP and non-KDP sub-districts (see next section). We used qualitative investigation to verify the accuracy of matches identified through quantitative techniques to ensure they reflected realities on the ground. Where the qualitative data was susceptible to the subjectivity of the individual researchers collecting it, we used less contextual methods to elicit more objective numerical data. Surveys, by definition, collect general non-case-specific data. By putting together a large-n dataset, consisting of indicators and variables that capture the wide range of factors we think may affect levels of violent conflict, we can test the validity and generality of a range of hypotheses. With this kind of data, we endeavored to isolate correlations between multiple contributing factors (local factors, civic institutions, KDP), and separate the impacts of KDP from the influence of particular characteristics of the research sites. The larger samples allowed us to test the extent to which our findings held across a broader range of research sites (villages, sub-districts, districts, and provinces). Quantitative Methods and Types of Data Collected Whereas the qualitative instruments collected both case and general data, the quantitative survey instruments collected only the less contextual general data. We used three quantitative instruments: a focused key informant survey, a larger but less focused key informant survey (PODES), and a nationwide household survey (GDS).9 In the third phase of the qualitative research, the research team administered surveys to a range of key informants at the sub-district and village levels, in the research sites where the qualitative research was conducted. The questionnaires focused largely on the role of KDP in the locality, the extent to which it was used to solve (KDP and non-KDP) problems, and the extent to which `spillover' effects could be determined. In each of the four research districts, the surveys were administered in three KDP sub-districts, and two or three villages each (thus giving a sample of twelve sub-districts and twenty-eight villages). The surveys were administered to three informants at the sub-district level, and eight at the village level, thus giving a sample of around 260 respondents. This data helped us control for how well KDP was working in each site and helped us explore KDP's relation to conflict and conflict management. However, generalizations from this data source are necessarily limited to the qualitative research sites. The Government of Indonesia's Central Bureau of Statistics' Village Potential series (or the PODES) is a long-standing survey that collects data at the lowest administrative tier of local government. It collects detailed information on a range of characteristics--from infrastructure to village finance--for Indonesia's sixty-nine thousand villages and neighborhoods. The 2003 9To be clear, we designed and fielded the key informant survey; contributed one module to the GDS survey but did not participate in the fielding; and borrowed data generated by the government GDS survey. 7 PODES was fielded at the end of 2002 as part of the Agricultural Census. For the first time, the 2003 PODES included a section on politics, conflict and crime. The PODES data helped to map out the incidence of conflict and violence across Indonesia. It also allowed researchers to test a range of basic hypotheses about factors correlated with higher levels of conflict (see Barron, Kaiser and Pradhan, 2004). The limitations of this data source are that the only respondents interviewed at the village level were Village Heads, who tend to underreport conflict in their villages and present a picture of peace and harmony in order to access development funding. The Governance and Decentralization Survey (GDS) is part of the Indonesian Decentralization Empirical Analysis (IDEA) project, which was conducted by the World Bank together with a research center at a major Indonesian university (the Centre of Public Policy Studies at Gajah Mada University). The survey aimed to ascertain the effects and impacts of the rapid decentralization that took place in Indonesia in 2001 on a number of factors ranging from the performance of local governments, to service delivery, to the functioning of the justice system. The initial (baseline) enumeration took place in 150 (almost half) of Indonesia's districts in 2001. As part of the follow-up survey implemented in 2006, a module on conflict and problem solving was inserted into the survey, and the sample was extended. The GDS thus provides a rich data-source on levels of conflict (as reported at the household level) and data on a range of other `institutional' and `governance' factors that can be regressed against conflict/violence levels (see McLaughlin and Perdana 2008). The formulation of this module was based on the emerging hypotheses from the qualitative fieldwork, newspaper analysis and the results of the PODES survey. Methodological insights from the latter also affected the survey design. Secondary Data Given the fact that violence is a non-random and relatively scarce event, and given the reality of limited budgets and hence sample sizes, it is difficult to use household survey instruments in order to accurately capture levels of violence in a locale. Key informant surveys overcome some of these problems but can encounter others such as major underreporting of conflict and biases about the types of conflicts reported.10 In response, post-qualitative fieldwork analyses of archives of local newspapers were used to a dataset recording each violent conflict `incident' in the research districts and neighboring areas over a three year period (Barron and Sharpe 2005). This data--with the use of other secondary data sources such as police criminal data and information from health care providers--allowed us to estimate aggregate levels of violence, to map the characteristics of the incidents (conflict type, actors involved, impacts, etc.), and to consider variation of both over time and space. Integration of Methods The research design uses an iterative strategy to integrate qualitative and quantitative methods, establishing an ongoing dialogue between the two approaches. By examining the subject matter from multiple angles, we used triangulation to verify that our research findings reflect the true relationship between KDP and conflict. To start, preliminary data research about the field sites was gathered before the intensive qualitative fieldwork began, and informed the design and implementation strategies for the 10See Barron, Kaiser and Pradhan (2004) and Barron and Sharpe (2005) for more on the shortcomings of survey instruments in measuring local conflict. 8 qualitative and quantitative work that followed. This initial work began the process of mapping conflict (its spatial distribution and its characteristics) to help in the sampling of districts and sub-district sites; provided input into the development of the research hypotheses, questions, and the data collection instruments; and, gathered and preliminarily analyzed background secondary research (statistics, academic writings, newspaper articles, etc.). Once underway, the in-depth qualitative research identified the right kinds of questions (and their wording) for inclusion in the more general quantitative surveys. Importantly, the quantitative instruments were designed and developed while the qualitative work was being conducted; as such, their design reflected the ongoing findings of--and methodological lessons learned from-- the qualitative fieldwork. In return, the quantitative fieldwork tested the generality of the hypotheses emerging from qualitative investigation. Analysis of newspaper archives and other secondary data sources was used to estimate aggregate levels of violent conflict in our research areas. Many of the areas of information on which we wanted to collect specific survey data mirrored the general data categories the qualitative researchers used. This was deliberate: looking at a problem from more than one methodological angle gives us increased confidence that what we are finding is true, and that the inevitable, although variable, bias built into any research method is not unduly distorting findings. In short, if we get the same answers to the same questions twice, while using different methods and taking different approaches, we can be reasonably confident that the answers are as true and objective as possible. In sum, Figure 1 illustrates how the combination of methods and types of data provide a more holistic picture of conflict and KDP impacts than any single source or methodological approach could alone. Each item provided a distinct contribution toward capturing the larger `truths' our research seeks to uncover. Figure 1: Data Sources Used Quantitative (breadth) Qualitative Decentralization survey (GDS) Village Potential Survey (PODES) Newspaper analysis (depth) Key informant survey Case studies 9 IV. Sampling Frame One of the most important aspects of the research design was the use of a quantitative sampling frame to select the sites for qualitative investigation. As indicated above, accounting for heterogeneity within the population gives us greater confidence that our research findings do not turn on the particular (idiosyncratic?) characteristics of our research sites. Provincial Level Variation The objective of the provincial selection for the qualitative work was to pick two very different provinces in which to work: East Java and NTT. As we sought to identify common patterns of project impact, our findings would be strengthened if they held up in different settings. We focused on a range of variables to help determine the nature of a province, including (a) population size and density, (b) ethnic homogeneity, (c) religious homogeneity (and dominant religious group), and (d) overall level of provincial development including provision of and access to public services and infrastructure. We excluded provinces with the highest and lowest levels of conflict. Since KDP operates at the sub-district level and below, any positive externalities it may produce are likely to be directed at managing conflicts that exist at those levels. Given the nature of conflict in many high-conflict provinces, where cleavages exist at the provincial or at least district level, if we had selected such provinces we would have biased our research against observing any project impact. Moreover, in areas of high-conflict--where levels of violence are significantly affected by external actors and exogenous factors (such as military action)--it would be much harder to separate out the potential impact of a local level project from all the other causal variables in the research site. District Level Variation In each province, we selected two districts--one with a `high capacity' to manage conflict, one with a `low capacity' to manage conflict. We selected these after extensive consultation at the provincial level with government, international and local NGOs, regional development experts, research institutes and religious institutes, universities, and KDP staff. Picking both `high' and `low' capacity districts for each province allowed us to defend our claims regarding the nature and extent of KDP's impact on local conflict resolution by showing that they take place irrespective of whether the broader environment is conducive to conflict resolution or not.11 Sub-district Matching For the first two phases of qualitative fieldwork, we chose two sub-districts within each of the research districts--a KDP site and a non-KDP site--that were as similar as possible. The former had already had KDP1 for three years (our `treatment' sites); the latter had not yet had KDP1 (our `control' sites). Selecting similar matches controls as much as possible for non-program effects that may stem from socio-economic, institutional, or other differences. We used mixed methods to identify matches. First, we used propensity score matching12 to select similar KDP and non-KDP sites on the basis of pre-interventions characteristics within the PODES and 11Ideally, we would have used statistical data on the incidence of conflict to help make these selections, however such evidence did not exist in Indonesia at the time. 12A statistical technique used to identify otherwise comparable `treatment' and `non-treatment' groups on the basis of the probability of being selected. See Rosenbaum and Rubin (1985) or Baker (2000) for a general introduction. 10 SUSENAS13 datasets. Then we used qualitative methods--interviews at the district level with government officials and other experts--to choose amongst the identified pairs and incorporate other sources of difference or similarity.14 This generated the overall sample frame in Figure 2. Figure 2: Sampling Frame East Java NTT High Capacity Low Capacity High Capacity Low Capacity Ponorogo Pamekasan Sikka Manggarai KDP Non-KDP KDP Non-KDP KDP Non-KDP KDP Non-KDP Badegan Sampung Proppo Palengaan Nita Maumere Lamba Kota Leda Kumba The third phase of qualitative research extended to an additional two sub-districts in each of our research districts. We did this to verify that KDP performance in research sites was representative of how it worked in other sub-districts within the same district, and to include new `treatment' sites to replace areas where KDP was not working as intended.15 We selected additional sub-districts that had had KDP for at least three years, had passed a minimum threshold of KDP performance acceptability relating to transparency and accountability, and were different from the other KDP sites we were studying in terms of their cultural, geographical and demographic characteristics. Thus in East Java we added two KDP sub-districts (Slahung and Jenangan) in Ponorogo, and two (Pasaen and Pademawu) in Pamekasan; and for NTT we added two (Talibura and Paga) in Sikka, and two (Cibal and Ruteng) in Manggarai. Case and Village Selection We determined specific qualitative research locations by conflict case, rather than by village. Our initial maps of major social tensions (constructed in Phase 1 of the research) helped us select the following primary cases in each sub-district to investigate in non-KDP sites (Phase 2A) and matched KDP sites (Phase 2B): 13SUSENAS is the National Expenditure Survey of Households, administered by the Government of Indonesia's Bureau of Statistics. 14We learned through this selection process that ensuring the accuracy of propensity score matches involves a substantial amount of field investigation. In particular, cultural differences influence conflict management and are difficult to measure and match statistically. 15Where the program was not working properly, KDP impacts were clearly difficult to find. To compensate, we purposively extended the sample into sub--districts where KDP was functioning properly. In effect, this means that our research investigated KDP impacts where it is functioning relatively well, and not KDP program impacts more generally. 11 Cases One and Two: Similar cases, different outcomes (within the same sub-district) We selected two similar cases of conflict within each sub-district, but with different outcomes: one violent, one peaceful. We evaluated similarity by type and scale of conflict, underlying tensions, and types of actors. By examining similar cases we gain a better idea of what factors are present, or missing, that result in different outcomes (violence or peace). Case Three: Similar cases in two sub-districts (KDP and non-KDP) We chose this case to match a conflict case in the non-KDP sub-district with a similar case in the KDP sub-district. We used the same similarity criteria above, but compared cases in the two different sub-districts (KDP and non-KDP) within the same district. Not only were the conflicts similar in type and scale, but the general characteristics of the villages were similar as well. Cases Four and Five: Peaceful resolution in violent area; violent case in peaceful area We used a qualitative version of a "difference in difference" strategy, in which we identified instances of peace in otherwise relatively high-conflict villages, and instances of high conflict in otherwise relatively peaceful villages.16 This yielded insights on local-level mechanisms for peace and conflict that were as independent as possible of the broader institutional environment (and thereby replicated at the local level the broader selection strategy adopted at the district level). As such, we compared similar cases in matched (by pre-intervention characteristics) KDP and non-KDP locations to evaluate any possible program impacts. Moreover, we selected similar villages and cases with similar conflict dynamics. Case Six: KDP-related case For the research in the treatment sites (Phase 2B), we also chose KDP-related conflict cases. This case may have been an example where KDP had--directly or indirectly--caused conflict, or where we had preliminary evidence that a KDP forum, actor, or other mechanism had been used to help resolve conflict. V. Concluding Thoughts Having outlined the main components of our research design, we close with a few comments about implementation realities and further thoughts on how this research strategy might be adapted to the assessment of other projects and contexts. Diversions from the Grand Plan Because implementation invariably diverges from original theory, it is important to be responsive and flexible on the ground. The study faced a few dramatic surprises--e.g., the eruption of a volcano in one research site and flash floods in another--that required us to adapt our original choices and quickly select new research locations. Election cycles and religious holiday periods also had to be negotiated. We added an unanticipated third phase of qualitative research when we felt we needed to expand the sample of KDP locations. As other problems and challenges surfaced, we continually and iteratively modified the research tools, data recording 16`Peaceful' resolution of conflict is a relative concept, which includes `less violent' and `nonviolent but not harmonious' resolution. In Indonesia, `peaceful' resolution at times means suppression of a particular incident, with the underlying conflict continuing to fester below the surface. 12 formats, and field guides. Originally, we had planned to field an extensive household survey in addition to the other quantitative instruments discussed above. However, after looking at the data we collected in the first three rounds of research, we decided to drop the quantitative household survey component of the research for two reasons: (a) household surveys are very expensive to administer; and (b) household surveys are not very good at investigating processes--that is, the ways in which particular variables become salient over time--which was of primary interest to us. As a result, we decided to replace the household survey with the scaled down key informant survey; to append a conflict module to an even larger national-level surveys (PODES) being conducted by other colleagues; and to analyze the data from the GDS conflict module. Replication Elsewhere This particular study benefited from a unique alignment of resources, talents, and commitments (both individual and institutional). Given this, how much of the research design, and which particular elements, can be transplanted to other areas? In principle, the framing of the study as an advanced version of "thinking quantitatively, acting qualitatively" (Woolcock 2001) can easily be adapted to other settings. From this starting point, prospective research teams can reinterpret and scale the details of implementation to match available resources. While this study focuses on community development projects and conflict, the design could also be applied to an evaluation of other types of interventions and substantive issues. Thus we believe the conceptual underpinning and basic evaluation principles can serve as a foundation for even the smallest, most resource-constrained study. Other aspects of the research, however, will require more effort to replicate. Extended, advanced fieldwork was crucial for generating specific hypotheses to test and designing research instruments that were relevant to the Indonesian context. The quality of the data collected, and hence of all subsequent analysis, depended on the extensive training, mentoring and buy-in of junior researchers. Commitment to the study and to ensuring its quality involved group learning for all levels of the research team. In any research environment, the adaptive decision-making process--including making time and space for ideas to gestate--is a critical prerequisite to crafting a strategy that more comprehensively uncovers the `truths' regarding the key questions under investigation. 13 References Baker, Judy (2000). Evaluating the Impact of Development Projects on Poverty. Washington D.C.: World Bank. Barron, Patrick, Rachael Diprose, and Claire Q. Smith (2004). "Training Manuals for the KDP and Community Conflict Negotiation Study." Unpublished Mimeo. Available online at www.conflictanddevelopment.org Barron, Patrick, Rachael Diprose, and Michael Woolcock (2006). "Local Conflict and Community Development in Indonesia: Assessing the Impact of the Kecamatan Development Program." Indonesian Social Development Paper No. 10. Jakarta: World Bank. Barron, Patrick, Rachael Diprose, David Madden, Claire Q. Smith, and Michael Woolcock (2004). "Do Participatory Development Projects Help Villagers Manage Local Conflicts? A Mixed Methods Approach to Assessing the Kecamatan Development Project, Indonesia." Conflict Prevention and Reconstruction Paper No. 9 (revised version). Washington D.C.: World Bank. Barron, Patrick, Kai Kaiser, and Menno Pradhan (2004). "Local Conflict in Indonesia: Measuring Incidence and Identifying Patterns." Policy Research Working Paper No. 3384. Washington, D.C.: World Bank. Barron, Patrick, and Joanne Sharpe (2005). "Counting Conflicts: Using Newspaper Reports to Understand Violence in Indonesia." Conflict Prevention and Reconstruction Paper No. 25. Washington, D.C.: World Bank. Friend, Theodore (2003). Indonesian Destinies. Cambridge, MA: Belknap Press. George, Alexander and Andrew Bennett (2005). Case Studies and Theory Development in the Social Sciences. Cambridge, MA: MIT Press. Gibson, Christopher and Michael Woolcock (forthcoming). "Empowerment, Deliberative Development and Local Level Politics in Indonesia: Participatory Projects as a Source of Countervailing Power" Studies in Comparative International Development Guggenheim, Scott E. (2006). "The Kecamatan Development Program, Indonesia", in Anthony Bebbington, Michael Woolcock, Scott E. Guggenheim, and Elisabeth Olsen (eds.) The Search for Empowerment: Social Capital as Idea and Practice at the World Bank. Bloomfield, CT: Kumarian Press, pp. 111-144. Mills, C. Wright (1959). The Sociological Imagination. New York: Oxford University Press. McLaughlin, Karrie and Ari Perdana (2008). "Conflict and Dispute Resolution in Indonesia: New Information from the Governance and Decentralization Survey." Indonesian Social Development Paper No. 12. Jakarta: World Bank. 14 Rao, Vijayendra and Michael Woolcock (2003). "Integrating Qualitative and Quantitative Approaches in Program Evaluation", in Francois J. Bourguignon and Luiz Pereira da Silva (eds.) Evaluating the Poverty and Distributional Impact of Economic Policies. New York: Oxford University Press, pp. 165-90. Ravallion, Martin (2005). "Randomized Trials of Development Policies and Projects: Some Comments" In George K. Pitman, Osvaldo N. Feinstein, Gregory K. Ingram (eds.) Evaluating Development Effectiveness New Brunswick, NJ: Transaction Publishers, pp. 240-247. Rosenbaum, Paul R. and Donald B. Rubin (1985). "Constructing a Control Group Using Multivariate Matched Sampling Methods that Incorporate the Propensity Score." The American Statistician 39: 33-38. Varshney, Ashutosh (2002). Ethnic Conflict and Civic Life: Hindus and Muslims in India. New Haven: Yale University Press. Wetterberg, Anna, and Scott E. Guggenheim (2003). "Capitalizing on Local Capacity: Institutional Change in the Kecamatan Development Program, Indonesia" World Bank, East Asia Department, Jakarta, Indonesia. Mimeo. Woolcock, Michael (2001). "Social Assessments and Program Evaluation with Limited Formal Data: Thinking Quantitatively, Acting Qualitatively." Social Development Briefing Note No. 68, Washington, D.C.: World Bank. 15