WPS5176 Policy Research Working Paper 5176 At the Frontier of Practical Political Economy Operationalizing an Agent-Based Stakeholder Model in the World Bank's East Asia and Pacific Region Barbara Nunberg Naazneen Barma Mark Abdollahian Amanda Green Deborah Perlman The World Bank East Asia and Pacific Region Poverty Reduction and Economic Management Sector Unit January 2010 Policy Research Working Paper 5176 Abstract Reform programs sometimes falter because they are approach entails a quantitative simulation of the politically infeasible. Policy change inevitably creates complex bargaining dynamics surrounding reform. The winners and losers, so those with vested interests strike model anticipates stakeholder coalition formation and bargains to determine how far and how quickly reform gauges the political feasibility of alternative proposed should advance. Understanding these micro political interventions. This paper provides a review of the Agent- dynamics of reform can mean the difference between a Based Stakeholder Model pilot experience, exploring successful intervention that gains political traction and what sets this model apart from more traditional a well-intentioned gambit that falls short of achieving approaches, how it works, and how it fits into the Bank's its developmental objectives. Donors like the World operational cycle at various stages. An overview of the Bank have been searching for ways to take these political Mongolia, Philippines, and Timor-Leste country cases factors more fully into account as they design programs is followed by an examination of policy-related insights to support country reforms. and lessons learned. Finally, the paper builds on this East This initiative sought to introduce a rigorous and Asian pilot experience, offering ideas on a potential way operationally usable political analysis tool that could forward for organizations like the World Bank to deepen be systematically integrated into the World Bank's and extend their political analysis capabilities. The paper country programming cycle. The East Asia and Pacific argues that the Agent-Based Stakeholder Model, utilized region carried out a multi-country pilot of the Agent- thoughtfully, offers a powerful addition to the practical Based Stakeholder Model. This innovative analytical political economy toolkit. This paper--a product of the Poverty Reduction and Economic Management Sector Unit, East Asia and Pacific Region--is part of a larger effort in the department to conduct and disseminate operationally focused political economy research. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The authors may be contacted at bn2144@columbia.edu and nbarma@worldbank.org. The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Produced by the Research Support Team At the Frontier of Practical Political Economy: Operationalizing an Agent-Based Stakeholder Model in the World Bank's East Asia and Pacific Region Barbara Nunberg, Naazneen Barma, Mark Abdollahian, Amanda Green, and Deborah Perlman Final Manuscript World Bank Policy Research Working Paper Series Barbara Nunberg, Naazneen Barma, Amanda Green, and Deborah Perlman are affiliated with the World Bank; Mark Abdollahian is affiliated with Claremont Graduate University. The authors wish to acknowledge the contributions of Marina Rabinowitz and Ha Na Oh as well as funding from the Bank Netherlands Partnership Program. The findings, interpretations, and conclusions in this paper are those of the authors and should not be attributed to the World Bank, its Executive Board of Directors, or any of its member countries. ii Table of Contents I Moving Beyond Traditional Approaches to Political Analysis 1 II How the Model Works 3 III What the Model Can Do: Country Case Studies 7 Upstream Strategy Development: Entry Points for Civil Service Reform in 8 Timor-Leste Informing Analytical Work: Input to the Mongolia Mining Sector CGAC 12 Facilitating Project Implementation: Guidance and Risk Mitigation for the 15 Philippines NPSTAR IV Key Insights from the ABSM Pilot 19 V Conclusion 27 iii At the Frontier of Practical Political Economy: Operationalizing an Agent-Based Stakeholder Model in the World Bank's East Asia and Pacific Region Reform programs sometimes falter because they are politically infeasible. Policy change inevitably creates winners and losers, so those with vested interests strike bargains to determine how far and how quickly reform should advance. These are the micro political dynamics of reform; understanding them can mean the difference between a successful intervention that gains political traction and a well-intentioned gambit that falls short of achieving its developmental objectives. Donors like the World Bank have been searching for ways to take these political factors more fully into account as they design programs to support country reforms.1 Recent reviews of the Bank's development effectiveness have all concluded that more attention to the political economy of reform is needed.2 Where political analysis has been undertaken, it has only rarely fed directly into the development of country assistance strategies or permeated the details of program design and implementation. Some political economy approaches were analytically sound but lacked practical applications. Others offered practical solutions that were not grounded in analytical rigor. This initiative therefore sought to introduce a rigorous and operationally usable political analysis tool that could be systematically integrated into the World Bank's country programming cycle. The East Asia and Pacific region (EAP) carried out a multi-country pilot of the Agent-Based Stakeholder Model (ABSM). This innovative analytical approach entails a quantitative simulation of the complex bargaining dynamics surrounding reform. The model anticipates stakeholder coalition formation and gauges the political feasibility of alternative proposed interventions. This paper provides a review of the ABSM pilot experience, exploring what sets this model apart from more traditional approaches, how it works, and how it fits into the Bank's operational cycle at various stages. An overview of the Mongolia, Philippines, and Timor-Leste country cases is followed by an examination of policy- related insights and lessons learned.3 Finally, the paper builds on this East Asian pilot experience, offering ideas on a potential way forward for organizations like the World Bank to deepen and extend their political analysis capabilities. I--Moving Beyond Traditional Approaches to Political Analysis Until recently, the World Bank's political analysis toolbox has relied primarily on relatively blunt- edged instruments pitched at understanding the macro political landscape in client countries. Such analytical instruments--which have included, over time, Institutional and Governance Reviews 1 See, inter alia, Keefer, Philip (2006) "Can political economy analysis make a difference for development policy?" Unpublished draft; Fritz, Verena, Kai Kaiser, and Brian Levy (2009) Problem-Driven Governance and Political Economy Analysis: A Good Practice Framework. Washington DC: World Bank. 2 See, for example, World Bank (2005) Improving the World Bank's Development Effectiveness: What Does Evaluation Show? Operations Evaluation Department (now the Independent Evaluation Group). Washington DC: World Bank; World Bank (2006) Engaging in Fragile States: An IEG Review of World Bank Support to Low-Income Countries Under Stress. Independent Evaluation Group. Washington DC: World Bank; World Bank (2008) Public Sector Reform: What Works and Why? An IEG Evaluation of World Bank Support. Independent Evaluation Group. Washington DC: World Bank. 3Extensive analytical work applying the ABSM was conducted in a fourth country. For reasons of political sensitivity in a volatile environment, a decision was taken not to discuss the case in this paper. Observations emerging from the case on the general utility of the ABSM approach are, however, included implicitly in the discussion. (IGRs), Poverty and Social Impact Analyses (PSIAs), and Country Governance and Anticorruption (CGAC) Assessments4--can provide informative context on the policy landscape and a broad understanding of the political drivers of key policy reforms. The forms of qualitative stakeholder analysis that frequently underpin these approaches have mapped the behavior of interested actors at a single point in time in response to reform initiatives. But these qualitative approaches tend to be subjective and static, and they are inevitably overwhelmed by highly complex reform environments with large numbers of relevant stakeholders. These conventional instruments are easy to use but they do not exploit the rigor of cutting-edge political science methods now available to analyze how coalition dynamics affect the political prospects of specific interventions. The Agent-Based Stakeholder Model (ABSM) moves beyond these traditional approaches to apply advanced political science techniques to stakeholder analysis. It offers a robust way to inject just-in- time realism into the design of Bank operations, helping guide the reform support strategy toward achievable goals or, where appropriate, signaling when to jettison programs that are going nowhere politically. The ABSM was introduced in EAP in an experiment to elevate the sophistication, accuracy, and operational relevance of political analysis at the Bank. The model draws on leading- edge work in microeconomics and rational choice theory to analyze the preferences and behaviors of relevant decision makers on politically driven issues. The ABSM algorithm simulates the complex round-by-round bargaining dynamics among a large number of stakeholders in a given reform process. The model anticipates how these dynamics play out over time and provides an empirical assessment of the likely extent of reform and degree of stakeholder support for this outcome. While not a substitute for comprehensive understanding of a country's political economy, the ABSM can add considerable value to other types of analysis (Table 1). Table 1: ABSM Adds Value to Other Common Political Analysis Tools Political economy Qualitative Informal political Agent-Based Stakeholder macro-analysis stakeholder intelligence Model (e.g., CGAC, PSIA) analysis What is it? Talking informally Understanding broad Static stakeholder Quantitative simulation of with clients and political drivers of key mapping based on complex political bargaining local observers policy reforms informed opinion dynamics that predicts about political stakeholder behavior in real trends and events time Why do Offers a quick take Contextualizes policy Provides a broad Provides a rigorous we do it? on the political issues in political snapshot of where understanding of complex landscape to help reality; sheds light on stakeholders stand stakeholder coalitions and a Bank country teams multiple factors behind on a key policy gauge of the political feasibility in daily client complex problems issue of proposed reform interactions interventions When To garner initial To inform country and To gain an initial To identify fruitful entry points should we views on political sector strategy sense of coalitions for reform, build consensus, use it? events and identify upstream and reassess supporting and and manage risks through potential reform program opposing reform explicit testing of alternate allies implementation in light and the degree of strategies of political change consensus 4 Levy, Brian, and Nick Manning (2002) Institutional and Governance Reviews ­ a new type of economic and sector work. PREM Note No. 75. Washington DC: World Bank; World Bank (2007) Tools for Institutional, Political, and Social Analysis of Policy Reform: A Sourcebook for Development Practitioners. Washington DC: World Bank. 2 II--How the Model Works At its heart, the ABSM simulates the bargaining dynamics among individual stakeholders with regard to a specific policy reform. It builds on three theories in microeconomics and rational choice.5 The median voter theorem identifies the political center of gravity at the median position on a reform spectrum; hence the winning reform coalition is at the position of the median stakeholder. According to risk theory, stakeholders who are close to this median outcome are assumed to be risk averse, unwilling to risk political capital by insisting on their original position at the possible expense of not reaching a deal; on the other hand, stakeholders who are far away from the median outcome are more likely to risk holding up consensus in hopes of achieving what they want (Figure 1). Using game theory, the ABSM simulates how, subject to these risk tradeoffs, each stakeholder attempts to achieve his preferred reform outcome. In sum, the algorithm models the interactions between each pair of stakeholders over a number of dynamic rounds and predicts the likely outcome of these iterated interactions. Figure 1: Stakeholder Risk Propensity Relative to the Median Position The ABSM approach uses a thorough data collection and recording process to translate the qualitative political economy knowledge of country and technical experts into a single dataset, which feeds into a dynamic quantitative analysis of stakeholder bargaining on a particular reform issue. The model predicts the specific outcome of a reform process based on stakeholder preferences. Through real-time scenario analysis, the analyst can then test the potential impact of alternative interventions. This process can inform the detailed strategic decisions needed to design and implement reforms. 5 On the median voter theorem, see Black, Duncan (1948) "On the Rationale of Group Decision-making," Journal of Political Economy Vol. 56: 23-24; and Black, Duncan (1958) The Theory of Committees and Elections. New York: Cambridge University Press. On risk theory, see Pratt, John W. (1964) "Risk Aversion in the Small and the Large," Econometrica Vol. 32: 122-36; and Arrow, Kenneth J. (1971) Essays in the Theory of Risk-Bearing. Chicago: Markham. On game theory, see Bueno de Mesquita, Bruce (1980) "An Expected Utility Theory of International Conflict," American Political Science Review Vol. 74(4): 917-31; and Lalman, David (1988) "Conflict Resolution and Peace," American Journal of Political Science Vol. 32(3): 590-615. For a more detailed explication of the theoretical underpinnings of the ABSM and how these various theories are merged in its underlying algorithm, see Abdollahian, Mark, and Jacek Kugler. "An Agent Based Stakeholder Model of Politics." Unpublished technical paper; and Abdollahian, Mark, Michael Baranick, Brian Efird, and Jacek Kugler (2006) "Senturion: A Predictive Political Simulation Model," Defense & Technology No. 32. Washington DC: National Defense University. 3 As it has been operationalized during the course of the EAP pilot, agent-based stakeholder analysis is carried out in five steps:6 1. Framing the question: The framing step involves articulating the overall objective of the political problem and breaking it down into its component parts for detailed analysis. The organizational logic of the ABSM provides a structured format to focus general discussions of strategy and policy into concrete issues for analysis. As such, the framing step can help discipline conceptualization of the policy puzzle within the broader political economy landscape. 2. Specifying the issue: An "issue," for the purposes of the ABSM, is defined as the continuum of options that stakeholders may support on a specific policy matter. An issue is specified in a linear manner from 0 to 100, either from the least politically difficult reform being considered to the most politically difficult (Figure 2), or in terms of two alternative poles of reform. In either case, intermediate reform steps are identified along the spectrum. Figure 2: Issue Continuum ­ The Range of Negotiated Policy Outcomes 0 = Minimum Advocated Position Maximum Advoca ted Position = 100 Issue specification is probably the most challenging aspect of applying the ABSM to the Bank's policy dialogue and operational programs; this step is where substantive expertise and the "art" of analytical creativity are most crucial. Taking time to specify the issue properly is also one of the model's benefits, as it helps the Bank achieve clarity on the reform issue, desired outcomes, and key stakeholders. 3. Collecting data: Modeling the policy dynamics behind a specific issue proceeds with a rigorous data collection process. Structured interviews are conducted with individuals who possess a rich understanding of the country context, the stakeholder landscape, and/or the policy issue being analyzed. The information gathered during these interviews is coded numerically for entry into the ABSM software. The aim is to obtain the following data for each specified issue: Stakeholders: A list of all individuals or groups who could potentially influence the outcome of reform. This list comprises government counterparts, other country officials and political parties, World Bank departments, the donor community, and private sector and civil society groups (including academics and the media as well as international and local non-governmental organizations) with an interest in the reform outcome. Positions: The extent of reform supported by each stakeholder, according to the issue spectrum defined. Each position is assigned a numerical value on the issue continuum (from 0 to 100) and can be pinpointed in relation to all other stakeholders' positions. Position data elicit not the outcome the expert thinks each stakeholder would eventually accept (reservation price), nor the stakeholder's true beliefs (revealed preference), but the policy position each stakeholder is currently advocating given the political environment. Influence: The potential influence, power, or resources that each stakeholder could credibly bring to bear on the issue. Each influence score is defined as a numerical value (from 0 to 100) relative to all other stakeholders' influence scores. 6 Below, each of these steps is revisited in terms of the analytical value of the ABSM process itself. 4 Salience: The importance of the issue to the stakeholder. Each salience score is defined as a numerical value (from 0 to 100) that captures how much time and energy a stakeholder is willing to devote to the issue, given all the other issues the stakeholder is concerned with. Collecting data to apply the ABSM to Bank policy questions is an iterative process. For each issue, initial data are collected from key members of the Bank's country team and technical experts. Interviews are then conducted with external experts who have been identified by Bank staff, and the issue specification and data are amended accordingly.7 Iteration allows data to be cross-checked to ensure that the model is run on the basis of accurate information. 4. Analyzing the basecase: The data described above are entered into the ABSM and its bargaining algorithm generates a picture of expected stakeholder interactions over time given current stakeholder positions and perceptions. This is the model's "basecase" outcome (Figure 3). Figure 3: ABSM Basecase Analysis8 Basecase round 1: Initial positions and effective power of each stakeholder (individual or group) Round by round simulation: Dynamic stakeholder interactions evolving over time as bargaining process plays out Basecase final outcome: Expected outcome of the reform process, demonstrating degree (or lack) of convergence of political support 7 In countries where confidentiality is critical, data collection can be restricted to a select set of trusted external experts, counterparts, and donors upon whose discretion the Bank can rely. This issue is discussed further below. 8 The ABSM picture here and those below are generated using the Senturion software, licensed by Sentia Group, Inc. 5 Analysis begins with a careful and detailed examination of the graphical display of basecase results to determine whether stakeholders are expected to converge in support of a particular reform position, to what degree, and how quickly; and, therefore, what level of reform could be expected under the current dialogue and with what level of support. The ABSM is then used to drill down into the details of the underlying bargaining dynamics to see how the basecase outcome evolves. This allows analysts to identify which stakeholders are expected to change position and in response to whom (Figure 4), opportunities the Bank may have to work with key stakeholders to build consensus and advance reform, and potential obstacles the Bank faces in achieving its desired level of reform. Figure 4: Pairwise Stakeholder Interaction Results in Four Possible Outcomes 5. Defining and interpreting alternative scenarios: Based on the analysis of the basecase expected outcome, the ABSM is used to run simulations of strategies the Bank (or other actors) might be able to use to improve reform outcomes.9 This exercise explicitly tests whether taking a different policy position could, for example, help the Bank achieve greater consensus in support of reform, or if the Bank could realistically exert leverage on a particular stakeholder to garner support for more extensive policy change. The model can thus provide a detailed illustration of the micro- politics surrounding the implementation of reforms, and the costs and rewards of alternative Bank strategies. The model also offers sensitivity analysis to determine whether changes in position, salience, or influence scores for certain stakeholders could have a significant impact on the outcome. Sophisticated statistical randomization techniques (such as Monte Carlo simulations) are also available to help minimize bias from inaccurate data.10 The ABSM is captured in a user-friendly software application that can easily be loaded onto a laptop, offering intuitive visual snapshots of data and analysis and the flexibility to refine issue specification and data input on the go. Staff with a background in political science or institutional economics can be trained on the model's methodology, analytical techniques, and practical application during a 9 "Improvement" is a relative term depending on how each stakeholder views the prospect of reform. Here, we refer to policy improvements from the point of view of the Bank team conducting the stakeholder analysis. 10 See Abdollahian et al and Abdollahian and Kugler, op. cit., for more details on these techniques. 6 three-day hands-on course. Members of country or task teams can thus use this portable tool themselves, on demand, helping to embed political analysis in operational decision making so as to have impact on real-time policy choices. III--What the Model Can Do: Country Case Studies The East Asia and Pacific region piloted the ABSM approach in four countries, attempting to integrate the model into the analytical and project cycle and thereby generate lessons on the model's applicability to Bank work.11 The overall aim of this exercise was to improve the quality of political analysis informing strategic programming choices and to enable the Bank to work with country counterparts to build consensus for important governance reforms. The model was applied, on demand, at various stages of the operational cycle (Table 2). Table 2: Five Pilots at Four Stages of the Bank's Operational Cycle Stage Potential uses of ABSM Pilot approaches Upstream Targeted insights on the scope and contours of political will to Timor-Leste: Entry brainstorming assess momentum behind a multi-faceted reform program points for civil service "Blue sky" exploration of potential new areas of Bank reform Country engagement Mongolia: Refining Assistance Strategy Helps identify key supporters and opponents, along with Bank transparency and (CAS) strategies to actively build political commitment by bolstering accountability development, supportive coalitions and defusing potential opposition approaches program design Sensitivity testing can be used to examine which strategies are robust across different possible states of political affairs, helping anticipate "political landmines" Informing Assessing nature of political commitment to core reform goals Mongolia: Options in analytical work analyzed in a public expenditure review or other core diagnostic natural resource Helps identify key supporters and opponents of reform management value chain Country Insights into the links between micro reform issues and macro- Governance and political context Anticorruption (CGAC) Assessments Calibrating Defining politically feasible prior actions for policy loans Calibrating conditionality Allocating resources efficiently to most promising reform conditionality or prior prospects actions for a broad- Development Helps identify specific opportunities for reform and develop risk based policy operation policy operation mitigation strategies to avoid potential derailment of an operation Helps build consensus for reform Smart design and Identifying opportunities for and threats to achieving project Philippines: Risk supervision objectives mitigation for NPSTAR Developing risk mitigation strategies project Projects Building consensus for reform and bolstering engagement 11 This work was supported by a Bank Netherlands Partnership Program (BNPP) grant on "Building Capacity for Political Stakeholder Analysis." Earlier work using a beta version of the model paved the way for this more extensive pilot. 7 In Timor-Leste, the ABSM explored the Bank's potential role in supporting new momentum behind a civil service reform program ahead of the next Country Assistance Strategy (CAS). The Mongolia country team applied the model twice, first to evaluate options for strengthening transparency and accountability in the public sector and then, in the context of the Country Governance and Anticorruption (CGAC) Assessment, to inform analytical work to support the transparent and effective use of mining revenues. In the Philippines, the Bank employed the ABSM to guide implementation of a stalled tax administration reform project, seeking to avoid the pitfalls encountered in previous attempts to improve the tax system. Extensive analytical work applying the ABSM was conducted in a fourth country to help assess political commitment to a broad-based development policy operation and to help define realistic reform targets in a difficult environment. The ABSM pilot experience is reviewed briefly below, highlighting one of the modeled issues in three countries to explore why the context was ripe for systematic political analysis, what we learned, how we changed our approach, and--where sufficient time has passed--what has happened since.12 Upstream Strategy Development: Entry Points for Civil Service Reform in Timor-Leste Administrative and civil service reform recently resurfaced as a pivotal issue in Timor-Leste. The fragile coalition government that came to power after the 2006 political crisis declared 2008 to be the "Year of Administrative Reform," pledging to restore public confidence in government by strengthening the institutional framework for accountability and service delivery. The Prime Minister personally took the lead in driving several key initiatives, chief among them the proposed establishment of a Civil Service Commission with wide-ranging powers. To date, the Bank has not been a lead player on administrative and civil service reform in Timor- Leste but it has been at the forefront of other public sector reform initiatives. In 2006, in concert with other development partners, the Bank launched the Planning and Financial Management Capacity Building Program (PFMCBP) to help tackle serious deficiencies in public spending and revenue management. After a slow start, the program was reshaped in 2008 to help the new Ministry of Finance manage crucial public financial management priorities while planning for longer-term professional development needs. The Ministry wanted to reorganize public financial management processes and develop the supporting system of incentives, rewards, and sanctions to build a stronger public service ethos. The Bank agreed to support the Ministry's objectives and, in addition, made a commitment during the 2008 Timor-Leste and Development Partners Meeting to support the country's administrative reform process more broadly. The ABSM was enlisted to help better understand the political dynamics operating beneath the surface of this complex agenda and to identify the most promising entry points for potential intervention. The analytical exercise focused on three dimensions of civil service reform: (1) the potential creation of a Civil Service Commission and its appropriate scope of powers; (2) the demand for a revised pay and grading system and the underlying debate between job- and 12 Exhaustive country case studies are available upon request. These provide extensive detail on each of the three to five issues modeled in each country, including: (1) a full dataset listing all stakeholders included in the analysis along with their position, influence, and salience data; (2) the ABSM modeling outcomes; and (3) a list of all experts interviewed during the course of the analysis. Each country pilot was also evaluated systematically through the application of a standardized assessment framework comparing ex ante expectations with the ABSM's impact over time; a synopsis of this evaluation is provided below. 8 qualification-based systems; and (3) the more general discussion about what would constitute a robust program of civil service reform and the level of political appetite for its implementation. Following is a brief review of the ABSM analysis on the third issue, the broader civil service reform agenda. Framing the question. Although the Government of Timor-Leste had articulated its commitment to improving public administration and service delivery, comprehensive civil service reform can be difficult to achieve anywhere. Well-intentioned reforms can easily be derailed by weak capacity, overlapping and ill-defined policies, feeble coordination across ministries and agencies, a fragile political coalition, and stakeholders' worries about what might happen to their own pay levels, positions, and recruitment and promotion opportunities. The ABSM analysis focused on the potential scope of support for wide-ranging reform and attempted to identify the major coalitions that might form on this issue. In Timor-Leste's social and political order--characterized by political schisms and corresponding patronage networks borne of a violent struggle for independence-- merit-oriented civil service reforms were highly contentious. Given the country's need to elevate government performance to achieve tangible results on service delivery as a means to political stability, the stakes were particularly high. Specifying the issue. The process of arraying the varied elements of a comprehensive civil service reform package along a linear continuum proved complicated. The team divided the overall agenda into three levels of cumulative reform, building from 0 to 100 (Figure 5). Figure 5: Policy Steps in Timor-Leste's Civil Service Reform Agenda 100 ­ Independent oversight of merit-based civil service regime 95 ­ Merit-based civil service regime implemented 80 ­ Robust establishment management system implemented & resourced 75 ­ Govt-wide pay & grading system implemented & adequately resourced 55 ­ Legislative basis for civil service institutional responsibilities 45 ­ Senior executive cadre defined, recruited, & adequately resourced 40 ­ Civil service management framework approved 30 ­ Technical analysis, including job evaluation & comparative pay 10 ­ Pay & grading pilots initiated 5 ­ Provisional pay & grading package approved in decree law 0 ­ Status quo The first level focused on the low-hanging fruit: temporary pay relief, pay and grading pilots, and technical analysis. The second level covered the design of a new system: approving a new framework for civil service management, creating a senior executive cadre, and defining the legal basis for institutional mandates regarding the civil service. The last--and most politically difficult--level 9 focused on the actual implementation of reforms: a new government-wide pay and grading system, establishment control, merit-based practices, and independent oversight.13 Collecting data. The team met with a total of thirty experts in Timor-Leste, gathering information through broad, contextual interviews as well as more structured data interviews. The data showed that key executive stakeholders supported basic reforms up to and including the definition of a legal basis for institutional oversight of the civil service. A few interested groups within the executive advocated a higher degree of reform, including changes to the pay and grading system, while most civil servant groups favored more moderate steps. Lower-level civil servants, temporary workers, and political opposition groups were resistant to anything but basic technical analysis. Within the Bank, the Poverty Reduction and Economic Management (PREM) department supported the most comprehensive level of reform, while the country team called for more gradual progress toward the definition of a senior executive cadre. The initial landscape of stakeholder positions is laid out in Figure 6. Figure 6: Initial Stakeholder Positions on Timor-Leste's Civil Service Reform Agenda INFLUENCE (POWER * SALIENCE) Executive Civil Service Parliament Private Sector Church NGOs Academia Donors Foreign Media Others Analyzing the basecase. Based on initial stakeholder positions and perceptions, the model anticipated some level of consensus on setting up the formal arrangements for civil service management, including outlining the framework for civil service management and creating a senior executive cadre (Figure 7). Key groups--including civil servants, the executive branch, and political parties--would for the most part be included in this consensus. A sizeable block of stakeholders would support greater reform, advocating legislation on institutional responsibilities and moving toward implementation of reforms. But the model did not foresee much political support for implementation of more comprehensive reforms during the timeframe covered by the analysis (generally about two years); only a handful of executive stakeholders, donors, and governance- focused NGOs would be likely to push for full implementation. 13 As with all issues specified for ABSM, support for the highest level of reform would include support for the intermediate steps along the way. 10 Figure 7: Anticipated Basecase Outcome of Timor-Leste's Civil Service Reform Agenda INFLUENCE (POWER * SALIENCE) Executive Civil Service Parliament Private Sector Church NGOs Academia Donors Foreign Media Others Defining and interpreting alternative scenarios. Scenario analysis suggested that the Bank could help tighten consensus around a somewhat more comprehensive and strategically articulated level of reform--including a legal basis for civil service management, moving toward reform implementation--if it were to devote more attention and resources to this issue. Conversely, if the Bank were to pull back from this issue, the model suggested that consensus would also tighten, but at a lower and more piecemeal level of reform--outlining only the framework for civil service management. Scenario analysis indicated that governance-focused NGOs could credibly pressure key civil service groups--including teachers, health workers, police, and district staff--to press for full reform; however, the ABSM also suggested that this could fracture consensus and possibly ignite political conflict if the urgency of reform demands were high. The team tested several other strategies, but none was sufficient to drive reforms beyond the formalistic steps foreseen in the basecase. What did we learn? The analytical findings advanced PREM and country management discussions about the possibility of marshaling the Bank's resources in support of lower but more feasible levels of reform or concentrating on more specific elements of the administrative reform agenda. ABSM framing and analysis sharpened the country team's view that governance reforms often take shape as a three-stage process in Timor-Leste. Moving from the existing situation--characterized by multiple, ad hoc, and often disconnected civil service systems--to creating formal institutional arrangements would not be that difficult. But progress would likely be stymied in the leap between laying out those arrangements and their actual implementation. In addition, the stakeholder analysis challenged the presumption, common among the Bank and other donors, that government policy is indeed amenable to successful intervention. The exercise emphasized, on the contrary, that development partners may have relatively little influence on the current civil service reform agenda, and that positions we might take could have unintended effects like fracturing coalitions and even triggering policy retrogression. Nevertheless, in a country where the reform debate is still evolving, the process of specifying the reform issue and conducting interviews with key players can actually shape the nature and content of that debate. 11 Informing Analytical Work: Input to the Mongolia Mining Sector CGAC As Mongolia's new democracy and market economy mature, Mongolian society is increasingly concerned about corruption and weak governance. As a significant source of revenue, the mining sector is particularly susceptible to rent seeking and corruption; in a recent survey, Mongolian citizens ranked mining as the second-most corrupt sector, after land utilization.14 The government has made important strides toward improving transparency and accountability in the sector, including its ongoing effort to implement the Extractive Industries Transparency Initiative (EITI). Still, problems persist in ensuring transparency and efficiency in the production and management of mineral resources. In particular, the current system for granting mining licenses is ad hoc, discretionary, and under-regulated, and manual processing opens the door to manipulation and corruption in the issuance of mining permits. Such concerns were part of the focus of a Country Governance and Anticorruption (CGAC) Assessment conducted in 2007 as part of a Bank-wide initiative for which Mongolia was a pilot country. The Mongolia CGAC undertook broad governance analysis to inform the Country Partnership Strategy. A problem-driven political economy framework was used to identify a series of potential reform steps across various elements of the natural resource value chain, including mining licensing, public financial management, public investment planning and procurement, and social transfer policies. These findings provided a springboard to help mainstream the Bank-wide governance and anti-corruption (GAC) agenda and methodology in the sectors, and the CGAC team worked closely with country management, technical and sector colleagues, and Mongolian stakeholders to identify the key measures needed to achieve "good enough" governance in the mining sector. As part of this initiative, the country team employed the ABSM to help assess the prospects for improving sector governance to make effective use of growing mining revenues. With national parliamentary elections slated for the following month and mining sector issues at the top of the political agenda, this pilot of the ABSM was conducted at a pivotal time. Among other issues related to transparency and accountability, the ABSM analysis focused on reform of the mining licensing and cadastre systems. Following is a brief review of the analysis on this issue. Framing the question. The issue of mining transparency in Mongolia encompassed procedures for awarding mining exploration licenses, including the degree to which these procedures are transparent, as well as the maintenance and public availability of cadastre records. License application and allocation procedures are highly complex and opaque, making it difficult for Mongolian citizens to learn about, evaluate, and challenge ongoing and proposed mining activities. There are some 29 steps in the application process and the cadastre system is not computerized. The resulting delays have tarnished the reputation of the cadastre office, as has the perceived role of patronage and favoritism in determining license awards. In response, the cadastre office has begun developing procedures and manuals to reduce the informality and discretion of license granting procedures and to streamline the number of necessary steps. The parliament holds authority over broader mining policy decisions, raising questions as to whether politicians and government officials will initiate changes that limit their power in the sector. The ABSM was applied to help gauge the level of political commitment for reform in this area, including improved regulation, information dissemination, and computerization of cadastre records. 14Asia Foundation and United States Agency for International Development (2008) Mongolia Corruption Benchmarking Survey 2006-2008. 12 Specifying the issue. The policy continuum for the mining transparency issue included a series of cumulative steps (Figure 8), building from the current ad hoc, discretionary, under-regulated, and politicized process for granting mining licenses (0) to the implementation of a clear legal framework and automated cadastre system, complete with internet-based query capabilities (100). In between, preliminary measures included decreasing the number of steps in the application process, supporting a cadastre website, and clarifying the status of land. The next stage of reform involved enacting increasingly stringent exploration requirements for license holders, rolling out the automated cadastre system, and holding public consultations on mining costs and benefits. The final stage of reform included establishing an interagency commission to oversee and ratify cadastre recommendations, fully deploying the automated cadastre system, and applying a clear regulatory framework for granting and renewing exploration licenses. Figure 8: Policy Steps to Improve Mining Transparency in Mongolia 100 ­ Regulatory framework applied in granting & renewing licenses; automated mining cadastre system fully developed, with web query enabled 87 ­ Clear regulatory framework applied, with publicly disclosed regulations above the ministerial level, preferably through cabinet or presidential decree 70 ­ Interagency commission established to oversee & ratify cadastre recommendations; automated system deployed; quarterly reports published 65 ­ Public consultation supported on mining costs and benefits as part of license granting process 50 ­ Mining leases granted only when economic feasibility is established; automated cadastre system deployed but not fully operational 40 ­ Provisions in place for exploration continuation until economic viability is established 30 ­ Minimum expenditure required on exploration licenses to minimize speculation; automated system partially deployed, with only geodetic data 20 ­ Status of land clarified via Ministry of Nature & Environment for reserves, protected areas, and special use lands 15 ­ Computerized cadastre and license records website supported 10 ­ Streamlining begun by decreasing the number of process steps through detailed operational procedures 0 ­ Ad hoc, discretionary, under-regulated, and politicized process for granting mining licenses Collecting data. The team discussed mining issues with about a dozen experts in government, parliament, civil society, mining companies, and donor organizations, including within the Bank. Although these experts were broadly willing to talk openly about mining issues, many of them had only superficial information on the specifics of mining license and cadastre system reforms; access to those on the inside of the reform debate was constrained due to the pre-election timing. The data showed that, at the time of the pilot exercise, stakeholder positions varied widely across the spectrum of possible reforms (Figure 9). Many executive stakeholders and political parties supported mid-level reforms such as granting mining leases only after economic feasibility is established, deploying an automated cadastre system even if not fully operational, and supporting public 13 consultation on mining costs and benefits. A few members of the executive--including the Prime Minister, Anticorruption Agency, National Audit Office, the head of the Mineral Resources and Petroleum Authority of Mongolia (MRPAM), and the director of the cadastre office--advocated more comprehensive reform, including the establishment of an interagency commission to oversee and ratify cadastre recommendations and the full deployment of an automated cadastre system with quarterly reports published. Cadastre office staff preferred to limit reforms to providing for exploration continuation while economic viability was being established. Mining sector representatives were split between opposing reform and supporting the application of a clear regulatory framework, depending on their position in the current system. Among those supporting only cosmetic initiatives were provincial governors and Erdenet, a state-run mining joint venture with Russia. The Bank's sector experts on mining and public sector reform advocated the application of a clear legal framework with publicly disclosed regulations. The Bank's country management team supported slightly more modest reform, closer to the establishment of an interagency commission and deployment of an automated cadastre system. Figure 9: Initial Stakeholder Positions on Mining Transparency in Mongolia INFLUENCE (POWER * SALIENCE) Executive Parliament Political Civil Society Private Sector SOE SME Donors Foreign Media Academia Analyzing the basecase. Based on initial stakeholder positions and perceptions, the model's results indicated that--despite the push for a better mining licensing regime supported by the director and staff of the cadastre office--no concrete agreement would be reached (Figure 10). Instead, the model anticipated that two countervailing coalitions would form. The pro-reform coalition would come together in support of establishing an interagency commission to oversee and ratify cadastre recommendations, including the Prime Minister and his supporters within the Mongolian People's Revolutionary Party (MPRP), the Democrats' Polar Star faction, the Ministry of Finance, the head of MRPAM, the Mongolian National Mining Association, parliamentary committees and some foreign mining companies. The less reformist coalition would only support the steps leading up to, and including, the granting of mining leases only after economic feasibility is established. Led by the President, this block would garner strong support from the President's supporters within MPRP, MRPAM staff, and an influential Mongolian energy sector consulting company. A few groups-- including governors in resource-rich provinces, the regional treasury, the Ministry of Energy and Fuels, the Republican party, and some private companies--were shown to have significant leverage to unravel any reform bargain on mining licensing. 14 INFLUENCE (POWER * SALIENCE) Figure 10: Anticipated Basecase Outcome of Mongolia's Mining Transparency Reform Executive Parliament Political Civil Society Private Sector SOE SME Donors Foreign Media Academia Defining and interpreting alternative scenarios. ABSM scenario analysis was used to explore several options to build consensus on mining transparency reforms. If the Bank were to make this issue a top priority, the model did not foresee that there would be any improvement in consensus; in fact, this move would be expected to cement support for only mid-level reforms. By making this issue a top priority and moderating its position to support the granting of mining licenses only when economic feasibility could be established, the Bank could help foster some consensus for slightly higher-level reforms--including public consultation on mining costs and benefits and the establishment of an interagency commission. On balance, though, this strategy was expected to have little beneficial impact, as the countervailing coalitions would become even more polarized. Scenario analysis also explored the possibility that the director of the cadastre office could sway his staff, provincial governors, and the regional treasury to back the creation of an interagency commission. While the model suggested that this tactic would galvanize some agreement on this step, it would not be expected to overcome the opposition of domestic political parties. What did we learn? This pilot of the ABSM was informed by several background notes on key reform issues, produced in the context of the Mongolia CGAC. This analytical preparation was of considerable use in providing the contextual and informational basis for the stakeholder analysis--it laid out broad hypotheses about the potential political space for reform, identified key stakeholders, and suggested potential entry points for the Bank's engagement in the policy process. The ABSM was used, in turn, to assess the degree of political commitment around those potential interventions in an operationally timely window as parties crystallized their platforms. Reflecting the highly politicized nature of the mining transparency issue, particularly given political polarization in the month before national elections, the analysis cautioned that there was little the Bank could do to effect sustained reform in this area at the time. This experience suggested that the relevance and utility of the ABSM could be enhanced if modeling capabilities were embedded within the country management unit, allowing for flexible and timely responses to rapidly shifting operational environments and priorities. Facilitating Project Implementation: Guidance and Risk Mitigation for the Philippines NPSTAR Tax administration reform has been a major governance challenge in the Philippines for over two decades. The tax system has been overhauled twice in the past twenty years--first in 1986 and again 15 in 1997. The 1986 reforms aimed to simplify collection systems and succeeded in boosting revenues. The later initiative attempted to bridge the revenue gap expected in the wake of economic liberalization; but it came to fruition during the 1997 Asian financial crisis and weak political commitment led to a failure to produce results. The Bank's own support to the sector has had an equally mixed record. The Tax Computerization Project of the mid-1990s achieved some important technical gains but reform impact foundered in the face of administrative obstacles and political opposition to implementation. The government's more recent, small-scale reform programs have fared better. The passage of a value added tax reform law in 2005 sparked a 20% increase in tax collections. Other wins have been achieved through compliance verification drives, third-party information gathering, expanded tax audits, and lifestyle checks on public employees. Large gains in tax collection have remained elusive, however. The ratio of tax revenue to GDP has begun to rise, but remains low relative to other East Asian countries. Out of a workforce of about 33 million, only 6 million taxpayers are registered. An aging information system has created a logjam, leaving tens of thousands of tax returns unprocessed. This backlog, together with extensive tax exemptions for well-connected individuals, does little to encourage workers to register. Despite the government's formal commitment to address these problems, systematic reforms that would make a real and sustainable impact on tax revenues consistently lack traction. This lack of movement on reforms points to deeper political obstacles. Building on continuing dialogue between the Bank, the International Monetary Fund, and the government, the Bank introduced a new tax administration reform operation in mid-2007. The multi-donor National Program Support for Tax Administration Reform (NPSTAR) seeks to achieve quick wins in its early phases, creating demonstration effects to build momentum for broader reforms down the line. The program's four components include tax compliance, enforcement and control, human resource development and management, and Bureau of Internal Revenue (BIR) governance and change management. In an effort to avoid the pitfalls of earlier reform efforts, the Bank applied ABSM to help understand the potential political obstacles to successful implementation of the NPSTAR and to formulate a risk mitigation strategy. The Philippines pilot analyzed four separate issues on three broad fronts: (1) the macro-political landscape for general tax reform; (2) the micro-issues surrounding BIR internal organization, including full enforcement of the Lateral Attrition Law--which was designed to provide incentives for BIR and customs staff to exceed revenue targets--and implementation of performance management systems; and (3) the micro-issue of increasing BIR tax collection capacity through taxpayer registration. Following is a brief review of the model's analysis of the taxpayer registration issue. Framing the question. The absence of accountable, fair, and clean taxpayer registration processes reinforces the population's negative attitudes toward the tax system, as does the backlog of returns. Expanding taxpayer registration appeared to be the most politically feasible tax reform issue on the table in the Philippines, but two key areas presented problems: computerization and access to the tax information base. At the time of the analysis, only 48 of the 112 district offices were computerized, making it difficult to keep track of taxpayers and handle delinquencies.15 The option of outsourcing registration in the short term had been raised, but there were concerns regarding confidentiality of taxpayer records. Registration improvements could also be complicated by 15Rodolfo Vicerra, Director General, Congressional Planning and Budget Department, Philippines House of Representatives (2005) "The Looming Fiscal Crisis Shouldn't Be that Taxing." Manila, Philippines. 16 opposition from corrupt officials who were unwilling to support reforms that might diminish their rent-seeking opportunities as well as by key stakeholders who had not prioritized the issue. Specifying the issue. The country team identified three main stages of taxpayer registration reform (Figure 11), building cumulatively from keeping the current system (0) to full taxpayer registration (80) to contracting out the registration process (100). Intermediate policy steps focused on expanding registration objectives, arranged in order of increasing political difficulty: reviewing content requirements and registration procedures; prioritizing classification categories; agreeing on procedures to clean the database by removing stop filers; agreeing on tax mapping objectives, approach, and methodology; and moving from internally-staffed full registration implemented at a gradual pace to full registration through external contracting, done quickly. Figure 11: Policy Steps to Improve Taxpayer Registration in the Philippines 100 ­ Full registration contracted out quickly 90 ­ BIR registration contracted out, with Commissioner's support 80 ­ Full registration rolled out slowly, without contracting out 70 ­ Objectives, approach, and methodology for tax mapping agreed 60 ­ Procedures agreed to clean & validate database & remove stop filers 45 ­ Priorities set for registration by classification (e.g. large taxpayers first) 25 ­ Content requirements & registration procedures reviewed 0 ­ Do nothing Collecting data. Interviews were conducted with over thirty experts on tax reform in the Philippines, including representatives from the government, civil society, the private sector, academia, and donor organizations. The data showed that most stakeholders supported considerable movement toward taxpayer registration reform (Figure 12). Figure 12: Initial Stakeholder Positions on Taxpayer Registration in the Philippines INFLUENCE (POWER * SALIENCE) BIR Executive Civil Society Congress Private Sector Church Donors Financial 17 Executive stakeholders, including the President and Department of Finance (DOF), favored full registration as long as it was not contracted out. The BIR Commissioner advocated the definition of objectives, approach, and methodology for tax mapping, while BIR staff and Revenue District Offices (RDOs) were only open to reviewing content requirements and registration procedures. Anti-reform camps in the House and Senate were willing to support prioritization of classification categories. The Bank advocated contracting out BIR registration, which was a controversial position given the requirements for confidentiality in handling taxpayer records. Analyzing the basecase. Based on these initial stakeholder positions and perceptions, ABSM analysis suggested that there would be tentative agreement on defining the objectives, approach, and methodology for tax mapping, but actual implementation was expected to encounter serious difficulties (Figure 13). Specifically, BIR examiners, RDO clerks, and other reform opponents would stand firm in advocating only the review of content requirements and registration procedures. These groups could unravel any reforms mandated by BIR management by swaying Congressional reformers, donors, and the private sector to take less forceful reformist positions. Figure 13: Anticipated Basecase Outcome of Taxpayer Registration in the Philippines INFLUENCE (POWER * SALIENCE) BIR Executive Civil Society Congress Private Sector Church Donors Financial Defining and interpreting alternative scenarios. ABSM scenario analysis explored several possible strategies for mitigating the program implementation risks posed by the strong political opposition predicted by the model. If the Bank were to take a softer stance--advocating full registration but not pushing for it to be contracted out--stakeholders would still be expected to come together only on defining the objectives, approach, and methodology for tax mapping but the implementation problems foreseen in the basecase would likely ease. A more promising approach would be for the Bank to express its dual positions, with country management pressing for full registration--contracted out and carried out quickly--and PREM settling for agreement on the parameters for tax mapping (Figure 14). If PREM were then to approach the BIR Management Committee to support full taxpayer registration conducted internally, this would be expected to prompt strong agreement on establishing at least the parameters for tax mapping with mid- to long- term political support for full registration by government. Opposition from BIR examiners, RDOs, local governments, and anti-reform elements in Congress would dissipate, although extended discussions would probably be required to settle on the exact level of reform. 18 INFLUENCE (POWER * SALIENCE) Figure 14: Scenario Analysis ­ Dual Bank Positions on Taxpayer Registration in the Philippines BIR Executive Civil Society Congress Private Sector Church Donors Financial What did we learn? The model's results made intuitive sense to members of the NPSTAR project team; they were aware that these reforms could provoke resistance but the analysis helped them think through these issues more systematically. Based on ABSM results, the project team gained a deeper understanding of the precise sources of resistance and ideas for how the project approach could be adjusted accordingly. In particular, the entrenched opposition from BIR staff suggested the need for more attention to change management within the agency and participatory activities to empower lower-level staff. In response, the project devoted a greater proportion of its resources to change management activities, including information dissemination campaigns and workshops to increase understanding of the reform process, and surveys to solicit feedback. Piloting the ABSM helped the country team unpack the sources and strength of political opposition to tax reform in the Philippines. It became clear that most stakeholders wanted reform, but wholesale progress was being thwarted by a smaller group of powerful, interested stakeholders. The analysis of the overall tax reform landscape indicated that revamping the entire system would not be feasible at this time. But the model also identified opportunities to achieve tangible progress through more targeted attention on key micro-level issues, several of which corresponded with components of the NPSTAR. These findings validated the program's approach of starting small to build momentum for future reforms. Indeed, this strategy was borne out by the results of the taxpayer registration analysis, where building consensus on lower-level reforms was expected to lay the foundation for more meaningful change later on. Even small changes in the Bank's position could help cement this needed consensus. IV--Key Insights from the ABSM Pilot The EAP multi-country pilot of agent-based stakeholder modeling has demonstrated the potential utility of mainstreaming this type of political analysis throughout the Bank's work, at various stages of the operational cycle depending on country team demand and the country environment. 19 The Evaluation Framework As part of the pilot exercise, a standardized assessment tool was developed to evaluate the process of applying and adapting the ABSM to country governance reforms as well as the model's impact on country team decision-making. In focusing on the process, the evaluation framework aimed to get a sense of how the ABSM affected the way the country team approached a particular governance problem--what their expectations were at the outset, how the experience of systematic data collection and analysis shaped their views, and what effect, if any, the results had on their approach to the issue on the ground. The evaluation tool also focused on results, seeking to determine whether the ABSM improved the quality of political analysis underpinning Bank programming decisions. To this end, the framework's questions (excerpted in Table 3) sought to examine whether the model's recommendations were adopted and whether reform outcomes were as expected. Table 3: ABSM Evaluation Framework ABSM analysis and Ex ante expectations Immediate outcomes Impact over time recommendations What is the Bank's desired What new insights and/or Did the Bank redefine its Did the altered strategy level of reform? relationships did ABSM desired level of reform? affect the reform outcome? reveal? Who are the key Did ABSM change your How did ABSM alter Did the altered strategy stakeholders and what are views about the behavior relations with client or achieve its objectives? their relationships? of key stakeholders or the other stakeholders? nature/difficulty of the reform issue? What should the Bank's Did the strategy change? Did you try any strategies What would have strategy be? What are the that emerged from ABSM happened if the Bank had advantages and analysis? What were your not changed its approach? disadvantages of this expectations for this strategy? strategy? How do you intend to use For which issues and If you did not change the What is the marginal ABSM? operational stages is ABSM strategy, why not? benefit of ABSM over most useful? time? The ABSM evaluation was conducted as a set of ex ante and ex post interviews of each analytical team involved in the pilot. These country team evaluations revealed useful insights drawn from each stage of the model's application--from thinking through the key elements of each reform issue and the relationships among stakeholders to testing some of the model's more promising strategic recommendations. Table 4 provides a brief summary of key evaluation insights on four of the five applications of ABSM in the East Asia and Pacific Region. In many of the multiple issues examined in each case, high initial expectations of reform progress were shown to be unrealistic in the face of strong opposition, often from expected quarters. In this respect, neither a broad civil service reform agenda in Timor-Leste nor the more targeted sequence of mining license reforms proposed in Mongolia appeared to have the necessary political traction on the ground for implementation. In other cases, the ABSM yielded granular insights on sources of support for reform where the Bank's initial expectations were originally more pessimistic. In the case of taxpayer reforms in the 20 Philippines, for example, the country team's expectations were low at the outset due to the failure of previous reform efforts to gain traction. The Bank had tried to influence major opposition groups, such as large taxpayers and anti-reform elements in Congress, through dialogue, but to no avail. Through ABSM analysis, the team was surprised to find that a small group of previously overlooked stakeholders--the tax examiners working in the Bureau of Internal Revenue--would likely block reforms otherwise supported by the majority of stakeholders. This insight prompted the team to shift tactics, introducing into the NPSTAR program a more aggressive change management strategy aimed at helping tax examiners and other lower-level staff understand the potential benefits to be gained from improved taxpayer registration and broader tax administration reform. Other cases demonstrated that shifts in the Bank's own positioning on reforms might yield better- than-expected results. In two cases, reflecting variance in internal views and expressing diverse Bank positions on reform led to a tightening of stakeholder consensus at a higher level of reform than initially anticipated. In one case, working toward a more moderate goal with greater inclusiveness yielded better-than-expected results, since key stakeholders for whom the reforms had high salience could drive progress. Maintaining a focus on tangible micro-reforms held the key to realizing limited but meaningful objectives in an otherwise stalled macro-reform environment. Table 4: Key Insights from ABSM Evaluations Pilot Ex ante ABSM analysis and Immediate Impact over time application expectations recommendations outcomes Mongolia Pre-election timing Powerful officials Bank made an effort Insights supported Budget considered a window strongly opposed to engage in by CGAC political transparency of opportunity for greater public access meaningful economy analysis transparency reforms to budgets consultations; Achieved reform Opportunities for increased emphasis on level Bank team was Bank to work with demand-side in aiming for: better than civil society to push country program basecase, but short of for reform Bank's stated position (full reform) Mongolia Pre-election timing No concrete "Wait and see" Team may reapply Mining considered a window agreement expected, approach model in post-election transparency of opportunity for despite support of key environment transparency reforms counterparts Philippines Difficult project One overlooked Bank introduced Project still in early Tax implementation due group of stakeholders change management implementation phase administration to disagreement over (tax examiners) could into project design; due to other design and failure of block implementation slightly adjusted unforeseen political past reform efforts Revise project timeline obstacles approach Timor-Leste Window of Some consensus on Little role seen for Civil Service Civil service opportunity on civil setting up formal the Bank at the time Commission created reform service reform seen to architecture of civil Conduct more with expansive scope open; Bank still service management analytical work while Little movement on forming its strategy Civil servants could awaiting Parliamentary broader civil service splinter consensus if action agenda mobilized 21 A more inclusive strategy resulted from applying the ABSM to budget disclosure reforms in Mongolia during the first ABSM pilot in that country. The team's initial approach was to work directly with key government counterparts to encourage greater transparency in the budget process. Reform expectations were high, as the dialogue was taking place during an election year and it was thought that this would encourage key stakeholders to support transparency and participation. In conducting stakeholder interviews, the team was surprised to find that stakeholder opinions varied more widely than initially thought. Moreover, ABSM analysis indicated that the Bank had relatively little influence over the Ministry of Finance on this issue. Instead, non-governmental organizations like the Open Society Institute were shown to have a great deal more leverage over government counterparts than the team had previously considered. This insight led the Bank team to take on a consensus-building role by bringing key stakeholders together around the negotiating table rather than working through direct dialogue with traditional government counterparts. Within a year or two, the Mongolian government began publishing the executive budget upon submission to the National Assembly. This outcome represented a significant improvement over the basecase result anticipated by the ABSM in the absence of further interventions. In this manner, on some issues, insights gleaned through the process and results of ABSM application led to adjustments in the Bank's strategy. This was not the case for all piloted issues, however. In some cases, country teams adopted a "wait and see" approach, having determined that the political situation was too fluid, the Bank's understanding of the reform issue too limited, or the Bank's overall strategic goals too underdeveloped to proceed on the basis of ABSM results. In the second Mongolia pilot, for example, data collection efforts were constrained by the pre-election timing of the analysis such that the team did not feel it had achieved a sufficient grasp on the stakeholder landscape to move forward on sensitive mining sector reforms. In Timor-Leste, applying the ABSM to macro civil service reform issues proved to be complicated by a limited understanding of the appropriate sequencing of reforms in that context. The Bank elected to wait until the Parliament had taken some action in the direction of civil service reform before making further decisions, and to conduct additional analytical work in the meantime. While this outcome may not have differed from what would have occurred without ABSM analysis, the decision to delay was taken more deliberately following application of the model. Overall, the ABSM evaluations show that country teams found value in the process of applying the model to critical reform issues. Although ABSM results were not considered to reveal definitive answers or to substitute for nuanced judgment, pilot teams generally found the model to be a useful tool in understanding the political economy of reforms and testing new approaches to complex reform issues. The analytical methodology crystallized the trade-offs among various dimensions of reform and the possible benefits to be gained from sacrificing some movement in the degree of reform in favor of reaching greater consensus, or in slowing down the speed of reform in order to achieve better results. Teams that tested ABSM-recommended strategies generally found them to make intuitive sense and yield useful results. The Value of the ABSM Logic In addition, the methodology underlying the application of ABSM and its structured format forced teams to think critically about the nature and sequencing of reform issues and comprehensively about the stakeholder landscape. Hence the process of applying the ABSM can itself be instructive. Framing, issue specification, and scenario analysis are beneficial even just in terms of the conversations they foster among the team as they think through problems and options more 22 systematically. The process of collecting data is also illuminating, as it encourages field staff to engage in structured conversations with their counterparts on specific issues and to rethink their assumptions about how politics affect the Bank's work. Implementing the ABSM gives shape and discipline to political analysis in the Bank, which can be helpful in an environment of heavy workloads and competing priorities. Although political analysis has come to be regarded as an important input to strategy development and program design, the absence of a standardized, mainstream mechanism for applying such analysis in an operationally useful way has blocked its full integration into the decision-making process. The ABSM can help make the application of political analysis more robust and the process of collecting data on stakeholder preferences can help get the ball rolling on better understanding of the political dynamics of reform. Framing the question. The first stage of ABSM analysis can help Bank teams articulate the overall aim of their interventions and how they fit into the broader political context. The organizational logic of the model's approach and its visual representation in the ABSM software can help guide strategy discussions and define the parameters of program design and implementation. Often, the process of framing and specifying potential issues for analysis can bring knowledge gaps to light. Specifying the issue. A number of substantive challenges are inherent in issue specification, often pushing Bank staff to think much more carefully about the steps along the critical path in a given reform program than would normally be required in preparing a support package or even a piece of analytic work. The process of issue specification can thus facilitate a productive discussion about the sequencing and comparative difficulty of a set of reforms on a given issue. For example, for an issue continuum to be linear, all stakeholders must order the various steps along the continuum in the same manner. For issues that are continuous and directional by nature, this is not a problem. It is much more challenging to lay out a linear continuum for issues that are made up of discrete reform "steps"--all stakeholders may not view the various reform steps in the same order of political difficulty or even see them as moving along a single reform track. For complex, multi-pronged reform programs, the analytical contortions required to specify a linear continuum may not be worth the analytical payoff. Decisions about the level of disaggregation in specifying the continuum must be viewed as a balance between modeling what is necessary to understand the reform process for better-informed decision-making versus recreating a high-fidelity model of the actual problem. Collecting data. The quality of the data determines the quality of the analysis: if garbage goes in, garbage comes out. The process of data collection through expert interviews has neither the luxury of statistical safeguards built into large-sample data analysis, nor the exhaustive nature of good qualitative research methods. Analysts using the ABSM must take care to ensure that data collected through expert interviews are accurate and representative of reality. Choices about a data collection strategy will vary across countries depending on the political economy context, and on the specific issues to be analyzed. There are two major dimensions guiding this choice (Figure 15): (1) the difficulty of the reform environment, including the Bank's relationship with the government, overall government effectiveness in achieving reforms, and the level of divisiveness in society as a whole; and (2) access to good and relevant data, which depends on factors such as the degree of openness in society, the Bank's access to non-governmental experts, and the number of experts who understand the country and the specific reforms being analyzed.16 16In low capacity environments, data quality may also depend on the Bank's ability to conduct interviews in a way that encourages local experts to apply their considerable local knowledge to the technical issues being explored. 23 Figure 15: Data Collection Strategies in Various Reform and Data Environments Data quality can be strengthened in various ways. First, an effort should be made to interview as many genuine issue experts as possible. One some issues, such as civil service reform in Timor-Leste or mining transparency in Mongolia, this number may actually be quite small given the breadth and depth of technical and political knowledge required. In other cases, as in macro tax reforms in the Philippines, the team can take advantage of the wide variety of experts with useful knowledge and varying viewpoints on the issue to increase confidence in the data. Second, during data collection, analysts continually perform internal consistency checks on the information provided by experts through careful, pair-wise comparisons of the numerical values assigned to position, influence, and salience for the stakeholders involved. Third, if analysts are uncertain about any particular piece of data, a sensitivity test can be carried out to changes in that piece of data. Finally, with each successive use of the ABSM in a given country, the Bank builds a library of data. Continued use of the ABSM software can compile useful information for tracking patterns over time and sharing knowledge, both within the country team and across the Bank. Analyzing the basecase and testing scenarios. The ABSM offers a wide-ranging menu of analytical options through its software package. Systematic interpretation of results and creative scenario analysis are facilitated by the software's use of a SWOT framework, in which analysts are encouraged to think through and record the strengths, weaknesses, opportunities, and threats associated with the basecase result and each alternative scenario. To help identify strengths and weaknesses, the team begins with a careful and detailed examination of the model's results. Is there 24 convergence on a specific reform position? If so, at what level? Who are the outliers, and are they strong enough to stall reforms? Where are the Bank's allies? Next, the team drills down into the details of the model's bargaining dynamics to better understand potential opportunities and threats. Which stakeholders are moving to what positions, and in response to whom? Can the Bank convince other stakeholders to support more reform? What leverage do other stakeholders have to push for a lower level of reform than the Bank is advocating? The model's scenario analysis capabilities then allow analysts to explore scenarios in which the above opportunities and threats are exercised and thereby anticipate how outcomes might change. The Bank can also test its own role in the reform process, adjusting its position or level of commitment (salience) to determine whether these changes could help nudge the reform agenda forward or bring additional stakeholders on board. In essence, the ABSM can lay out strategic choices in a way that helps country and project teams think through the political opportunities and consequences inherent in their programming decisions. The ABSM's Operational Value for the Bank As part of the pilot exercise, the ABSM has been used to think through various political and reform puzzles facing the Bank and its country clients and to provide real-time analytical inputs to programming decisions, often under intense time pressure. Through this pilot experience, the team has refined the process of applying the model's analytical approach to Bank work and illuminated possible ways to integrate the ABSM into the Bank's regular programming decisions. Here, we highlight the ABSM's major analytical benefits. Modeling a complex, multi-stakeholder reality. The capacity to model complex problems allows the ABSM to capture intricate interactions among a large number of stakeholders. Traditional, qualitative stakeholder tools cannot achieve this, rendering their explanatory power largely intuitive. Assuming good data are fed into the model, the ABSM is likely to provide a more robust and comprehensive picture of the political forces that affect reform. Moreover, the flexibility to change the menu of stakeholders, alter their positions, and test varying scenarios allows the tool to more closely model policy-making conditions in the real world. This dynamic quality, accompanied by accessible and intuitive graphical simulations in the accompanying software, makes the ABSM particularly useful in sifting through intricate political realities to help zero in on more feasible reform strategies. Illuminating macro-micro political relationships. The ABSM is a micro-political tool that captures finely grained details a more macro-political analysis cannot. There are both advantages and drawbacks to this. The ABSM can elucidate the micro-political relationships among stakeholders that can be so central to reform processes and outcomes, although it is less able to embed these bargaining dynamics in the overall macro-political context. Still, where macro-level reforms have stalled, drilling down to examine stakeholder positions and the influence they exert on each other on a specific sub-issue can reveal new insights about the opportunities and threats facing reforms, which are hard to see from a more aerial macro-political perspective. In the Philippines and Timor- Leste, this capability helped the country team clarify the Bank's comparative advantage in supporting micro-reforms within a macro-political agenda that appeared stalled. Over time, collecting data on where stakeholders stand on these micro issues can help track patterns in the broader landscape of political relationships. 25 Testing and challenging conventional wisdom. The ABSM tests prevailing assumptions about a political or reform context. By formalizing the data collection process through interviews with experts, the ABSM can assess the degree of consensus on the generally accepted view of reform. In a political environment where the conventional wisdom has been quite static over time, i.e., where no new political dynamics have been revealed, the model can dig down into individual perceptions of issue-specific reform possibilities to reveal new insights that are not readily apparent through casual--or even close--observation. In gathering information from Bank and donor staff and core government counterparts, the analyst can gauge whether each individual's on-the-ground experiences reinforce or challenge the conventional wisdom about relationships among stakeholders; their relative positions, influence, and salience; and reform processes and outcomes. Without such a check, there is a danger that donors are reinforcing conventional wisdom in a circular process amongst themselves without testing whether it is representative of reality. In the Timor-Leste pilot, for example, the analysis of the proposed Civil Service Commission's mandate pointed out that a few key stakeholders who were originally thought to be relatively powerless instead held quite a bit of potential sway over the outcome of reform. Breaking into the black box of "political will." Another advantage of the ABSM is that it disaggregates down to individual stakeholders the concept of political will and thereby breaks into this ubiquitous "black box" of donor speak. Using the ABSM's data and analytical tools, the Bank can potentially understand much more deeply the relationships among stakeholders that can either fuel or dampen reform progress, thus unveiling the contours and substance of political will. This knowledge can help to identify politically motivated factors that could improve impact or pave the way for greater reform consensus down the road. In the Philippines, after years of stalled initiatives, the NPSTAR team was surprised to learn how much desire there was for tax reform; at the same time, a wide array of pro-reform groups was unable to overcome the entrenched opposition of a few powerful stakeholders. In Mongolia, ABSM interviews and analysis revealed previously unseen splits within the major political parties and unexpected coalitions of stakeholders. Helping to prioritize and sequence reforms. A critical lesson emerging in the Bank's effort to improve the relevance and effectiveness of its development assistance is the need for greater selectivity and flexibility in its programs. The ABSM can help tease out the reform initiatives most deserving of donor attention and resources by illuminating the degree of political support and consensus in a given reform environment. While the Philippines pilot anticipated little tangible progress on proposed macro-level reforms, the analysis helped identify micro-level issues where there was enough traction to move forward in the meantime. The pilot experience also helped draw attention to important "no go" areas, where the Bank might inadvertently squander political capital by pushing for something prematurely. In Mongolia, the Bank learned that caution on mining transparency issues would be wise in the highly politicized pre-election environment. In the Philippines, pressing for implementation of a performance management system was shown to harbor the potential for a hostile response among staff who feared its effects on their jobs and wages. In Timor-Leste, taking a position on the type of civil service regime was considered counterproductive, as the Bank would not be able to help move the debate forward amid entrenched viewpoints on either side of the issue. Analytical Limitations Although the ABSM can provide, for the reasons outlined above, a powerful lens through which to look at reform possibilities in client countries, it is not a silver bullet. As in any other modeling 26 exercise, there is no way to create a perfect model of reality. There are concerns, for example, that the ABSM's reliance on formal policy positions can distort results by neglecting informal factors such as clientelist ties and political tradeoffs across unrelated issues. This potential data bias can be addressed by thinking consciously about informal undercurrents when collecting and cross-checking data and in interpreting the model's results. The software's sensitivity analysis capabilities allow analysis teams to run tests on particularly hazy data. Another worry is that the model's grounding in the theory of the median voter will beget the same "race to the center" witnessed in majority-based political contests, thereby encouraging only partial reforms. Yet the pilot exercise demonstrated that the decision to focus on mid-level reforms to strengthen consensus is a choice that the Bank and its counterparts can make in response to the model's results; it is not by any means a foregone conclusion. Teams using the ABSM should remain mindful that it is not a 1:1 model of the world; instead, the ABSM is a powerful new tool in the Bank's political economy toolbox. V--Conclusion While the need for better political economy analysis to improve on-the-ground results is now roundly acknowledged across the World Bank and other development assistance institutions, adopting robust approaches that are both rigorous and operationally usable poses a substantial challenge. East Asia's experience with the Agent-Based Stakeholder Model has yielded important lessons on the development of "practical political economy" tools. The country cases showed a continuum of ABSM applications at up-, mid-, and down-stream points in the operational cycle, with different policy implications. In Timor-Leste, the analysis was helpful in constructing the country and donor policy debate on civil service reform, though the prospects for short-term agreement were small; in Mongolia, the analysis helped identify key coalition partners and underscored the need for continuing, embedded analysis as electoral and economic conditions evolved; and findings in the Philippines underscored the need to tackle smaller reform modules on which broad consensus could be achieved, even if this meant scaling back on important reform aspirations. In general, the tool was helpful in crafting micro-political policy agreement on specific issues, even when the wider reform environment was inhospitable to broad advances. While the predictive value of the tool was helpful in some instances, it came as something of a surprise that the ABSM's heuristic contribution in helping teams think through the sequence and strategies for reform was particularly well received. This suggests that the ABSM would be well- suited to convening country or sector strategy brainstorming and would be most helpful if carried out on a regular, institutionalized basis. The ABSM instrument was distinguished by the systematic, just-in-time, and dynamic quality of its analytic power, but the tool did not offer a stand-alone, silver-bullet solution and was most effective when complemented by other, broader types of political economy analysis. The relative intensity of the field protocol and the need for up-front investment in training must be noted as well, but the East Asia experience suggests that these could be overcome if the tool were adopted in a high-volume business model. In sum, ABSM offers a powerful addition to the political economy toolkit, if utilized thoughtfully. While East Asia's pilot initiative was confined to a limited number of cases, the experience strongly suggests that mainstreaming ABSM as a regularly applied analytic instrument could significantly improve the operational track record of development institutions such as the World Bank in supporting politically realistic reforms in client countries. 27