47009 www.dsfindonesia.org Making Decentralization Work for Development: Methodology of the Local Government Performance Measurement (LGPM) Framework December 2008 Decentralization Support Facility Indonesia Stock Exchange Building Tower I 17th Suite 1701 Jl. Jend. Sudirman Kav. 52-53 Jakarta 12190 Tel: +62 21 5299-3199 Fax: +62 21 5299-3299 Email: info@dsfindonesia.org Printed in December 2008. The findings, interpretations, and conclusions expressed herein do not necessarily reflect the views of DSF or the governments they represent. DSF does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of DSF concerning the legal status of any territory or the endorsement of acceptance of such boundaries. December 2008 Making Decentralization Work for Development: Methodology of the Local Government Performance Measurement (LGPM) Framework Making Decentralization Work for Development: Methodology of the Local Government Acknowledgements Performance Measurement (LGPM) Framework Acknowledgments This Local Government Performance Measurement (LGPM) methodology paper was prepared by a core team from the World Bank led by Jessica Ludwig-Maaroof, Adrianus Hendrawan and Aurelien Kruse, together with Ahya Ihsan, David Elmaleh, Sukmawah Yuningsih, Edmund Malesky, Günther Schulze, and Peter Rooney. The team also included I Dewa Wisana, Ni Wayan Suriastini, Harryanto and M. Natsir Kadir from Hassanuddin University and Suhanda from Andalas University. This paper is the result of a larger initiative conducted in collaboration between the Government of Indonesia and the World Bank, DSF, USAID-LPSP, GTZ, TAF and KPPOD. The DSF provided financial support. Valuable inputs were obtained from the Government of Indonesia: Prof. Mardiasmo, Heru Subiyantoro and Kadjatmiko from the Ministry of Finance; Max Pohan, Bambang Widiyanto, Himawan Hariyoga, Arifin Rudyanto, and Agus Prabowo from the State Ministry for Development Planning (Bappenas); and Eko Subowo, Saut Situmorang, Ramses Hutagalung, and Bonar Sihite from the Ministry of Home Affairs. The team would also like to thank the walikota (mayors) of Blitar and Tangerang and the bupati (district heads) of Biak-Numfor and Solok for their support during the piloting stage. The team appreciates valuable inputs from a larger group within the World Bank that comprises Blane Lewis, Emmanuel Skoufias, Soekarno Wirokantoro, Bambang Suharnoko, Enrique Blanco Armas, Ahmad Zaki Fahmi, Francisco Javier Arze del Granado, Cut Dian Agustina, Bastian Zaini, Elif Yavuz, Claudia Rokx, Puti Marzuki, Andy Ragatz, Pandu Harimurti, Vicente Pacqueo, Menno Pradhan, and Rajiv Sondhi. Thanks also go to Peter Milne for editing and improving the final version of the text and Arsianti for lay-out, design and production. Wolfgang Fengler provided supervision and oversight of the overall process, together with Susan Wong and Gregorius DV Pattinasarany. ii December 2008 Making Decentralization Work for Development: Foreword Methodology of the Local Government Performance Measurement (LGPM) Framework Foreword The advent of decentralization in Indonesia in 2001 devolved a wide range of authorities and responsibilities to local governments. Taken together, these now account for about 35 percent of total public spending in Indonesia. As a consequence, the role of local governments in delivering services and achieving the country's development goals is now larger than ever. But how do we measure the progress of local governments eight years after decentralization? Without systematic monitoring, evaluation and performance-measurement systems, this crucial question cannot be answered accurately and the broader policy implications of decentralization will remain unclear. If decentralization in Indonesia is to succeed and receive the full backing of the communities that it is intended to benefit, a credible method for evaluating the performance of local governments is crucial. As a result, stakeholders now realize the importance of local government performance measurement (LGPM). Under the umbrella of the multi-donor Decentralization Support Facility (DSF), the World Bank's Public Finance and Regional Development team and the economic program division at The Asia Foundation (TAF) have worked together to develop means for measuring four core dimensions of local government performance. The World Bank focused on the three pillars of public financial management, fiscal performance and service delivery, while The Asia Foundation focused on the investment climate. This report describes the methods developed for measuring these aspects of local government performance and suggests how they might be applied. We hope that the LGPM methods presented in this report will be of use to local governments across Indonesia, possibly in collaboration with service providers and other interested stakeholders, to establish how well provinces, districts and cities are carrying out these aspects of their work. The ultimate goal is to help local governments to improve the quality of services that they deliver to their communities. December 2008 iii Making Decentralization Work for Development: Methodology of the Local Government Glossary of Terms Performance Measurement (LGPM) Framework Glossary of Terms APBD Local Government Budget Bappeda Local Development Planning Agency Bawasda Local Internal Audit Agency BKD Local Civil Service Agency BPKD Local Finance Management Agency CPS Country Partnership Strategy DAK Special Allocation Fund DAU General Allocation Fund Dispenda Local Revenue Office DPRD Local House of Representatives DSF Decentralization Support Facility GoI Government of Indonesia GRDP Gross Regional Domestic Product HDI Human Development Index Kabupaten District Kota City KKD Local Cash Office KPPOD Regional Investment Attractiveness Business Perception Report LG Local Government (refers to both provincial and district/city) LGPM Local Government Performance Measurement MDG Millennium Development Goal MoF Ministry of Finance MoHA Ministry of Home Affairs PAD Own-Source Revenue PEFA Public Expenditure & Financial Accountability Perda Regional Regulation/By-Law. PFM Public Financial Management Puskesmas Community Health Center at Sub-District Level RPJMD Regional Medium-Term Development Plan RSUD Regional Public Hospital Sekda Secretary of a Province/District/City Setda Secretariat of a Province/District/City SIKD Regional Finance Information System SiLPA Budget Surplus SKPD LG Working Unit SP2D Cash Disbursement Order Letter SPM Payment Order TAF The Asia Foundation UPTD Local Technical Implementing Unit iv December 2008 Making Decentralization Work for Development: Table of Contents Methodology of the Local Government Performance Measurement (LGPM) Framework Table of Contents Contents Acknowledgments ii Foreword iii Glossary of Terms iv Table of Contents v Chapter 1 Summary 1 Chapter 2 Why local government performance monitoring is important and who stands to benefit 3 A) Avoiding`decentralization failures'by tackling the information bottleneck 4 B) Users and uses: converging interests from different perspectives 4 Chapter 3 The framework: architecture and scoring 7 A) Four areas of intervention: reflecting LGs'new development responsibilities 8 Public Financial Management 8 Fiscal Performance 9 Service Delivery 9 Investment Climate 9 B) Functional areas within pillars 9 C) Individual Indicators 10 Chapter 4 Methodological choices and challenges 11 A) Initial constraints and choices: Defining and capturing performance 12 B) `Scoring'of indicators 14 C) Aggregating within and across LG performance pillars 15 D) Determining performance groups 16 E) Assessing relevance 16 Chapter 5 Piloting activities 17 Annexes 21 Annex 1: Proposed list of indicators 22 Annex 2: Mapping performance groups: an illustration 31 Annex 3: Hybrid indicators 32 December 2008 v Making Decentralization Work for Development: Methodology of the Local Government Table of Contents Performance Measurement (LGPM) Framework Figures Figure 1. Overall architecture 8 Figure 2. Composition by pillar and strategic area 10 Figure 3. The various options and stages in measuring local government performance 13 Figure 4. Local government performance index 19 Recommendations Recommendation 1: Measuring outcomes or processes 12 Recommendation 2: Scoring of indicators 15 Recommendation 3: The issue of weighting 15 Recommendation 4: The issue of performance groupings 16 Tables Table Annex 2. List of simulated service delivery indicators 31 vi December 2008 Chapter 1 Summary Making Decentralization Work for Development: Chapter 1 Summary Methodology of the Local Government Performance Measurement (LGPM) Framework Previously one of the most centralized countries in the world, over the past eight years Indonesia has undergone a transformation that has made it one of the most decentralized countries. Since 2001, local governments (LGs) have become the main drivers of public service delivery. In order to meet their new responsibilities LGs have had to dramatically expand their capabilities. Since the success of decentralization will ultimately be determined by the performance of LGs, measuring and tracking their performance is critically important. In view of this, the development partner community and various branches of the Government of Indonesia (GoI) have launched a number of monitoring initiatives at the local level.1 For instance, KPPOD andThe Asia Foundation (TAF) have focused on the investment climate in Indonesian districts,2 and the World Bank, together with the Ministry of Home Affairs (MoHA), has developed a tool to rate achievements in the area of public financial management (PFM).3 This local government performance management (LGPM) tool is a continuation of these initiatives, but contains a number of important differences. The goal is to capture the four key dimensions of LG performance, namely: (i) public financial management; (ii) fiscal performance; (iii) service delivery; and (iv) the investment climate.4 The tool includes more than one hundred indicators, providing a snapshot of overall performance giving insight into the specific domains that drive overall performance. It highlights areas requiring further scrutiny, ideally in conjunction with other more targeted survey instruments. The tool builds on a uniquely rich set of district-level budgetary data and survey results that are either already available or collected for the purpose of this exercise. Many methodological choices still need to be made, but the overarching ambition is clear: to provide both central and local policy-makers, development partners, and citizens with a simple and transparent tool for gauging LG performance across districts and within different domains of LG activity, as well as a set of best practices that can be replicated. Furthermore, the idea is to measure the performance of LGs against targets that are known to be achievable within a relatively short timeframe and within the Indonesian context. Decentralization will play a major role in determining development outcomes in Indonesia and LGs are at the core of the decentralization strategy. Measuring their performance and ensuring their success in carrying out their new tasks is not only central to the strategy of the GoI, but also of great significance for the development partners involved, such as the Decentralization Support Facility (DSF) and the World Bank Group through its Country Partnership Strategy (CPS) 2009-12. This paper lays out the background of the LGPM framework and presents its main features. It also explains the methodological challenges and choices that lie ahead in developing a nationwide index of LG performance. Finally, it presents lessons learned from the experience of piloting in selected districts. 1 See WB Strengthening Public Services in Decentralizing Indonesia: Approaches for measuring Performance of Local Governments, (2005) p38 for a comprehensive survey of existing initiatives. These include USAID's LGSP and "Financial Trend & Fiscal Indicators" monitoring projects, the MoHA's"Evaluasi Penyelenggaraan Pemerintah Daerah". 2 KPPOD. Regional Investment Attractiveness, Business Perception Report (yearly). 3 WB Local Government Financial Management ­ A Measurement Framework (2005). The survey has already been rolled out in Gorontalo, Nias and Aceh's 21 district governments. 4 The investment climate pillar was led by TAF/KPPOD. 2 December 2008 Chapter 2 Why local government performance monitoring is important and who stands to benefit Making Decentralization Work for Development: Chapter 2 Why local government performance monitoring is Methodology of the Local Government important and who stands to benefit Performance Measurement (LGPM) Framework The LGPM initiative is deeply rooted in the context of Indonesia's bold transformation towards decentralized government. Since 2001 and the onset of the country's `big bang' decentralization, LGs have seen an extensive broadening of their responsibilities and a correspondingly large increase in their revenues. Districts (kabupaten) and cities (kota) have obtained considerable authority in domains that are key development drivers, such as local health and education services, as well as infrastructure development. LG spending now represents about 35 percent of total public expenditures in Indonesia. Indonesia's move towards decentralization was initially designed to preserve national unity and stability in a period of crisis. However, it has now become a major feature of the national governance architecture. The challenge now is to make sure that decentralization delivers on its development promises. This is a formidable task: considering the speed and the depth of the decentralization process, it is hardly a surprise that LGs have not yet made full and efficient use of their new powers. It is also an urgent task: if decentralization fails to bring about positive development outcomes, there is a risk of disenchantment once the system has outlived its short-term political uses. The purpose of an LGPM tool is precisely to determine how effective LGs are in carrying out their tasks and identifying those areas in which they are falling behind. The following section briefly reviews the importance of accurate performance information in ensuring that the benefits of decentralization can be realized fully and identifies the groups and institutions that the proposed LGPM tool seeks to benefit. A) Avoiding `decentralization failures' by tackling the information bottleneck Decentralization theory was originally formulated in developed countries, where the expected benefits depended on a set of assumptions, including the availability of full information. If these assumptions fail to hold in reality, and in the particular context of developing countries, the process can be undermined and lead to failure. Under decentralization, efficient outcomes are expected to be driven by inter-district competition, which (i) acts"as a check on political power, much as market competition acts as a check on corporate power"(Breton, forthcoming), and (ii) gives citizens a clearer idea of how effective their LGs in terms of delivering public goods and services, and collecting associated taxes. But in the absence of reliable and transparent information competition breaks down. LGs will strive to achieve optimal outcomes only if they fear that citizens will hold them accountable through competitive elections or that individuals and firms might opt to re-locate to better-managed districts. However, this pressure to achieve optimal outcomes disappears if there are no comparative reference points. Producing and sharing performance benchmarks are therefore of crucial importance. B) Users and uses: converging interests from different perspectives A comprehensive and reliable system of LG performance tracking will first and foremost benefit Indonesia's citizens as a whole. Individual voters and advocacy groups can use this information to hold their LGs accountable -- not against unrealistic best practices derived from international experience but against what is feasible in the short term in the context of Indonesia. Inaddition,manyLGsareunawareofbestpracticesineachpolicyareaand/oroffeasiblestepstowardsimplementing them. This LGPM tool would be one step towards enabling LGs to compare themselves with their peers, in particular those districts possessing similar geographic or socio-economic characteristics. A well-designed tool could also provide explicit guidance and suggest feasible ways in which LGs can improve their performance in the short run. Otherpotentialusersincludelocalandinternationalinvestors,anddevelopmentagenciesanddevelopmentpartners, as well as the central government. Large firms would be able to look closely at districts'governance characteristics before making new investments or expanding existing businesses, and local businessmen could use the results to lobby their local authorities. Development partners would be able to use the results for geographic and sectoral targeting of their activities. In the longer term, if a comprehensive index could be established and performance 4 December 2008 Chapter 2 Why local government performance monitoring is Making Decentralization Work for Development: important and who stands to benefit Methodology of the Local Government Performance Measurement (LGPM) Framework monitoring repeated at regular intervals, analysts would be able to link development outcomes to specific changes in government performance, allowing development partner agencies to focus their actions on the most efficient interventions. There are no perfect solutions and the goal is to make sure that the choices on methodology are made to best serve the LGPM tool's stated objectives and desired outcomes. The following section outlines the broad features of the tool and puts remaining design choices into perspective. December 2008 5 Making Decentralization Work for Development: Chapter 2 Why local government performance monitoring is Methodology of the Local Government important and who stands to benefit Performance Measurement (LGPM) Framework 6 December 2008 Chapter 3 The framework: architecture and scoring Making Decentralization Work for Development: Chapter 3 The framework: architecture and scoring Methodology of the Local Government Performance Measurement (LGPM) Framework The LGPM tool is a simple tree, in which individual performance indicators can be grouped into functional areas within four broad thematic pillars. In turn, the overall performance indicator is an aggregation of the four pillars. The following section presents each level successively: the four pillars, the functional areas within these pillars, and the individual indicators. Methodological options and challenges are presented in Section IV. A) Four areas of intervention: reflecting LGs' new development responsibilities Decentralization has affected LG responsibilities most significantly in four ways: (i) transfers from the center dramatically increased the financial resources under their responsibility; (ii) their fiscal powers were broadened significantly; (iii) they now have responsibilities that include social services delivery and local infrastructure; and (iv) they also have the authority to shape local economic policy. In order to capture overall `performance', the LGPM tool seeks to measure LG performance in each of these four areas, not only because they correspond to LGs' new responsibilities but also because they each represent a fundamental building block of what is typically understood by good governance. As a result, the tool aggregates four thematic `pillars' covering LG performance in (i) public financial management, (ii) fiscal performance, (iii) service delivery, and (iv) the investment climate, into one simple indicator of overall performance. Figure 1. Overall architecture Pillar 1: Pillar 2: Pillar 3: Pillar 4: Public Financial Fiscal Service Investment Management Performance Delivery Climate Overall Performance Indicator Public Financial Management Decentralization gave LGs sudden control over very substantial financial resources. However, the capacity to manage these resources transparently and efficiently is lagging in most districts. This makes Public Financial Management (PFM) a crucial area for improvement in the coming years. The World Bank has already developed a comprehensive PFM survey instrument, which has been implemented by 60 local and provincial governments. However, because of the sophistication of the PFM survey instrument and the resources required to roll it out on a large scale, the PFM pillar of this LGPM tool has been simplified by selecting a subset of indicators from the existing World Bank survey. Preliminary tests indicate that the results generated by this simplified subset are closely correlated to the results of the full survey, suggesting that they constitute a good proxy for the full exercise.5 5 The rank correlation coefficient between the full survey results and the results of the subset of indicators was 0.9. 8 December 2008 Chapter 3 The framework: architecture and scoring Making Decentralization Work for Development: Methodology of the Local Government Performance Measurement (LGPM) Framework Fiscal Performance At around 80 percent, transfers from the center make up the bulk of LG revenues and expenditures.6 However, the importance of own-source revenues (PAD) for current and especially future local policy should not be under- estimated. LGs have not paid sufficient attention to strengthening their capacity to raise their own revenue efficiently. For instance, LGs still collect a number of taxes with very low yields and fail to fully exploit more efficient tax instruments that are now at their disposal. On the expenditure side, the recent World Bank's 2007 PER7 has revealed an important missing link in the revenue-expenditure-outcome chain: while LG expenditures have risen, allocations and the effectiveness with which the funds have been used are often been sub-optimal. This fiscal performance pillar seeks to capture how well LGs are doing in fiscal administration, in raising revenues, and in spending these resources. Service Delivery LGs now carry almost sole responsibility in three key service delivery areas: education, health and local infrastructure. In the service delivery pillar, the LGPM tool aims to track outcomes in these three sectors, holding LGs accountable for meeting a number of performance benchmarks. Investment Climate Many LGs underestimate the contribution they can make towards improving the business climate in their regions. This is particularly true for homegrown businesses seeking to scale up their operations. Because investment drives local economic growth and revenue-raising potential, this is a fundamental dimension of LG intervention.8 B) Functional areas within pillars Each pillar is subdivided into a number of functional areas that correspond to the different dimensions of LGs' ability to affect outcomes (Figure 2). In the service delivery pillar, each of the sub-areas (education, health, infrastructure, and cross-sectoral) is further divided into three functional dimensions, namely (i) planning and monitoring, (ii) implementation, and (iii) pro- poor programs. 6 McCulloch, Neil and Bambang Suharnoko Sjahrir. 2008. "Endowments, Location, or Luck: Evaluating the Determinants of Subnational Growth in Decentralized Indonesia." World Bank DSFG Country Study Working Paper. Forthcoming. 7 Spending for development: Making the Most of Indonesia's New Opportunities, Indonesia Public Expenditure Review 2007, World Bank, 2007. 8 The investment climate component of the LGPM tool derives from the work of TAF/KPPOD, which uses the term"Economic Governance Index". KPPOD and The Asia Foundation. 2008."Local Economic Governance in Indonesia: A Survey of Businesses in 243 Regencies/ Cities in Indonesia, 2007"http://www.kppod.org/ (accessed on August 26, 2008). December 2008 9 Making Decentralization Work for Development: Chapter 3 The framework: architecture and scoring Methodology of the Local Government Performance Measurement (LGPM) Framework Figure 2. Composition by pillar and strategic area Local Government Performance Measurement Indicator Coverage Public Financial Fiscal Performance Service Delivery Investment Climate Management Fiscal Land Access & Regulatory Management in Education Security of Framework the LG-level Tenure Planning and Tax & Levy Business Health budgeting Administration Licensing Cash LG & Business Infrastructure Management Interaction Procurement Sectoral Fiscal Business Dev. and asset Cross-sectoral Management Programs management Reporting and Capacity/Integrity accounting of District Head Local Taxes, Audit Charges & Transaction Costs Local Infrastructure Local Regulations C) Individual Indicators Individual indicators are the primary units in developing a performance measuring tool and their definitions, and scoring methods, together with the aggregation formula, will ultimately determine the meaning, shape and range of an overall index. The simplest scenario would consist of adopting a binary scorecard approach in constructing and scoring each indicator, giving them equal weight. Under this approach each question contains a benchmark performance that is either met or not. A satisfactory performance results in a score of 1, while failure results in a score of 0. As a result, the pillar score is simply equal to the number of benchmarks met within each pillar and the overall performance measurement indicator score is the sum of these pillar scores. However, a pure scorecard approach fails to capture incremental improvements.Therefore, a number of more sensitive methodological choices are examined in the following section. 10 December 2008 Chapter 4 Methodological choices and challenges Making Decentralization Work for Development: Chapter 4 Methodological choices and challenges Methodology of the Local Government Performance Measurement (LGPM) Framework The construction and main features of an overall performance measurement index are determined by the need to: (i)`capture'the elusive concept of performance; (ii) find an objective and workable method to score each individual indicator; and (iii) aggregate scores within and across pillars. A) Initial constraints and choices: defining and capturing performance Performancecanbeunderstoodasameasureofoutcomesand/orasanassessmentofprocesses.AnLG'sperformance is reflected both in the quality of the processes that it establishes in order to achieve its development goals and in the success of these processes as they translate into outcomes.The philosophy of the LGPM framework requires that both dimensions be reflected: while citizens probably care most about results, fiscal rewards and incentives should also go to those whose LGs are doing their utmost to achieve outcomes with the resources available to them, even if their efforts have yet to be reflected in outcomes. a. Outcomes: Mapping outcome levels (such as school enrollment rates, average waiting time in clinics) is useful in targeting interventions geographically and observing the evolution of outcomes over time under decentralization. However, this approach does not recognize the different contexts in which LGs operate, provides limited understanding of how these outcomes came about and offers no policy guidance to LGs. Because outcomes typically improve only gradually, focusing exclusively on outcomes penalizes the poorer districts irrespective of the quality of their LGs. It also risks producing a static picture year after year, discouraging effort and providing few incentives to LGs. Likewise, progress variables (understood as improvements in outcome levels, such as `How much have vaccination rates improved in the past two years?'), penalize districts that are already excelling, as statistically significant improvements are easier to accomplish from a low starting point. b. Processes: Measuring performance by relying on process variables (`Have laws X and Z been passed?', `Does a monitoring system exist?') is risky because the link between processes and outcomes is often hypothetical and untested. The resulting measure of performance could well be at odds with realities on the ground. Ideally, the aim should be to measure outcomes that originate from an LG's processes. Regression analysis could be used to control for initial endowments (the outcome components not attributable to LG processes) and to measure precisely how different `processes'(policies) contribute to outcomes.9 However, this assumes that a benchmark`performance'variable can be identified and used as a dependent variable. Given that this LGPM tool defines performance as a composite across very different areas, identifying one such variable is problematic. Likewise, at the pillar level there are considerable practical obstacles. Furthermore, adopting sophisticated econometric techniques may run against the requirements of transparency and simplicity, which are essential if ordinary citizens are to understand and trust the results. Recommendation 1: Measuring outcomes or processes Given the technical impossibility of elucidating exactly how processes translate into outcomes and that citizens care most about outcomes, the recommended option it to use a mixture of both outcome indicators and process indicators. This allows LGs to be rewarded for their good practices (i.e. processes) and their attempts to improve performance through better processes, as well as for the results (i.e. outcomes) that they actually achieve. Figure 3 illustrates the options available, the choices made and the various recommendations offered in designing this performance measurement methodology. 9 Malesky, Edmund. 2007. "The Vietnam Provincial Competitiveness Index: Measuring Economic Governance for Private Sector Develop- ment." Final Report, Vietnam Competitiveness Initiative Policy Paper #12. USAID's Vietnam Competitiveness Inititative: Hanoi, Vietnam. 12 December 2008 Chapter 4 Methodological choices and challenges Making Decentralization Work for Development: Methodology of the Local Government Performance Measurement (LGPM) Framework Figure 3. The various options and stages in measuring local government performance How to measure LG performance Processes or Outcomes Processes and Outcomes Recommendation 1 eg. PEFA MDG-style scorecard "real value" indicators Binary Graduated Scorecard Scorecard Recommendation 2 Continuous Discreet Continuous Discreet n-ary binary n-ary binary Level or Hybrid change (level+change) Level Conversion Scaling Scaling 0/1 (0,1,2,...) (0/1) Weighting Recommendation 3 = decision Nature of the indicators Scoring process Scores Performance Recommendation 4 groupings Overall score Note: PEFA stands for Public Expenditure & Financial Accountability. PEFA is a partnership between the World Bank, the European Commission, the UK's Department for International Development, the Swiss State Secretariat for Economic Affairs, the French Ministry of Foreign Affairs, the Royal Norwegian Ministry of Foreign Affairs, and the International Monetary Fund. PEFA aims to support integrated and harmonized approaches to assessment and reform in the field of public expenditure, procurement and financial accountability. December 2008 13 Making Decentralization Work for Development: Chapter 4 Methodological choices and challenges Methodology of the Local Government Performance Measurement (LGPM) Framework B) `Scoring' of indicators Three factors influence the`scoring'of indicators: selecting the overall scoring approach; measuring levels or changes; and determining thresholds. When measuring both processes and outcomes, the most common options for an overall scoring approach are either simply presenting the real values of indicators or using a scorecard.The drawback of using real value indicators is that the range of data would make it impractical to arrive at an overall aggregated score. Therefore, the scorecard approach is considered preferable. The scorecard approach can be further classified into a binary or graduated scorecard. In the binary scorecard, the values of indicators can only be `0' or `1', while in the graduated scorecard they can be a range of values. Each approach, whether binary or graduated, has its pros and cons: the binary scorecard is easy to understand and apply, but can over-simplify the real picture and lacks detailed information; conversely, the graduated scorecard is more complex. While a scoring process for binary indicators is obviously not required, it needs to define the conversion of non-binary indicators. In the binary scorecard, there is a need to convert non-binary indicators into binary scores, while in the graduated scorecard there is a need to convert continuous indicators into discreet scores.The conversion itself should be done by setting performance benchmarks or thresholds. In the case of the binary scorecard approach, both continuous and"n-ary"indicators need to be converted into binary scores. This can be done by measuring either levels or change, or a combination of both. As with the overall scoring approach (binary or graduated), measuring either levels or change also has pros and cons for local governments. If choosing levels, LGs with a high level of performance will always be rewarded, but will have little incentive to improve. Meanwhile, LGs that register a significant improvement but are still below the threshold will obviously be disappointed, since their improvement will go unrecognized. On the other hand, measuring only change will be unfair to LGs that already perform well but do not register a significant improvement. Another alternative is to combine level and change in one hybrid indicator. 10 For instance, the indicator could be: "Is the literacy rate above X% or has it improved by at least Y% over the last year?" This would allow LGs committed to reform to be rewarded irrespective of their original starting point while not penalizing the best performers whose margins of improvement are minimal. After considering the alternatives, the best choice for a scorecard approach is to use levels as benchmarks as opposed to change.The key reason for this is that ideally performance measurement should be repeated over time, allowing for an analysis of not only the best overall performance, but also the greatest improvement between the iterations. Whether a binary or graduated scorecard approach is used, determining the thresholds -- or break points that allow the conversion of continuous data into scores -- is critical.Two key choices to be made are: (i) determining thresholds against predetermined levels or in relation to observed country-wide performance; and (ii) adopting benchmarks fixed in time or updated yearly. Given that the objective of the LGPM tool is to measure LG performance against feasible standards, the model should be calibrated using Indonesia-specific criteria, making sure that thresholds are high enough to demonstrate performance and remain relevant over time, while still constituting achievable targets and providing incentives to lagging districts. As a result, LGs will be encouraged to improve their performance against the levels actually attained by other districts within the same overall socio-political environment. 10 See Annex 3 for a more detailed discussion of the use of hybrid benchmarks in a scorecard. 14 December 2008 Chapter 4 Methodological choices and challenges Making Decentralization Work for Development: Methodology of the Local Government Performance Measurement (LGPM) Framework Recommendation 2: Scoring of indicators The graduated scorecard approach is preferred to the alternatives. As one of the main stakeholders, local governments should find the graduated scorecard approach easy to understand, while the approach will also provide an incentive for continual LG improvement. Although to some extent this approach does simplify reality, the multiple possibilities of the values of indicators still reflect important information about LG performance. With the binary scorecard approach, there is a danger that once a particular threshold has been met, a district leader will have little incentive to improve performance beyond that marker, as the district leader has already received his/her point. Similarly, citizens in the district will have little sense of the improvement of their district relative to other districts. Simply counteracting this lack of incentive by choosing high indicator scores would lead to too many zero scores and decreased variation in the ranking. The selection of the graduated scorecard approach means that levels, and not changes, will be measured. While discreet indicators can be scored instantly, continuous indicators need to be scaled based on certain thresholds. The exact threshold can only be chosen once the distribution of the data is available. For example, threshold levels could be chosen by percentiles, such that (i) above the 90th percentile; (ii) between the 75th and 90th percentile; (iii) between the 50th and 75th percentile; (iv) between the 25th and 50th percentile; and (v) below the 25th percentile. The actual score and not the percentile ranking would be the threshold level. These scores should then be kept over time, so that districts can monitor either improvement. C) Aggregating within and across LG performance pillars Given the absence of a clear-cut indicator of overall LG performance at this time and, hence without the help of regression analysis, attributing weights to the different indicators within pillars and to the pillars themselves amounts to guesswork.Therefore, keeping in line with the requirements for clarity and transparency, the simplest aggregation method is to weight each indicator equally for each pillar. However, this affects the meaning of the overall index, which should be understood not so much as an accurate ranking tool based on exact and fully objective scores, but more as a quick diagnostic tool providing a snapshot of overall performance in each district, as well as giving insights into which specific areas or sectors are driving the observed overall performance. Alternatively, different stakeholders have the latitude to re-weight the components and compute new scores reflecting their individual priorities. For example, a province that places greater store on basic service provision could give more weight to the service delivery pillar for those LGs under its jurisdiction. Recommendation 3: The issue of weighting Equal weight for all indicators and pillars is recommended in this framework. However, stakeholders are also encouraged to weight pillars and indicators according to their individual needs. Caution is needed to ensure that there is no implicit weighting based on the number of indicators in each pillar or sub-components of pillars. Also, where cross-LG comparisons are envisaged, a similar weighting technique must be applied to all those districts included in the comparison. December 2008 15 Making Decentralization Work for Development: Chapter 4 Methodological choices and challenges Methodology of the Local Government Performance Measurement (LGPM) Framework D) Determining performance groups Where the data are concentrated, small improvements in an LG's overall score could lead to a large jump in rank. By contrast, organizing the data into broad categories of performance could provide a vivid picture of the districts/ regions whose performances are outstanding, average or sub-standard. A map of Indonesia's LGs colored by performance groups is provided in Annex 2 for illustrative purposes. Several options exist for identifying such performance groups and the collected data will help to evaluate their respective merits.The most straightforward option in determining performance groups is to identify`jumps'or break points in the observed scores. Recommendation 4: The Issue of performance groupings Equal weight for all indicators and pillars is recommended in this framework. However, stakeholders are also encouraged to weight pillars and indicators according to their individual needs. Caution is needed to ensure that there is no implicit weighting based on the number of indicators in each pillar or sub-components of pillars. Also, where cross-LG comparisons are envisaged, a similar weighting technique must be applied to all those districts included in the comparison. E) Assessing relevance The LGPM tool is necessarily based on a model version of reality using a limited set of indicators as proxies for a much broader spectrum of variables. It is therefore by no means a given that an overall score is an perfect reflection of reality, particularly when providing an assessment of a broad range of factors as in the case of LGPM. At least two methods can be used to validate the relevance of the tool: - Comparingtheindividualpillarscoreswiththeresultsofmoredetailedsurveys. For instance, the results of the PFM pillar were compared with the scores generated by the full-scale PFM assessment results.11 A rank correlation close to 0.9 was observed, indicating a high degree of congruence. The comparison could be extended to the investment climate pillar once the results of the KPPOD pilots become available. However, as yet no comparable surveys exist for the Fiscal Performance or Service Delivery components of the survey. - Comparingoverallindicatorswiththeresultsofothersurveys. For instance, matching LGPM overall indicators with the results of perception surveys on the quality of local government would establish whether LGPM results are`in sync'with public opinion. 11 The full-scale PFM assessment comprises 156 indicators, while the PFM pillar of LGPM consists of 25 indicators. 16 December 2008 Chapter 5 Piloting activities Making Decentralization Work for Development: Chapter 5 Piloting Activities Methodology of the Local Government Performance Measurement (LGPM) Framework The team developed a list of potential indicators (see Annex 1) to be piloted in a number of districts. The LGPM tool was piloted in three locations in September-November 2007 with the following objectives: 1. To check the relevance of indicators in the PFM, fiscal performance, and service delivery pillars with conditions in LGs. 2. To test the draft survey questionnaires. 3. To obtain inputs for the survey implementation and identify potential problems. To meet these objectives, locations were selected based on following factors: 1. Level of perceived performance. 2. Geographical location. 3. Accessibility. Based on these factors, Kota Tangerang, Kabupaten Solok, and Kabupaten Biak-Numfor were selected as piloting locations. Kabupaten/ Kota Kabupaten Solok Kabupaten Biak-Numfor Kota Tangerang Province Sumatra Barat Papua Banten Population 347,288 105,015 1,481,591 Area (sq. km) 3,738.00 3,554.62 186.97 HDI 68.28 66.93 73.86 GRDP per capita (IDR) 7,023,230 7,743,959 20,262,450 Before the piloting, the LGPM team developed the survey instruments that would guide the field surveyors during the survey implementation. The instruments provided a series of questionnaires and a list of secondary data to be collected. Learning from other surveys, some indicators have to be dropped after the piloting. For this reason, all indicators that were considered relevant in this survey were piloted. In each piloting, the survey implementation was simulated by collecting the data directly from LG working units (SKPD) and interviewing the relevant officials. After obtaining an official permit, piloting started with a courtesy visit to the head of LG. Guided by the draft survey instruments, this was followed by visits to SKPDs that included interviews with relevant officials and secondary data collection. On average, all activities in a piloting location were completed in four days or less. After finishing the piloting in one location, all findings and materials were reviewed to improve survey questionnaires and the implementation plan. Results from piloting showed that most indicators were relevant in capturing LG performance. Based on the filled survey instruments all qualitative indicators, as well as some quantitative ones with already defined thresholds, were measured. Some indicators were discarded because they were not supported by convincing evidence or considered not sufficiently relevant. The most significant omission was the procurement-related indicators in the PFM pillar. The absence of central procurement units in districts made it almost impossible to measure these indicators. In practice, procurement committees are formed on an ad hoc basis and dissolved once a contract has been awarded. 18 December 2008 Chapter 5 Piloting Activities Making Decentralization Work for Development: Methodology of the Local Government Performance Measurement (LGPM) Framework Figure 4. Local Government Performance Index SD (17) 0.7 FP (20) 0.6 PFM (25) 0.5 0.4 0.3 0.2 0.1 0 Kota Tangerang Kab. Solok Kab. Biak Numfor The varying performance of piloted LGs showed that the indicators were sensitive in measuring their performance. In total, the piloting data managed to measure 25, 20, and 17 indicators for the PFM, FP, and SD pillars, respectively. Achievements in each pillar were aggregated, resulting in a LG performance index illustrated above. In addition to the substantive findings, the piloting also revealed numerous important operational findings, such as: 1. Obtaining official permits from LG heads is necessary if full cooperation is to be forthcoming from all officials; 2. The early phases of the fieldwork should be dedicated to assessing data availability; and 3. In order to ensure the effectiveness and efficiency of the data collection in an LG, it is important that the team identify and set up appointments with relevant officials prior to the fieldwork. After the completion of the piloting, the LGPM team discussed the survey implementation strategy and came to the following conclusions: 1. The formal letter to the bupati/ walikota should be submitted at least four weeks before the fieldwork in one particular district. 2. One week after sending the letter, that LG is contacted to ensure that the letter has been received and followed up. We should also ask the LG to send a copy of the bupati/ walikota's endorsement letter to be brought by the enumerators in their fieldwork. 3. Teams can not begin the fieldwork in a particular district without obtaining the bupati/ walikota's endorsement letter. 4. After obtaining the endorsement letter, teams should send the data sheet to the related SKPDs and ask them to prepare the documents for cross-checking purposes. 5. The first activity of the fieldwork in a particular district is to pay a courtesy visit to the bupati/ walikota or the secretary of the province/district (Sekda). It is strongly recommended that bupati/ walikota be requested to arrange meetings with all heads of surveyed SKPDs in order to gain better access and acceptance. December 2008 19 Making Decentralization Work for Development: Chapter 5 Piloting Activities Methodology of the Local Government Performance Measurement (LGPM) Framework 20 December 2008 Annexes Making Decentralization Work for Development: Annexes Methodology of the Local Government Performance Measurement (LGPM) Framework Annex 1: Proposed list of indicators Indicators Source Note Pillar 1: Public Financial Management I. Local Regulatory Framework 1. LG has a law (Perda) on local financial Setda-Law Section The Perda complies with management Government Regulation (GR) No. 58/2005. 2. There is an integrated agency for financial Setda-Law & finance sections Dispenda, finance section, and local management (BPKD) cash office (KKD) are merged into one agency (BPKD) 3. LG has a law (Perda) on transparency and/ Setda-Law Section There is a Perda that requests LG or participation disclosure of all budget and financial data to the public. 4. An incentive and sanctions system has Setda-Personnel or Law Section been implemented under a decree/ regulation issued by the head of the region. 5. The public has access to budget sessions DPRD Secretariat in the local parliament II. Planning and Budgeting 6. Programs and activities in RPJMD can be Bappeda measured quantitatively 7. Annual budget includes measurable Bappeda outcome indicators 8. Budget priorities and ceilings are set Bappeda before the budgeting process in SKPD starts 9. The difference between planned and APBD/ Setda-Finance Section This is meant to capture inefficiency realized budget is not more than 5% in on the spending side the previous financial year 10. The draft budget is submitted to the Setda-Finance Section and DRPD local parliament by the first week of secretariat November before the budget year at the latest III. Cash Management 11. Cash disbursement order letters (SP2D) Setda-Finance Section are issued no later than two working days after SPM is received 12. All cash receipts are banked no later Setda-Finance Section than one working day upon receipt 13. No draft Perda on local taxes and Setda-Law Section and DG Fiscal levies has been rejected by provincial Balance government or MoF 14. LG has analyzed potential tax revenue to Setda-Finance Section estimate the revenue target 15. Reconciliations of bank accounts with Setda-Finance Section bank book are conducted monthly 22 December 2008 Annexes Making Decentralization Work for Development: Methodology of the Local Government Performance Measurement (LGPM) Framework IV. Procurement and Asset Management 15. At least one member of the central SKPD Setda- Equipment and bidding committee has a certificate in Development Sections, BKD procurement proficiency. 16. Invitations to bid are announced SKPD Setda- Equipment and Bids above Rp 100 million have to in newspapers, on the national Development Sections be announced in local newspapers. procurement website, or on For bids between Rp 50 million and announcement boards. Rp 100 million posting on boards is sufficient. For lesser amounts direct appointment is acceptable. 17. Bidding explanations are conducted SKPD Setda- Equipment and Only bids above Rp 50 million are openly and attended by all participants, Development Sections concerned who are named in a list of attendees 18. An assessment of the value of locally- SKPD Setda- Equipment Section owned goods is made using the cost principle or relevant regulations in the Government Accounting Standard 19. Local legislation on asset management Setda-Law Section The Perda complies with GR No. 8/ exists 2006. 20. Local legislation on standard unit prices Setda-Law Section exists 21. Goods users carry out a yearly stock Setda-Equipment Section inventory V. Reporting and Accounting 22. The heads of each department in the BKD, BPKD or Cash Office, Setda- Financial Management Unit have an Finance Section, Dispenda accounting or financial management academic background 23. Journals, ledgers, supporting ledgers BPKD Setda - Finance Section and balance sheets all exist 24. Six-monthly budget realization reports Setda-Finance Section and DPRD are reviewed and followed up by the bupati/ walikota 25. Annual financial statements for audit are Setda- Finance Section & Internal audit submitted at the end of April after the agency fiscal year at the latest to BPK 26. Unspecified expenditure is coded and Bappeda disclosed in the financial statement VI. Audit 27. More than half of Internal Audit Agency Bawasda and BKD There is one specific`audit staff are qualified as functional auditors certification'exam to become a (skilled, expert or chief auditor) financial auditor 28. Internal audit agency applies a program SKPD - BKD (audit manuals) There are two manuals: manual and internal audit procedures - one listing the items to be audited - one on the methodology They are sometimes merged. 29. Internal audit agency reviews all SKPD - BKD (audit reports) Each SKPD and local enterprise has activities of the LG including any to be audited once a year commercial activities December 2008 23 Making Decentralization Work for Development: Annexes Methodology of the Local Government Performance Measurement (LGPM) Framework 30. Internal audit reports are available to the SKPD - BKD, BAWASDA Internal audit reports for each SKPD external auditor had to be sent to the governor cc to BPKP, BPK and provincial audit agency 31. Externally audited financial reports are SKPD Setda- Finance (Budget It should be part of the budget publicized in the local mass media, on realization) realization report, which is submitted the official announcement board, or to the local parliament through a website Pillar 2: Fiscal Performance I. Fiscal Management 32. Share of local spending on core admin Setda-Finance section / Dispenda Core admin services include services is below X%12 spending on: Mayor, DPRD, Setda 33. Share of the surplus invested in financial Setda-Finance section/ APBD The surplus (SiLPA) is from the assets below X%12 previous FY. 34. DAK is used accordingly and detailed list Setda-Finance section/ APBD of DAK use is available 35. Share of total local spending going to Setda-Finance section/ APBD Ratio of capital spending to total investment above X%12 spending 36. Absorption rate of public expenditure Setda-Finance section The figure is available in the first-half for the first half of 2007 is at least 30% budget realization report II. Tax and Levy Administration 37. There is an effective hotline/procedure Dispenda or City Hall for taxpayers to report improper behavior of Dispenda agents 38. A website or public display of applicable Dispenda taxes and corresponding rates is available 39. Payments by bank transfer allowed as Dispenda (records) Levies are excluded well as cash 40. All local taxes and levies are in the Dispenda and Setda-Law Section areas/forms stated in Laws No. 34/ 2001 and No. 66/2001, respectively, and established as Perda 41. No local tax/levies exceed the maximum Dispenda and Setda-Law Section amount/rate set in PP No. 65/ 2001 and No. 66/2001 respectively 42. No tax instrument accounts for less than Dispenda/ APBD This includes taxes, charges, levies 5% of PAD revenues and attempts to capture the efficiency of the tax system 43. There are no regressive subsidies Setda-Finance section/ Dispenda 44. There are no charges/fees limiting inter- Setda-Finance section / Dispenda district trade III. Revenue Effort 12 The`X'value can only be determined after the data collection 24 December 2008 Annexes Making Decentralization Work for Development: Methodology of the Local Government Performance Measurement (LGPM) Framework 45. Penalties are imposed on late payments Dispenda of local taxes and charges and an updated list of outstanding debtors is kept 46. The BPKD/ Dispenda sets quantified Dispenda revenue targets based on an analysis of revenue growth potential 47. PAD revenues have grown by 5% in real APBD/ Dispenda terms in at least two of the past three years IV. Sectoral Fiscal Management 48. Revenues from local taxes and levies APBD/ Dispenda are recorded in respective collecting LG working units 49. RSUD and Puskesmas are treated as District Health Agency and APBD Separating them from District SKPD/ UPTD Health Agency account should improve accountability 50. Revenues from public hospitals and District Health Agency and RSUD health centers are described in detail 51. Transportation-related levies are Dinas Perhubungan or Dispenda reported in detail 52. Direct costs of education / total Setda-Finance section/ APBD The 2006 budget still used Kepmen spending on education >25% 29/ 2002 format and needs some adjustment 53. Direct costs of health / total spending Setda-Finance section/ APBD on health >70% 54. Share of APBD for infrastructure Setda-Finance section/ APBD Road, bridges, irrigation, stations, development > X%13 ports 55. Share of APBD for infrastructure Setda-Finance section/ APBD maintenance > X%13 56. Combined budgeted expenditure on APBD/ Setda-Finance Section health and education account for > X%13 of the budget Pillar 3: Service Delivery I. Education Planning and Monitoring 57. System for monitoring education District Education Office (Dinas) The idea is to see if the Dinas is program is in place and functioning well comparing data/achievements from year to year 58. Mechanism to use monitoring results District Education Office for program planning is in place and these results actually influence district education program 59. Teachers'opinions are considered in Education Council devising education programs 13 The`X'value can only be determined after the data collection December 2008 25 Making Decentralization Work for Development: Annexes Methodology of the Local Government Performance Measurement (LGPM) Framework Implementation 60. Net enrollment rates for elementary District Education Office / Susenas # of children age 7-12 in primary school over # of children age 7-12 61. Gross enrollment rates for junior District Education Office/ Susenas # of children age 13-15 in junior sec secondary school over # of children age 13-15 62. Literacy rate for 15-24 year District Education Office/ Susenas 63. Junior secondary completion rate for District Education Office/ Susenas # of children age 16-18 completing 16-18 year old junior sec Pro-poor Programs 64. Programs exist for enrolling out-of- District Education Office This does not include any program school youth and are included in the distributed or managed through the District Health Agency's annual budget. school since they do not reach the targeted beneficiaries 65. System of financial support/grants for District Education Office under-privileged children is in place 66. Special programs exist for schools that District Education Office (with lag behind (on monitoring indicators) monitoring officers) and are included in the annual budget. II. Health Planning and Monitoring 67. System for monitoring health program District Health Agency (Dinas) Considering the magnitude of the is in place and functioning well health programs, immunization and pre-natal care are chosen as proxies. 68. Monthly meetings are held between District Health Agency Heads of Puskesmas and District Health Agency 69. Planning unit of the health agency has District Health Agency received training to use District Health Accounts (DHA) system 70. District Health Agency planning unit is District Health Agency reporting according to DHA system 71. A formal registration for private District Health Agency healthcare providers exists and is applied regularly Implementation 72. Childhood immunization coverage District Health Agency / Susenas (proportion of children up to 12 months old having completed all scheduled immunizations 73. Proportion of deliveries assisted by a Susenas trained professional 74. Share of the people using traditional Susenas / District Health Agency healthcare facilities as primary providers 75. Ratio of medical personnel over 100,000 District Health Agency / available in population and over service area Podes for 2005 Pro-poor Programs 76. Share of out-of-pocket expenditures Susenas when using healthcare facilities 26 December 2008 Annexes Making Decentralization Work for Development: Methodology of the Local Government Performance Measurement (LGPM) Framework 77. Minimum service standard (SPM) at the District Health Agency / GDS 2 district level is applied. 78. Share of health spending going to District Health Agency To account for pro-poor spending primary healthcare III. Infrastructure Planning and monitoring 79. General spatial plan (RUTR) exists District Public Works or Spatial Planning Agency 80. Detailed spatial plan (RDTR) exists 81. Green spatial plan (RTRH) exists Implementation 82. Share of district roads in good condition District Public Works Agency Might not be applicable to districts that are largely comprise of islands 83a. Ratio of green area to total district/city District Public Works or Spatial Only for cities area Planning Agency 83b. Ratio of irrigated rice fields District Public Works or Agriculture Only for districts Agency Pro-poor Programs 84. Share of population with access to clean District Public Works Agency / Susenas Based on MDG definition water 85. Share of population with access to Based on MDG definition proper sanitation 86. Share of population with access to electricity IV. Cross-cutting Issues 87. Existence of job-creation LG programs Bappeda 88. Existence of a citizens charter defining Office of the Mayor/District Head obligations and rights of social service users Pillar 4: Investment Climate14 I. Access to Land and Security of Tenure 89. Time taken to obtain a land certificate Business firms 90. Perceived ease of obtaining land Business firms 91. Frequency of evictions in the region Business firms 92. Overall assessment of the significance of Business firms land problems II. Business Licensing 93. % of firms that have a business Business firms registration license (TDP) 94. Firms'perceptions of how easy it is to Business firms obtain a TDP and the average number of days it takes to obtain a TDP 95. The cost of the TDP and the extent to Business firms which the cost bothers firms 14 KPPOD and the Asia Foundation (2008). Data for all indicators are extracted from Local Economic Government Survey (LEG) conducted in 2007. December 2008 27 Making Decentralization Work for Development: Annexes Methodology of the Local Government Performance Measurement (LGPM) Framework 96. The extent to which firms agree that the Business firms licensing process is efficient, and free from illegal charges and collusion 97. % of firms that say that there is a Business firms complaint mechanism III. LG and Business Interaction 98. % of firms that say there is a Forum Business firms Komunikasi between the private sector and the LG 99. The extent to which firms agree that Business firms LG officials provide practical solutions to their problems that meet their expectations and follow up on actions already agreed by the district leader 100. The extent to which firms agree that Business firms the LG understands the needs of the business community; consults on policy changes; has meetings to discuss the problems faced by businesses; and provides facilities to support the development of local businesses 101. Assessment by firms of whether the Business firms LG is extracting revenue from the business sector or is more interested in promoting investment in the region 102. Assessment by firms of whether the Business firms LG treats all firms equally or is biased towards a small minority of firms 103. The extent to which firms agree that, in Business firms practice, the policies of the LG do not increase their costs 104. The extent to which firms agree that, in Business firms practice, the policies of the LG do not increase the level of uncertainty that they face 105. Overall assessment by firms of to what Business firms extent issues associated with the interaction of the LG with the business community constraint their business activities. IV. Business Development Programs 106. Average share of firms saying that Business firms six types of business development program exist (business management training; workforce training; trade promotion; connecting large and small firms; credit application training for SMEs; and business matchmaking programs) 28 December 2008 Annexes Making Decentralization Work for Development: Methodology of the Local Government Performance Measurement (LGPM) Framework 107. Average share of firms saying that Business firms they participated in these six types of business development program 108. Average level of satisfaction with these Business firms programs 109. Overall assessment of the impact of Business firms these programs V. Capacity and Integrity of the Mayor/ Regent 110. Extent to which firms agree that the Business firms local leader has a good understanding of the problems facing business 111. Extent to which firms agree that the Business firms appointment of bureaucrats dealing with business issues in the LG is based on their experience and is appropriate for the section in which they work 112. Extent to which firms agree that the Business firms local leader takes strong action against every instance of corruption by LG officials 113. Extent to which firms agree that the Business firms local leader themselves undertake corrupt actions for their own benefit 114. Extent to which firms agree that the Business firms local leader is a strong leader 115. Firms overall assessment of the extent Business firms to which issues associated with the capacity and integrity of the local leader constrain their business activities VI. Local Taxes, User Charges and other Transaction Costs 116. The extent to which firms say that they Business firms are hindered by user charges 117. % of firms that say that there are official Business firms user charges for transporting goods across district borders and the log of the user charges for distributing goods across district borders per firm employee 118. % of firms that say that they have had Business firms to pay donations or contributions to the LG in the last year and the extent to which these payments bothered the firms 119. Overall assessment by firms of how Business firms much issues associated with licensing constraint their business activities 120. The share of firms that say that they Business firms have to make additional payments to the police December 2008 29 Making Decentralization Work for Development: Annexes Methodology of the Local Government Performance Measurement (LGPM) Framework 121. Overall assessment by firms of how Business firms much issues associated with such transaction costs constrain their business activities VII. Local Infrastructures 122. Average perceived quality of five Business firms types of infrastructure (district roads, street lighting, water from the local water authority (PDAM), electricity, telephone) 123. Average time (in days) to fix problems Business firms with these types of infrastructure 124. % of firms that have a generator Business firms 125. The number of times in a week that the Business firms electricity is cut off 126. The overall assessment of how large a Business firms constraint is posed by problems with infrastructure supplied by the LG VIII. Security and Conflict Resolution 127. % of firms saying that they have Business firms experience theft in the last year 128. The extent to which firms agree that Business firms the police handle cases in a punctual fashion, benefiting the firm, and minimizing the firm's loss of time and money 129. The extent to which firms agree that Business firms the police handle cases of worker demonstrations in a punctual fashion, minimizing the firm's loss of time and money 130. Overall assessment by firms of how Business firms much issues associated with security and conflict resolution constrain their business activities IX. Local Regulations 131. The Sub-index for the Quality of Local Regulations was constructed around the three categories of potential problems: legality, substance and principle, each with a subset of supporting variables. The detailed method is available in KPPOD and the Asia Foundation (2008). 30 December 2008 Annexes Making Decentralization Work for Development: Methodology of the Local Government Performance Measurement (LGPM) Framework Annex 2: Mapping performance groups: an illustration To illustrate the measurement of LG performance, the team simulated the"Doing Business"indexing methodologies to the quantitative service delivery indicators listed in Annex Table 2 by using the Susenas data. The index of each district is a simple average of district percentile rankings on each of the 10 indicators. Table Annex 2. List of simulated service delivery indicators Indicators Explanation Net enrollment rates for elementary Percentage of children aged 7-12 years who are currently in elementary education Net enrollment rates for junior secondary Percentage of children aged 13-15 years who are currently in junior secondary education Junior secondary completion rates for people aged Percentage of youth aged 16-18 years who have completed junior 16-18 secondary education Literacy rate for people aged 15-24 Percentage of people aged 15-24 years who are able to read and write in Latin alphabets Immunization coverage Percentage of babies aged 12-23 months who have taken all scheduled immunizations Child delivery assisted by trained professional Percentage of deliveries assisted by trained professional The use of traditional health care Percentage of sick people who go to traditional health facilities Access to clean water Percentage of households with access to clean water Access to proper sanitation Percentage of households with access to proper sanitation Electricity coverage Percentage of households with electricity Geographical map based on relative performance of all LGs based on selected service delivery indicators Performance Groups >=80 (5.08%) 70-79.9 (12.24%) 60-69.9 (17.09%) 50-59.9 (12.70%) 40-49.9 (21.25%) 30-39.9 (21.02%) 20-29.9 (6.47%) 0-19.9 (4.16%) December 2008 31 Making Decentralization Work for Development: Annexes Methodology of the Local Government Performance Measurement (LGPM) Framework Annex 3: Hybrid indicators While it is true that focusing on levels privileges those who have better endowments at the onset of decentralization, the inclusion of change rates unfortunately means that comparison periods must be continually updated. This will damage year-to-year comparability. Related to this point is that using levels conforms better to the scoring approach. While thresholds can be easily identified and can be selected to hold over time for level indicators, change-rate thresholds need to be constantly updated as the mean level of indicators across the sample increases. For instance, when literacy is generally low among districts, a 10 percent annual increase might be considered a worthwhile benchmark. But 15 years from now, when mean literacy is higher (perhaps around 80 percent), a 10 percent increase would be much harder to achieve. The thresholds would need to be changed, which means the overall index scores cannot be reasonably compared over time. District leaders would find this unsatisfactory. Moreover, keeping change rates along with levels could create the odd situation in which a district's overall ranking falls on a particular pillar between years while its overall level improves. This might occur if an LG improved its overall score level, but the degree of improvement was smaller than the average improvement of those in other districts. In this case, this LG would fall in the overall ranking. It is also fair to assume that the eight years since decentralization is a decent time-frame over which to hold LGs accountable for levels on many of these indicators. After eight years, a level should be a reasonable approximation of annual effort in achieving reform goals. Nevertheless, if after data collection and analysis it becomes apparent that initial conditions have a significant impact on LG performance on the four pillars, the final report can sub-divide districts by their initial endowments and rank provinces by their performance for a given level of initial conditions. 32 December 2008 Annexes Making Decentralization Work for Development: Methodology of the Local Government Performance Measurement (LGPM) Framework December 2008 33